Fundus image enhancement method and system based on machine learning, electronic equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence and fundus image enhancement, and discloses a fundus image enhancement method and system based on machine learning, electronic equipment and a storage medium, and the method comprises the steps: obtaining an original fundus spectral image, and carrying out the preprocessing of the original fundus spectral image, and obtaining a preprocessed spectral image; constructing a backbone network based on a residual network, and extracting multi-scale features of the preprocessed spectral image in combination with cavity convolution; introducing a channel-space-spectrum multi-attention module into the backbone network, and performing multi-attention fusion on the multi-scale features to obtain an enhanced feature map; and performing adversarial training on the backbone network by using the generative adversarial network and the enhanced feature map, and performing image enhancement on the collected fundus spectral image by using the trained network to obtain an enhanced fundus spectral image. According to the method, more-dimensional image support is provided for medical diagnosis, the quality of the fundus image can be effectively improved, the diagnosis accuracy of a doctor on fundus lesions is improved, and the method has important clinical application value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and fundus image enhancement technology, and in particular to a fundus image enhancement method, system, electronic device and storage medium based on machine learning. Background Art

[0002] Fundus images, as the core basis for the diagnosis of ophthalmic diseases, can directly reflect the morphology of retinal blood vessels, optic disc structure and macular lesions (such as hemorrhage, exudation, etc.), and are crucial for the early screening of diseases such as diabetic retinopathy (DR), glaucoma, and age-related macular degeneration (AMD). However, due to factors such as the performance of imaging equipment, shooting environment and patient cooperation, the fundus images actually collected often have the following quality problems: (1) Low contrast: due to pupil reflex, turbidity of refractive medium or insufficient exposure of the equipment, the distinction between blood vessels and background is reduced; (2) Noise interference: including sensor noise (such as Gaussian noise, shot noise), motion artifacts (micro-movement of the patient's eye) and optical scattering noise; (3) Uneven illumination: the center of the image is overexposed and the edge is underexposed, forming a ring-shaped brightness gradient; (4) Blurred details: small lesions (such as microaneurysms, hard exudates) are difficult to identify due to resolution limitations or defocus problems.

[0003] Traditional image enhancement techniques primarily rely on manually designed feature extraction and transformation methods. For example: Histogram equalization (HE) and its variant (CLAHE): These techniques enhance contrast by expanding the image's grayscale distribution, but can easily lead to over-enhancement of normal tissue, noise amplification, and inability to distinguish pathological areas from the background; Wavelet transform and multi-scale retinex (MSR): These techniques achieve spectral correction based on frequency domain decomposition, but their ability to suppress complex noise is limited, and parameters require manual adjustment, resulting in poor generalization; Morphological operations and vascular enhancement filtering: These techniques exploit the tubular structure of blood vessels to design directional filters (such as Frangi filters), but are only applicable to specific vessel types and are ineffective in enhancing lesions (such as hemorrhages).

[0004] In recent years, deep learning technology has provided new insights for medical image enhancement. Convolutional neural network (CNN)-based methods (such as U-Net and ResNet) can automatically map degraded images to clear images through end-to-end learning. However, existing research still has significant limitations: Insufficient model generalization: Existing algorithms are mostly trained on a single device or specific dataset, making them difficult to adapt to different imaging devices (such as fundus cameras and OCT), different resolutions (such as 30° and 45° fields of view), and ethnic differences (such as low contrast caused by dark irises); Risk of medical feature loss: Traditional mean squared error (MSE) or L1 loss functions tend to produce smoothed results, which may lead to microvascular rupture, blurring of tiny lesions, and even the creation of false structures (such as artifact vessels), misleading clinical diagnosis; Real-time bottlenecks: While complex network structures (such as densely connected networks) can improve enhancement effects, they have large parameter counts and slow inference speeds, making them difficult to integrate into embedded medical devices or mobile applications; Lack of interpretability: Black-box models struggle to provide visual evidence of the enhancement process, limiting physicians' trust in the results, especially in medical scenarios with high legal risks.

[0005] Furthermore, existing methods often ignore the practical needs of clinical workflows. For example, in telemedicine or primary care screening, images may undergo multiple compression transmissions, resulting in information loss, and existing enhancement algorithms are not optimized for such degradation patterns. In dynamic diagnosis and treatment scenarios (such as intraoperative navigation), existing methods cannot meet the millisecond-level real-time processing requirements. Therefore, the current technical pain point can be summarized as: how to achieve high-fidelity, highly robust fundus image enhancement that meets clinical diagnostic requirements under complex degradation conditions. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a fundus image enhancement method based on machine learning, the method comprising:

[0007] Acquiring an original fundus spectral image, and preprocessing the original fundus image to obtain a preprocessed spectral image;

[0008] Constructing a backbone network based on a residual network and extracting multi-scale features of the preprocessed spectral image in combination with dilated convolution;

[0009] Introducing a channel-spatial-spectral multi-attention module into the backbone network, and performing multi-attention fusion on the multi-scale features to obtain an enhanced feature map;

[0010] The backbone network is trained adversarially using a generative adversarial network and the enhanced feature map, and the trained network is used to perform image enhancement on the collected fundus spectral image to obtain an enhanced fundus spectral image.

[0011] Preferably, the pretreatment method comprises:

[0012] Splitting the original fundus spectral image according to wavelength bands to obtain wavelength band spectral images;

[0013] Normalizing each of the wavelength band spectral images to obtain a normalized image;

[0014] The normalized image is subjected to spectral correction and noise suppression to obtain a preprocessed spectral image.

[0015] Preferably, the method for extracting the multi-scale features includes:

[0016] Based on the ResNet-34 model architecture, the fully connected layers of the model are removed and four residual layers are retained to obtain the backbone network.

[0017] Introducing the dilated convolution into the third residual layer and the fourth residual layer of the backbone network to improve the backbone network;

[0018] The first residual layer and the second residual layer are used to extract low-level feature maps, and the improved third residual layer and the improved fourth residual layer are used to extract high-level feature maps;

[0019] The low-level feature map and the high-level feature map are connected through lateral connection, and the high-level features are upsampled to the same resolution as the low-level feature map using bilinear interpolation, and the low-level feature map is fused with the upsampled high-level feature map to obtain the multi-scale features.

[0020] Preferably, the method for obtaining the enhanced feature map includes:

[0021] Input the multi-scale features into the channel attention module, generate a channel description vector through global average pooling, and use a fully connected layer to learn the inter-channel dependency through the channel description vector to obtain the channel attention weight;

[0022] The multi-scale features are input into the spatial attention module along the channel dimension, and the spatial feature map is generated by using maximum pooling and average pooling. The generated spatial feature map is spliced ​​and passed through the convolution layer to obtain the spatial attention weight;

[0023] The multi-scale features are input into the spectral attention module to analyze the spectral dimension, calculate the similarity between different bands, and generate spectral attention weights;

[0024] The channel attention weight, the spatial attention weight, the spectral attention weight and the multi-scale features are fused, and the fusion result is refined by 3D convolution to obtain the enhanced feature map.

[0025] The present invention also provides a fundus image enhancement system based on machine learning, wherein the system is used to implement any of the above methods, including:

[0026] Image preprocessing module, feature extraction module, feature fusion module and image enhancement module;

[0027] The image preprocessing module is used to obtain an original fundus spectral image and preprocess the original fundus image to obtain a preprocessed spectral image;

[0028] The feature extraction module is used to construct a backbone network based on a residual network and extract multi-scale features of the preprocessed image in combination with dilated convolution;

[0029] The feature fusion module is used to introduce a channel-space-spectral multi-attention module into the backbone network and perform dual-attention fusion on the multi-scale features to obtain an enhanced feature map;

[0030] The image enhancement module uses the generative adversarial network and the enhanced feature map to perform adversarial training on the backbone network, and uses the trained network to perform image enhancement on the collected fundus spectral image to obtain an enhanced fundus spectral image.

[0031] Preferably, the image preprocessing module includes: an image splitting unit, a normalization unit and a correction and denoising unit;

[0032] The image splitting unit is used to split the original fundus spectral image according to bands to obtain band spectral images;

[0033] The normalization unit normalizes each of the band spectrum images to obtain a normalized image;

[0034] The correction and denoising unit is used to perform spectral correction on the normalized image and perform noise suppression to obtain a preprocessed spectral image.

[0035] Preferably, the feature extraction module includes: a network construction unit, a network improvement unit, a feature extraction unit and a first feature fusion unit;

[0036] The network construction unit is based on the ResNet-34 model architecture, removing the fully connected layer of the model and retaining 4 residual layers to obtain the backbone network;

[0037] The network improvement unit is used to introduce the dilated convolution into the third residual layer and the fourth residual layer of the backbone network to improve the backbone network;

[0038] The feature extraction unit extracts low-level feature maps using the first residual layer and the second residual layer, and extracts high-level feature maps using the improved third residual layer and the improved fourth residual layer;

[0039] The first feature fusion unit connects the low-level feature map and the high-level feature map through a horizontal connection, upsamples the high-level features to the same resolution as the low-level feature map using bilinear interpolation, and fuses the low-level feature map with the upsampled high-level feature map to obtain the multi-scale features.

[0040] Preferably, a channel attention calculation unit, a spatial attention calculation unit, a spectral attention calculation unit and a second feature fusion unit;

[0041] The channel attention calculation unit is used to input the multi-scale features into the channel attention module, generate a channel description vector through global average pooling, and use a fully connected layer to learn the inter-channel dependency through the channel description vector to obtain a channel attention weight;

[0042] The spatial attention calculation unit is used to input the multi-scale features into the spatial attention module along the channel dimension, generate a spatial feature map using maximum pooling and average pooling, and splice the generated spatial feature map and pass it through the convolution layer to obtain the spatial attention weight;

[0043] The spectral attention calculation unit is used to input the multi-scale features into the spectral attention module, perform spectral dimension analysis, calculate the similarity between different bands, and generate spectral attention weights;

[0044] The second feature fusion unit is used to fuse the channel attention weight, the spatial attention weight, the spectral attention weight and the multi-scale feature, and perform 3D convolution refinement on the fusion result to obtain the enhanced feature map.

[0045] The present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the fundus image enhancement method based on machine learning when executing the program.

[0046] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, the fundus image enhancement method based on machine learning is implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The present invention first splits the spectral fundus image into multi-band images and adopts an adaptive normalization algorithm and spectral correction to effectively solve the noise interference problem existing in the original image, improve the image quality and signal-to-noise ratio, and lay a good foundation for subsequent feature extraction;

[0049] (2) The residual network-based backbone network constructed by the present invention combined with dilated convolution can efficiently extract multi-scale features. This design not only retains the detailed information of the image, but also expands the receptive field through dilated convolution, further enhancing the feature expression capability of key structures such as blood vessels and lesions in fundus images;

[0050] (3) The channel-spatial-spectral multi-attention module introduced in the present invention can accurately fuse multi-scale features, further strengthening the expression of spectral information in the feature map, enabling the network to focus more on key areas in the fundus image, and improving the pertinence and effectiveness of feature extraction;

[0051] (4) The present invention uses a generative adversarial network (GAN) for adversarial training to further optimize the performance of the backbone network, so that the enhanced fundus images are significantly improved in contrast, clarity and detail, closer to the real fundus structure, and provide higher-quality image support for medical diagnosis. This enhancement method can not only effectively improve the quality of fundus spectral images, but also improve the accuracy of doctors' diagnosis of fundus lesions, and has important clinical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0054] Figure 2 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention.

[0055] Among them, 1010 is a processor; 1020 is a memory; 1030 is an input / output interface; 1040 is a communication interface; and 1050 is a bus. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0058] Example 1

[0059] In this embodiment, if Figure 1 As shown, a fundus image enhancement method based on machine learning includes:

[0060] S1. Obtain an original fundus image and preprocess the original fundus image to obtain a preprocessed image.

[0061] Preprocessing methods include:

[0062] Splitting the original fundus spectral image according to wavelength bands to obtain wavelength band spectral images;

[0063] Normalizing each of the band spectral images to obtain a normalized image;

[0064] The normalized image is subjected to spectral correction and noise suppression to obtain a preprocessed spectral image.

[0065] In this embodiment,

[0066] The original fundus spectral image is split according to the bands to obtain band spectral images:

[0067] The spectral image contains multiple bands (such as ultraviolet, visible light, near infrared, etc.), each band provides different spectral information. In this embodiment, the original fundus spectral image is split according to ultraviolet, visible light and near infrared to obtain band spectral images of the corresponding bands.

[0068] Normalize each of the band spectral images to obtain a normalized image:

[0069] For each band of the spectral image, calculate its global minimum (min) and maximum (max). Normalize each pixel value (I) to obtain the normalized result I norm , the formula is as follows:

[0070]

[0071] The normalized image is spectrally corrected and noise suppressed to obtain a preprocessed spectral image:

[0072] Through spectral filtering technology, spectral noise in the spectral image is removed, effective spectral information is retained, and the quality of the spectral image is improved. In this embodiment, a Gaussian filter is used for spectral filtering: the parameters of the Gaussian filter, including the standard deviation (σ) and the filter window size, are set, which can be selected according to the spectral resolution and noise characteristics of the spectral image. For the spectral image of each band, a Gaussian filter is applied along the spectral dimension (i.e., the wavelength direction) to smooth the spectral curve of each pixel. The formula is as follows:

[0073]

[0074] Where I(λ) is the pixel value of the original spectral image at wavelength λ, and G(Δλ) is the weight of the Gaussian filter at the wavelength offset Δλ.

[0075] S2. Construct a backbone network based on the residual network and combine it with dilated convolution to extract multi-scale features of the preprocessed image.

[0076] The method for extracting multi-scale features includes: based on the ResNet-34 model architecture, removing the fully connected layer of the model, retaining 4 residual layers, and obtaining a backbone network; introducing void convolution in the third residual layer and the fourth residual layer of the backbone network to improve the backbone network; using the first residual layer and the second residual layer to extract low-level feature maps, and using the improved third residual layer and the improved fourth residual layer to extract high-level feature maps; connecting the low-level feature map and the high-level feature map through horizontal connection, and using bilinear interpolation to upsample the high-level features to the same resolution as the low-level feature map, and fusing the low-level feature map with the upsampled high-level feature map to obtain multi-scale features.

[0077] In this embodiment, improvements are made based on the ResNet-34 architecture, and the first four residual layers are retained as the core modules for feature extraction. Specifically: (1) The input layer uses a 7×7 convolution kernel for initial feature extraction, with a step size of 2 and 64 output channels; (2) The four residual layers contain 3, 4, 6, and 3 residual blocks, respectively, corresponding to downsampling multiples of 4, 8, 16, and 32; (3) The global average pooling layer and the fully connected layer at the end of the original network are removed to retain the convolutional feature output capability; (4) The output feature map sizes of each residual layer are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image resolution, respectively. Then, multi-level dilated convolution is introduced in the third and fourth residual layers to expand the receptive field: (1) the standard 3×3 convolution of the third residual layer is replaced by dilated convolution with a dilation rate of 2; (2) the fourth residual layer adopts a cascaded dilated convolution structure, and sets three groups of parallel convolution branches: branch 1: 3×3 standard convolution (dilation rate 1), branch 2: 3×3 dilated convolution (dilation rate 2), branch 3: 3×3 dilated convolution (dilation rate 4); (3) multi-scale receptive field information is fused through feature splicing; (4) a "hybrid dilated convolution" strategy is adopted to avoid the grid effect and ensure continuous coverage. Then, low-level feature maps and high-level feature maps are extracted, including: (1) low-level feature extraction: the first residual layer (1 / 4 resolution) and the second residual layer (1 / 8 resolution) are used to capture detailed features such as vascular texture; (2) high-level feature extraction: the improved third and fourth residual layers (1 / 16 and 1 / 32 resolutions) are used to extract semantic features such as lesion areas; (3) feature resolution alignment: high-level features are bilinearly upsampled to low-level resolution by 4×, and the number of channels is adjusted to match the low-level features using 1×1 convolution, and learnable upsampling parameters are introduced to improve interpolation accuracy. Finally, the low-level feature map is fused with the upsampled high-level feature map: (1) A top-down feature propagation path is established to fuse the high-level semantic features with the low-level detail features step by step; (2) An element-by-element addition fusion method is adopted to retain the original feature distribution characteristics; (3) Batch normalization and ReLU activation processing are performed before fusion; (4) The final output contains a feature pyramid with four scales (1 / 4, 1 / 8, 1 / 16, and 1 / 32); (5) The number of channels of feature maps of different scales is set to 256, 512, 1024, and 2048, respectively, and finally a multi-scale feature F is obtained.

[0078] S3. Introduce a channel-spatial-spectral multi-attention module into the backbone network, and perform multi-attention fusion on multi-scale features to obtain an enhanced feature map.

[0079] The method for obtaining the enhanced feature map includes: inputting multi-scale features into the channel attention module, generating a channel description vector through global average pooling, and using a fully connected layer to learn the inter-channel dependency through the channel description vector to obtain the channel attention weight; performing maximum pooling and average pooling on the multi-scale features along the channel dimension to generate two spatial feature maps, splicing the two generated spatial feature maps and passing them through a 7×7 convolution layer to obtain the spatial attention weight; fusing the channel attention weight, spatial attention weight and multi-scale features, and performing 3×3 convolution refinement on the fusion result to obtain the enhanced feature map.

[0080] In this embodiment, a channel-spatial dual attention module is first constructed: (1) the module input is the multi-scale feature pyramid (4-level feature map) output by the S2 stage; (2) a parallel dual-branch structure is adopted: the channel attention branch: based on the SE-Net improvement, dynamic weight calibration is added; the spatial attention branch: combined with the CBAM spatial attention mechanism, a multi-scale receptive field is introduced; (3) a cross-feedback mechanism is designed: the channel attention output is used as a priori constraint for spatial attention, and the spatial attention result feeds back to the channel weight update; (4) the overall calculation process of the module: input feature → channel weight calibration → spatial position enhancement → feature reconstruction → output enhanced feature. The channel attention mechanism is as follows: (1) Global feature compression: Global average pooling (GAP) is performed on the input feature map to generate a C×1×1 channel description vector, and global standard deviation pooling (GSP) is added as a supplementary statistic; (2) Dynamic weight learning: A two-layer fully connected network (FC-ReLU-FC) is constructed, where the first layer FC compresses the number of channels to C / 16 (ReLU activation), and the second layer FC restores the original channel dimension (Sigmoid activation); (3) The calculation formula of the channel attention weight is:

[0081] M c (F)=σ(FC2(δ(FC1([GAP(F);GSP(F)]))))

[0082] Among them, σ represents the Sigmoid function, δ represents the ReLU activation function, and M c (F) represents the channel attention weight. The spatial attention mechanism is as follows: (1) Feature aggregation: Max pooling (MaxPool) and average pooling (AvgPool) are performed along the channel dimension to generate two H×W×1 primary spatial feature maps; (2) Feature fusion: The double pooling results are concatenated into channels (H×W×2), and cross-position relationship modeling is performed through 7×7 depthwise separable convolution. Group normalization (GroupNorm) is used to stabilize the training process; (3) The calculation formula of spatial attention weight is:

[0083] M s (F)=σ(Conv 7×7([MaxPool(F);AvgPool(F)]))

[0084] Among them, Conv7×7 contains 3 layers: convolution layer, GN layer and PReLU activation layer.

[0085] The multi-scale features are input into the spectral attention module to analyze the spectral dimension, calculate the similarity between different bands, and generate the spectral attention weight:

[0086] The input multi-scale feature map is analyzed in the spectral dimension, and the similarity between different bands is calculated to generate a spectral similarity matrix. The spectral similarity matrix is ​​used to generate spectral attention weights. Normalization is performed to ensure that the sum of the spectral attention weights is 1.

[0087] Finally, the channel attention weight, spatial attention weight and multi-scale features are fused:

[0088]

[0089] F fusion =α·F c +β·F s +λ·F j +γ·F

[0090] Among them, F fusion Represents the fused feature map, α, β, λ, and γ represent learnable parameters, which are set to 0.5, 0.5, 0.5, and 0.2 respectively. 3D convolution is performed on the fused feature map to obtain the enhanced feature map:

[0091] F out =F fusion +Conv(F fusion )

[0092] Among them, F out Represents the enhanced feature map.

[0093] S4. Use the generative adversarial network and the enhanced feature map to perform adversarial training on the backbone network, and use the trained network to enhance the collected fundus image to obtain the enhanced fundus image.

[0094] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0095] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, it should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0096] Example 2

[0097] In this embodiment, a fundus image enhancement system based on machine learning includes: an image preprocessing module, a feature extraction module, a feature fusion module and an image enhancement module.

[0098] The image preprocessing module is used to obtain an original fundus spectral image and preprocess the original fundus image to obtain a preprocessed spectral image.

[0099] The image preprocessing module includes: an image splitting unit, a normalization unit and a correction and denoising unit; the image splitting unit is used to split the original fundus spectral image according to bands to obtain band spectral images; the normalization unit normalizes each band spectral image to obtain a normalized image; the correction and denoising unit is used to perform spectral correction on the normalized image and perform noise suppression to obtain a preprocessed spectral image.

[0100] The feature extraction module is used to construct a backbone network based on a residual network and extract multi-scale features of the preprocessed image in combination with dilated convolution.

[0101] The feature extraction module includes: a network construction unit, a network improvement unit, a feature extraction unit and a first feature fusion unit; the network construction unit is based on the ResNet-34 model architecture, removes the fully connected layer of the model, retains 4 residual layers, and obtains the backbone network; the network improvement unit is used to introduce the void convolution into the third residual layer and the fourth residual layer of the backbone network to improve the backbone network; the feature extraction unit uses the first residual layer and the second residual layer to extract the low-level feature map, and uses the improved third residual layer and the improved fourth residual layer to extract the high-level feature map; the first feature fusion unit connects the low-level feature map and the high-level feature map through horizontal connection, and uses bilinear interpolation to upsample the high-level features to the same resolution as the low-level feature map, and fuses the low-level feature map with the upsampled high-level feature map to obtain multi-scale features.

[0102] The feature fusion module is used to introduce the channel-spatial-spectral multi-attention module into the backbone network and perform dual attention fusion on multi-scale features to obtain the enhanced feature map.

[0103] The feature fusion module includes: a channel attention calculation unit, a spatial attention calculation unit, a spectral attention calculation unit and a second feature fusion unit; the channel attention calculation unit is used to input the multi-scale features into the channel attention module, generate a channel description vector through global average pooling, and use a fully connected layer to learn the channel dependency through the channel description vector to obtain the channel attention weight; the spatial attention calculation unit is used to input the multi-scale features into the spatial attention module along the channel dimension, use maximum pooling and average pooling to generate a spatial feature map, splice the generated spatial feature map and pass it through the convolution layer to obtain the spatial attention weight; the spectral attention calculation unit is used to input the multi-scale features into the spectral attention module, perform spectral dimension analysis, calculate the similarity between different bands, and generate spectral attention weights; the second feature fusion unit is used to fuse the channel attention weights, spatial attention weights, spectral attention weights and multi-scale features, and perform 3D convolution refinement on the fusion results to obtain the enhanced feature map.

[0104] The image enhancement module uses the generative adversarial network and the enhanced feature map to perform adversarial training on the backbone network, and uses the trained network to enhance the collected fundus image to obtain the enhanced fundus image.

[0105] The system of the above embodiment is used to implement the corresponding machine learning-based fundus image enhancement method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0106] It should be noted that the above-mentioned fundus image enhancement system based on machine learning is embodied in the form of functional units. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0107] For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group of processors) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0108] Example 3

[0109] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the machine learning-based fundus image enhancement method described in any of the above embodiments is implemented.

[0110] Figure 2 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0111] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0112] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0113] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0114] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (e.g., USB (Universal Serial Bus), network cable, etc.) or a wireless method (e.g., mobile network, WIFI (Wireless Fidelity), Bluetooth, etc.).

[0115] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0116] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0117] The system of the above embodiment is used to implement the corresponding machine learning-based fundus image enhancement method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0118] Example 4

[0119] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the machine learning-based fundus image enhancement method as described in any of the above embodiments.

[0120] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0121] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the fundus image enhancement method based on machine learning as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0122] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0123] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0124] Although the present invention has been described in conjunction with specific embodiments of the present invention, many replacements, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0125] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0126] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A fundus image enhancement method based on machine learning, characterized in that: The method comprises: Acquiring an original fundus spectral image, and preprocessing the original fundus image to obtain a preprocessed spectral image; Constructing a backbone network based on a residual network and extracting multi-scale features of the preprocessed spectral image in combination with dilated convolution; Introducing a channel-spatial-spectral multi-attention module into the backbone network, and performing multi-attention fusion on the multi-scale features to obtain an enhanced feature map; The backbone network is trained adversarially using a generative adversarial network and the enhanced feature map, and the trained network is used to perform image enhancement on the collected fundus spectral image to obtain an enhanced fundus spectral image.

2. The fundus image enhancement method based on machine learning according to claim 1, characterized in that: The pretreatment method comprises: Splitting the original fundus spectral image according to wavelength bands to obtain wavelength band spectral images; Normalizing each of the band spectral images to obtain a normalized image; The normalized image is subjected to spectral correction and noise suppression to obtain a preprocessed spectral image.

3. The fundus image enhancement method based on machine learning according to claim 1, characterized in that: The method for extracting the multi-scale features includes: Based on the ResNet-34 model architecture, the fully connected layers of the model are removed and four residual layers are retained to obtain the backbone network. Introducing the dilated convolution into the third residual layer and the fourth residual layer of the backbone network to improve the backbone network; The first residual layer and the second residual layer are used to extract low-level feature maps, and the improved third residual layer and the improved fourth residual layer are used to extract high-level feature maps; The low-level feature map and the high-level feature map are connected through lateral connection, and the high-level features are upsampled to the same resolution as the low-level feature map using bilinear interpolation, and the low-level feature map is fused with the upsampled high-level feature map to obtain the multi-scale features.

4. The fundus image enhancement method based on machine learning according to claim 1, characterized in that: The method for obtaining the enhanced feature map includes: Input the multi-scale features into the channel attention module, generate a channel description vector through global average pooling, and use a fully connected layer to learn the inter-channel dependency through the channel description vector to obtain the channel attention weight; The multi-scale features are input into the spatial attention module along the channel dimension, and the spatial feature map is generated by using maximum pooling and average pooling. The generated spatial feature map is spliced ​​and passed through the convolution layer to obtain the spatial attention weight; The multi-scale features are input into the spectral attention module to analyze the spectral dimension, calculate the similarity between different bands, and generate spectral attention weights; The channel attention weight, the spatial attention weight, the spectral attention weight and the multi-scale features are fused, and the fusion result is refined by 3D convolution to obtain the enhanced feature map.

5. A fundus image enhancement system based on machine learning, the system being used to implement the method according to any one of claims 1 to 4, characterized in that: include: Image preprocessing module, feature extraction module, feature fusion module and image enhancement module; The image preprocessing module is used to obtain an original fundus spectral image and preprocess the original fundus image to obtain a preprocessed spectral image; The feature extraction module is used to construct a backbone network based on a residual network and extract multi-scale features of the preprocessed image in combination with dilated convolution; The feature fusion module is used to introduce a channel-space-spectral multi-attention module into the backbone network and perform dual-attention fusion on the multi-scale features to obtain an enhanced feature map; The image enhancement module uses the generative adversarial network and the enhanced feature map to perform adversarial training on the backbone network, and uses the trained network to perform image enhancement on the collected fundus spectral image to obtain an enhanced fundus spectral image.

6. The fundus image enhancement system based on machine learning according to claim 5, characterized in that: The image preprocessing module includes: an image splitting unit, a normalization unit and a correction and denoising unit; The image splitting unit is used to split the original fundus spectral image according to bands to obtain band spectral images; The normalization unit normalizes each of the band spectrum images to obtain a normalized image; The correction and denoising unit is used to perform spectral correction on the normalized image and perform noise suppression to obtain a preprocessed spectral image.

7. The fundus image enhancement system based on machine learning according to claim 5, characterized in that: The feature extraction module includes: a network construction unit, a network improvement unit, a feature extraction unit and a first feature fusion unit; The network construction unit is based on the ResNet-34 model architecture, removing the fully connected layer of the model and retaining 4 residual layers to obtain the backbone network; The network improvement unit is used to introduce the dilated convolution into the third residual layer and the fourth residual layer of the backbone network to improve the backbone network; The feature extraction unit extracts low-level feature maps using the first residual layer and the second residual layer, and extracts high-level feature maps using the improved third residual layer and the improved fourth residual layer; The first feature fusion unit connects the low-level feature map and the high-level feature map through a horizontal connection, upsamples the high-level features to the same resolution as the low-level feature map using bilinear interpolation, and fuses the low-level feature map with the upsampled high-level feature map to obtain the multi-scale features.

8. The fundus image enhancement system based on machine learning according to claim 5, characterized in that: The feature fusion module includes: a channel attention calculation unit, a spatial attention calculation unit, a spectral attention calculation unit and a second feature fusion unit; The channel attention calculation unit is used to input the multi-scale features into the channel attention module, generate a channel description vector through global average pooling, and use a fully connected layer to learn the inter-channel dependency through the channel description vector to obtain a channel attention weight; The spatial attention calculation unit is used to input the multi-scale features into the spatial attention module along the channel dimension, generate a spatial feature map using maximum pooling and average pooling, and splice the generated spatial feature map and pass it through the convolution layer to obtain the spatial attention weight; The spectral attention calculation unit is used to input the multi-scale features into the spectral attention module, perform spectral dimension analysis, calculate the similarity between different bands, and generate spectral attention weights; The second feature fusion unit is used to fuse the channel attention weight, the spatial attention weight, the spectral attention weight and the multi-scale feature, and perform 3D convolution refinement on the fusion result to obtain the enhanced feature map.

9. An electronic device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the fundus image enhancement method based on machine learning as described in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, the fundus image enhancement method based on machine learning as described in any one of claims 1 to 4 is implemented.

Images