Fundus image enhancement method based on multi-attention and generative adversarial network
Through the multi-attention and generative adversarial network method, combined with retinal artifact perception and medical structure perception module, the accuracy and effect problems of deep learning methods when processing fundus images are solved, achieving higher quality image enhancement.
Patent Information
- Application Number
- CN202411197507.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-13
AI Technical Summary
The existing fundus image enhancement method based on deep learning has low processing accuracy and poor effect when processing retinal artifacts and physiological structures such as blood vessels and optic trays generated by dust particles on the lens.
Using a multi-attention and generative adversarial network method, artifacts in fundus images are identified and enhanced through retinal artifact perception module, medical structure perception module, channel attention module, multi-scale fusion module and global-local discriminator. and medical structure to improve image clarity.
Improve the enhanced processing accuracy and effect of low-quality fundus images, and generate clearer images, solving the shortcomings of existing methods in dealing with retinal artifacts and physiological structures.
Smart Images
Figure CN119130831A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a fundus image enhancement method based on multi-attention and generative adversarial networks. Background Art
[0002] Among the existing methods for fundus image enhancement, the methods based on traditional image processing require manual parameter selection and generally lack good adaptability. In the process of rapid development of deep learning technology, more and more related image enhancement methods have been proposed based on deep learning, and the effect of image enhancement has become better and better. Due to the uniqueness of medical images, the enhancement methods of low-quality fundus images have certain limitations in the application of deep learning technology. First, it is difficult to obtain medical images. Therefore, the data set is relatively scarce and the amount of medical image data is small. At the same time, there is a lack of pairing of low-quality fundus images and high-quality fundus images. In addition, due to the clinical use of medical images, the enhanced images must retain their pathological feature information to avoid interference with clinical diagnosis. Among the image enhancement methods for low-quality images, the existing image enhancement methods based on deep learning still have room for improvement in the processing effect of retinal artifacts caused by dust particles on the lens and physiological structures such as blood vessels, optic cups and optic discs. Summary of the invention
[0003] The embodiment of the present invention provides a fundus image enhancement method based on multi-attention and generative adversarial network, so as to at least solve the technical problems of low-quality image enhancement methods, that is, the existing deep learning-based image enhancement methods have low processing accuracy and poor effect on retinal artifacts caused by dust particles on the lens and physiological structures such as blood vessels, optic cups and optic discs.
[0004] According to one aspect of an embodiment of the present invention, a fundus image enhancement method based on multi-attention and generative adversarial network is provided. The method may include: obtaining a number of fundus retinal images; dividing the fundus retinal images according to a preset ratio to obtain a training set and a test set of the fundus retinal images; inputting the training set and the test set of the fundus retinal images into a pre-constructed image enhancement network, training and testing the pre-constructed image enhancement network, and obtaining a successfully trained image enhancement network, wherein the pre-constructed image enhancement network includes a retinal artifact perception module, a medical structure perception module, a channel attention module, a multi-scale fusion module, a backbone network, and a global-local discriminator; inputting the fundus retinal image to be detected into the successfully trained image enhancement network to obtain a prediction probability map and a segmentation mask map corresponding to the fundus retinal image to be detected.
[0005] Optionally, the retinal artifact perception module is a small U-Net network, and the retinal artifact perception module is used to: extract low-level features and high-level features of the fundus retinal image through the encoder module and the decoder module respectively, and fuse the low-level features and the high-level features through the connection layer to obtain an initial segmentation feature map corresponding to the fundus retinal image; integrate the initial segmentation feature map corresponding to the fundus retinal image into the backbone network of the image enhancement network.
[0006] Optionally, the medical structure perception module is a network based on CE-Net, and the medical structure perception module is used to: extract low-level features and high-level features of the fundus retinal image through the encoder module and the decoder module respectively, and fuse the low-level features and the high-level features through the connection layer, wherein the encoder and the decoder are connected through the context extraction module; integrate the features of different depths extracted by the encoder module, the context extraction module and the decoder module into the backbone network of the image enhancement network; and output the segmentation mask image corresponding to the fundus retinal image in the last layer of the decoder.
[0007] Optionally, the multi-scale fusion module includes: performing feature fusion on two inputs of the multi-scale fusion module, passing the fused features through a global pooling layer, a 1×1 convolution layer, and a LeakyRelu layer in sequence, inputting the features passing through the LeakyRelu layer into two 1×1 convolution layers respectively, passing the features passing through the two 1×1 convolution layers through a softmax layer to obtain w1 and w2, performing element-by-element summation of w1 and the first input end of the multi-scale fusion module to obtain a first feature, performing element-by-element multiplication of w2 and the second input end of the multi-scale fusion module to obtain a second feature, and performing element-by-element summation of the first feature and the second feature to obtain a fusion feature of the first feature and the second feature.
[0008] Optionally, the global-local discriminator is used to: input the initial enhanced image corresponding to the fundus retinal image output by the backbone network into the global-local discriminator to obtain a predicted probability map corresponding to the fundus retinal image.
[0009] Optionally, before dividing the plurality of fundus retinal images according to a preset ratio to obtain a training set and a test set of the plurality of fundus retinal images, the method further includes: adjusting the size of the plurality of fundus retinal images to 512px×512px.
[0010] Optionally, when a training set and a test set of several fundus retinal images are input into a pre-constructed image enhancement network, and the pre-constructed image enhancement network is trained and tested, a batch size of 8 for each iteration of the pre-constructed image enhancement network is used for training and testing.
[0011] Beneficial effects of the present invention:
[0012] The present invention proposes a fundus image enhancement method based on multi-attention and generative adversarial network, uses a retinal artifact perception module to identify retinal artifacts, and uses a medical structure perception module based on CE-Net to perceive important clinical medical structures (such as blood vessels, optic cups and discs, etc.) in the fundus retina, and integrates this information into a backbone network; a channel attention module is used in the fusion process to capture effective feature information in each channel; on the traditional skip connection, a multi-scale feature fusion module is used to perform learnable selective fusion of high-level features and low-level features at different scales; finally, a global-local discriminator is used to discriminate the generated image, so that a pre-constructed image enhancement network can generate a clearer image, which solves the technical problem that the existing deep learning-based image enhancement method has low processing accuracy and poor effect for retinal artifacts generated by dust particles on the lens and physiological structures such as blood vessels, optic cups and discs, and achieves the image enhancement method of low-quality images. The pre-constructed image enhancement network is used to enhance the low-quality image, which improves the processing accuracy and good effect of image enhancement for low-quality images. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0014] Figure 1 is a flowchart of a fundus image enhancement method based on multi-attention and generative adversarial network according to an embodiment of the present invention;
[0015] Figure 2 is a schematic diagram of a pre-constructed image enhancement network according to an embodiment of the present invention;
[0016] Figure 3 is a schematic diagram of a channel attention module according to an embodiment of the present invention;
[0017] Figure 4 is a schematic diagram of a multi-scale feature fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0020] Example 1
[0021] According to an embodiment of the present invention, a fundus image enhancement method based on multi-attention and generative adversarial network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0022] Figure 1 is a flowchart of a fundus image enhancement method based on multi-attention and generative adversarial network according to an embodiment of the present invention, such as Figure 1 As shown, the method may include the following steps:
[0023] Step S101, obtaining a plurality of fundus retinal images.
[0024] In the technical solution provided in the above step S101 of the present invention, before performing the pre-constructed image enhancement network, a plurality of fundus retinal images are first acquired, wherein the plurality of fundus retinal images are low-quality fundus retinal images.
[0025] Step S102, dividing a plurality of fundus retinal images according to a preset ratio to obtain a training set and a test set of a plurality of fundus retinal images.
[0026] In the technical solution provided in the above step S102 of the present invention, a plurality of fundus retinal images are divided according to a preset ratio to obtain a training set and a test set of a plurality of fundus retinal images.
[0027] Step S103, inputting a training set and a test set of several fundus retinal images into a pre-constructed image enhancement network, training and testing the pre-constructed image enhancement network, and obtaining a successfully trained image enhancement network, wherein the pre-constructed image enhancement network includes a retinal artifact perception module, a medical structure perception module, a channel attention module, a multi-scale fusion module, a backbone network, and a global-local discriminator.
[0028] In the technical solution provided in step S103 of the present invention, the pre-constructed image enhancement network is trained and tested on a training set and a test set of a plurality of input fundus retinal images to obtain a successfully trained image enhancement network. Figure 2 is a schematic diagram of a pre-constructed image enhancement network according to an embodiment of the present invention. Figure 2 In the paper, the retinal artifact perception module is used to identify retinal artifacts, and the medical structure perception module based on CE-Net is used to perceive important clinical medical structures in the fundus retina (such as blood vessels, optic cups and discs, etc.), and this information is integrated into the backbone network. The channel attention module is used in the fusion process to capture effective feature information in each channel. On the traditional skip connection, a multi-scale feature fusion module is used to perform learnable selective fusion of high-level features and low-level features at different scales. Finally, a global-local discriminator is used to discriminate the generated image, driving the network to generate clearer images. The feature maps transmitted in deep neural networks usually have a large depth, but only some of the channel features may play a role in subsequent tasks, and some features will not contribute to the learning of subsequent network layers. Therefore, there is a need for an attention mechanism to extract useful information from the convolutional feature channel flow, while suppressing the propagation of invalid information in the network. Figure 3 is a schematic diagram of a channel attention module according to an embodiment of the present invention, such as Figure 3 As shown, the channel attention module is used to complete the adaptive learning and screening of different contribution features.
[0029] Step S104, inputting the fundus retinal image to be detected into the successfully trained image enhancement network to obtain a prediction probability map and a segmentation mask map corresponding to the fundus retinal image to be detected.
[0030] In the technical solution provided in the above step S104 of the present invention, the successfully trained image enhancement network processes the fundus retinal image to be detected to obtain a prediction probability map and a segmentation mask map corresponding to the fundus retinal image to be detected.
[0031] The above method of this embodiment is further introduced below.
[0032] As an optional implementation method, step S103, the retinal artifact perception module is a small U-Net network, and the retinal artifact perception module is used to: extract low-level features and high-level features of the fundus retinal image through the encoder module and the decoder module respectively, and fuse the low-level features and the high-level features through the connection layer to obtain an initial segmentation feature map corresponding to the fundus retinal image; integrate the initial segmentation feature map corresponding to the fundus retinal image into the backbone network of the image enhancement network.
[0033] In this embodiment, if Figure 2 As shown in the figure, in order to improve the network's ability to recognize and perceive retinal artifacts, a retinal artifact perception module is introduced. This module uses a miniaturized U-Net structure. Since artifacts caused by unwanted objects on camera lenses have similar appearances, a small network can be used to locate the shape and position of the defect, and the initial segmentation feature map corresponding to the fundus retinal image is integrated into the backbone network of the image enhancement network.
[0034] As an optional implementation method, step S103, the medical structure perception module is a network based on CE-Net, and the medical structure perception module is used to: extract low-level features and high-level features of the fundus retinal image through the encoder module and the decoder module respectively, and fuse the low-level features and the high-level features through the connection layer, wherein the encoder and the decoder are connected through the context extraction module; integrate the features of different depths extracted by the encoder module, the context extraction module and the decoder module into the backbone network of the image enhancement network; and output the segmentation mask image corresponding to the fundus retinal image at the last layer of the decoder.
[0035] In this embodiment, if Figure 2 As shown in the figure, in order to increase the focus on clinical medical structures, a medical structure perception module is introduced to learn retinal medical structures. CE-Net is used as the module network to predict the segmentation mask of the retinal medical physiological structure, and the mean square error loss function is used to constrain the prediction. The features extracted by the decoder parts of the network at different depths are integrated into the corresponding features of the backbone image correction network. The features extracted by the encoder module, the context extraction module, and the decoder module at different depths are integrated into the backbone network of the image enhancement network. The segmentation mask corresponding to the fundus retinal image is output at the last layer of the decoder.
[0036] As an optional implementation method, step S103, the multi-scale fusion module is used to: perform feature fusion on the two inputs of the multi-scale fusion module, pass the fused features through the global pooling layer, the 1×1 convolution layer, and the LeakyRelu layer in sequence, input the features passing through the LeakyRelu layer into two 1×1 convolution layers respectively, pass the features passing through the two 1×1 convolution layers through the softmax layer to obtain w1 and w2, sum w1 and the first input end of the multi-scale fusion module element by element to obtain the first feature, multiply w2 and the second input end of the multi-scale fusion module element by element to obtain the second feature, sum the first feature and the second feature element by element to obtain a fusion feature of the first feature and the second feature.
[0037] In this embodiment, Figure 4 is a schematic diagram of a multi-scale feature fusion module according to an embodiment of the present invention. Figure 4 In this paper, based on the SK convolution, a multi-scale feature fusion module is introduced to provide a self-attention mechanism to fuse features of different scales. This module can learn the weights of features of different scales, filter out more useful features from multi-scale features for fusion, and then extract more effective information.
[0038] As an optional implementation method, in step S103, the global-local discriminator is used to: input the initial enhanced image corresponding to the fundus retinal image output by the backbone network into the global-local discriminator to obtain a predicted probability map corresponding to the fundus retinal image.
[0039] In this embodiment, if Figure 2 As shown in the figure, the generative adversarial network is finally used to make the image after defect elimination have better visual effects. Since the input image may have a specific local area that needs to be enhanced, while other areas already have good visual effects and do not need to be processed. The global discriminator often cannot make a good judgment in this case. When the image is globally or mostly enhanced, the global discriminator can identify it well; when a specific area is enhanced, the global discriminator is difficult to adaptively adjust the local enhancement effect. In order to solve this problem, a global-local discriminator is used so that the discriminator can effectively respond to the enhancement of global defect factors and local defect factors. The global discriminator and the local discriminator use the PatchGAN network.
[0040] As an optional implementation manner, in step S101, before dividing the plurality of fundus retinal images according to a preset ratio to obtain a training set and a test set of the plurality of fundus retinal images, the method further includes: adjusting the size of the plurality of fundus retinal images to 512px×512px.
[0041] As an optional implementation method, in step S103, when a training set and a test set of several fundus retinal images are input into a pre-constructed image enhancement network, and the pre-constructed image enhancement network is trained and tested, the batch size of each iteration of the pre-constructed image enhancement network is 8.
[0042] Experimental part:
[0043] 1. Experimental Setup
[0044] The operating system used in the experiment is Ubuntu 20.04, the deep learning framework is Pytorch 1.10, the parallel computing architecture is CUDA 11.60, the CPU used is Inter Pentium G2020, and the GPU used is GeForce GTX 1080Ti.
[0045] 2. Model training
[0046] During the training process using the pre-built image enhancement network, the input degraded low-quality image will be resized to 512px×512px, and the batch size for each iteration is 8. First, the network model used by the retinal artifact perception module and the medical structure perception module is trained, and then the entire image correction network is trained end-to-end. The image correction network is optimized using stochastic gradient descent. The model is trained for a total of 300 cycles. In the first 150 iterations, the learning rate is initialized to 0.0001; then, in the next 150 cycles, the learning rate gradually decays to 0.
[0047] 3. Model Usage
[0048] After the pre-built image enhancement network is trained, the low-quality image is input into the pre-built image enhancement network to obtain the image enhancement result.
[0049] In an embodiment of the present invention, a plurality of fundus retinal images are obtained; the plurality of fundus retinal images are divided according to a preset ratio to obtain a plurality of training sets and test sets of the fundus retinal images; the plurality of training sets and test sets of the fundus retinal images are input into a pre-constructed image enhancement network, the pre-constructed image enhancement network is trained and tested, and a successfully trained image enhancement network is obtained, wherein the pre-constructed image enhancement network includes a retinal artifact perception module, a medical structure perception module, a channel attention module, a multi-scale fusion module, a backbone network, and a global-local discriminator; the fundus retinal image to be detected is input into the successfully trained image enhancement network to obtain a prediction probability map and a segmentation mask map corresponding to the fundus retinal image to be detected, thereby solving the technical problem that the existing deep learning-based image enhancement method in the image enhancement method of low-quality images has low processing accuracy and poor effect for retinal artifacts generated by dust particles on the lens and physiological structures such as blood vessels, optic cups and optic discs, and achieving the image enhancement method of low-quality images, using a pre-constructed image enhancement network to enhance the low-quality images, thereby improving the processing accuracy and good effect of image enhancement for low-quality images.
[0050] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0051] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0052] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0053] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0054] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0055] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A fundus image enhancement method based on multi-attention and generative adversarial network, characterized in that: include: Acquire several fundus retinal images; Dividing a plurality of fundus retinal images according to a preset ratio to obtain a training set and a test set of a plurality of fundus retinal images; Inputting a training set and a test set of several fundus retinal images into a pre-constructed image enhancement network, training and testing the pre-constructed image enhancement network, and obtaining a successfully trained image enhancement network, wherein the pre-constructed image enhancement network includes a retinal artifact perception module, a medical structure perception module, a channel attention module, a multi-scale fusion module, a backbone network, and a global-local discriminator; The fundus retinal image to be detected is input into the successfully trained image enhancement network to obtain the prediction probability map and segmentation mask map corresponding to the fundus retinal image to be detected.
2. The method according to claim 1, characterized in that The retinal artifact perception module is a small U-Net network. The retinal artifact perception module is used to: The encoder module and the decoder module are used to extract low-level features and high-level features of the fundus retinal image respectively, and the low-level features and high-level features are fused through the connection layer to obtain an initial segmentation feature map corresponding to the fundus retinal image; The initial segmentation feature map corresponding to the fundus retinal image is integrated into the backbone network of the image enhancement network.
3. The method according to claim 1, characterized in that The medical structure perception module is a network based on CE-Net. The medical structure perception module is used for: The low-level features and high-level features of the fundus retinal image are extracted through the encoder module and the decoder module respectively, and the low-level features and the high-level features are fused through the connection layer, wherein the encoder and the decoder are connected through the context extraction module; Integrate the features of different depths extracted by the encoder module, context extraction module and decoder module into the backbone network of the image enhancement network; The last layer of the decoder outputs the segmentation mask image corresponding to the fundus retinal image.
4. The method according to claim 1, characterized in that The multi-scale fusion module is used to: The two inputs of the multi-scale fusion module are fused, and the fused features are passed through the global pooling layer, the 1×1 convolution layer, and the LeakyRelu layer in sequence. The features passing through the LeakyRelu layer are input into two 1×1 convolution layers respectively, and the features passing through the two 1×1 convolution layers are passed through the softmax layer to obtain w1 and w2. The first feature is obtained by element-by-element summation of w1 and the first input end of the multi-scale fusion module, and the second feature is obtained by element-by-element multiplication of w2 and the second input end of the multi-scale fusion module. The first feature and the second feature are summed element-by-element to obtain the fusion feature of the first feature and the second feature.
5. The method according to claim 1, characterized in that The global-local discriminator is used to: The initial enhanced image corresponding to the fundus retinal image output by the backbone network is input into the global-local discriminator to obtain the predicted probability map corresponding to the fundus retinal image.
6. The method according to claim 1, characterized in that Before dividing the plurality of fundus retinal images according to a preset ratio to obtain a training set and a test set of the plurality of fundus retinal images, the method further includes: The size of several fundus retinal images was adjusted to 512px×512px.
7. The method according to claim 1, characterized in that When a training set and a test set of several fundus retinal images are input into a pre-built image enhancement network, and the pre-built image enhancement network is trained and tested, a batch size of each iteration is 8 when the pre-built image enhancement network is trained and tested.
8. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 7 when running.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.