Fundus Image Enhancement Method, System, Device and Medium Based on Feature Consistency

Abstract

The present invention discloses a fundus image enhancement method, system, device and medium based on feature consistency. The method includes: inputting a sequence of low-quality fundus images into a Laplacian pyramid to obtain multi-layer pyramid features of the sequence of low-quality fundus images; constructing an image enhancement network, and sequentially inputting each layer of pyramid features into the image enhancement network from large to small in terms of resolution to obtain a sequence of enhanced fundus images; constructing a first loss function through the sequence of enhanced fundus images; constructing a second loss function through the fused image features obtained by the pyramid features in the image enhancement network; calculating a final loss function through the first loss function and the second loss function, and supervising the learning of the image enhancement network by the final loss function until convergence. The present invention can extract key features of low-quality fundus images, effectively preserve and enhance key information of fundus images, and improve the effect of enhancing low-quality fundus images.

Description

Technical Field

[0001] The present invention relates to the technical field of fundus image enhancement, and in particular to a fundus image enhancement method, system, device and medium based on feature consistency. Background Art

[0002] Fundus color photographs are widely used in routine clinical examinations. By using the images taken, fundus structures such as the retina, optic disc, and macula can be observed, and ophthalmologists can diagnose various retinal diseases for patients. However, the process of taking fundus color photographs is easily restricted by shooting conditions, resulting in varying degrees of degradation of fundus color photographs, including image degradation caused by improper settings of fundus cameras, insufficient light, and eye movement during shooting. These degraded fundus images will affect the analysis of ophthalmologists and auxiliary analysis systems, leading to errors in diagnostic results.

[0003] Existing studies have proposed various methods for enhancing low-quality fundus images. These methods can be roughly divided into methods based on statistical histograms, based on filters, based on unpaired contrast learning, and based on domain adaptation. However, due to the large number and complexity of noise types in low-quality fundus images, it is not conducive to the design of methods based on statistical histograms and filters. And when using the above methods for image enhancement, although the areas with insufficient light can be improved and the contrast can be increased, there will be serious color distortion. The fundus structure features in fundus images are very crucial for clinical diagnosis. However, when training a model using the method based on unpaired contrast learning, it is easy for the model to ignore these key features, affecting clinical diagnosis and screening. The styles of fundus images in clinics are diverse, and it is difficult to adapt to all clinical noise styles using the domain adaptation algorithm. And its requirement for clinical data during the training process will lead to cumbersome deployment of the model in clinical applications. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. For this purpose, the present invention provides a fundus image enhancement method, system, device and medium based on feature consistency, which is easy to be deployed in clinical applications, can ensure both the reliability of enhancing low-quality fundus images and focus on extracting the key features of low-quality fundus images, and effectively maintain and enhance the key information of fundus images.

[0005] In a first aspect, an embodiment of the present invention provides a fundus image enhancement method based on feature consistency, including:

[0006] Obtain a sequence of low-quality fundus images;

[0007] Input the sequence of low-quality fundus images into a Laplacian pyramid to obtain multi-layer pyramid features of the sequence of low-quality fundus images;

[0008] Construct an image enhancement network, and input each layer of the pyramid features into the image enhancement network in order from the largest to the smallest resolution to obtain the fused image features of each layer of the pyramid features in the image enhancement network, and obtain an enhanced fundus image sequence according to the fused image features; construct a first loss function through the enhanced fundus image sequence; calculate the pooling features of the fused image features of each layer of the pyramid features in the image enhancement network, and construct a second loss function according to the pooling features;

[0009] Calculate the final loss function through the first loss function and the second loss function, and supervise the learning of the image enhancement network until convergence through the result of the final loss function to obtain the final image enhancement network; the final image enhancement network is used to perform image enhancement on low-quality fundus images.

[0010] According to the method of the embodiment of the present invention, it has at least the following beneficial effects:

[0011] This method first inputs a low-quality fundus image sequence into the Laplacian pyramid to obtain multiple layers of pyramid features. The Laplacian pyramid can more effectively preserve image features, and the Laplacian pyramid has low memory requirements and is easy to deploy; input the pyramid features into the image enhancement network to obtain an enhanced fundus image sequence and perform training on the image enhancement network, which can extract invariant features of low-quality fundus images; supervise the image enhancement network through the first loss function constructed by the enhanced fundus image sequence and the second loss function constructed by the mean of the pooling features until the image enhancement network converges to obtain the final image enhancement network. At this time, the obtained final image enhancement network effectively enhances the low-quality image and focuses on enhancing the key fundus information therein, and introduces the loss of the pooling features to guide the network to extract key features in the low-quality fundus images, improving the effect of enhancing low-quality fundus images.

[0012] According to some embodiments of the present invention, the image enhancement network is a U-shaped network, and the number of layers of the enhancement network is the same as the number of layers of the pyramid features.

[0013] According to some embodiments of the present invention, input each layer of the pyramid features into the image enhancement network in order from the largest to the smallest resolution, and obtain the enhanced fundus image sequence through the following method:

[0014] Input the first layer of the pyramid features into the corresponding first layer of the image enhancement network to obtain the image features extracted by the first layer of the image enhancement network;

[0015] Input the pyramid feature of the second layer into the corresponding image enhancement network of the second layer to obtain the image feature extracted by the image enhancement network of the second layer; and fuse the image feature extracted by the image enhancement network of the second layer and the image feature extracted by the image enhancement network of the first layer through downsampling to obtain the fused image feature output by the image enhancement network of the second layer;

[0016] Input the pyramid feature of the third layer into the corresponding image enhancement network of the third layer to obtain the image feature extracted by the image enhancement network of the third layer; and fuse the image feature extracted by the image enhancement network of the third layer and the fused image feature output by the image enhancement network of the second layer through downsampling to obtain the fused image feature output by the image enhancement network of the third layer;

[0017] And so on, until the pyramid feature of the L-th layer is input into the corresponding image enhancement network of the L-th layer to obtain the image feature extracted by the image enhancement network of the L-th layer; and fuse the image feature extracted by the image enhancement network of the L-th layer and the fused image feature output by the image enhancement network of the (L - 1)-th layer through downsampling to obtain the fused image feature output by the image enhancement network of the L-th layer; Upsample the fused image feature output by each image enhancement network in turn from the L-th layer until the upsampling of the fused image feature output by the image enhancement network of the first layer ends, to obtain the enhanced fundus image sequence, where the pyramid feature has a total of L layers.

[0018] According to some embodiments of the present invention, the fused image feature is obtained by the following formula:

[0019]

[0020] where, represents the fused image feature of the k-th image in the (l + 1)-th layer fused image feature output by the image enhancement network, Conv(·) represents the convolution operation, [·] represents the feature concatenation operation, represents the pyramid feature of the k-th image in the (l + 1)-th layer pyramid feature, represents the fused image feature of the k-th image in the l-th layer fused image feature output by the image enhancement network, and

[0021] According to some embodiments of the present invention, the first loss function is:

[0022]

[0023] where L N represents the first loss function, K represents the number of low-quality fundus images in the low-quality fundus image sequence, I represents the high-quality fundus image, denotes the enhanced fundus image obtained after enhancing the k-th low-quality fundus image.

[0024] According to some embodiments of the present invention, the formula for constructing the second loss function by using the mean of the pooling features corresponding to each layer of the pyramid features includes:

[0025]

[0026]

[0027] where, L C denotes the second loss function, denotes the consistency constraint loss of the fused image features of the l-th layer, denotes the pooling features after spatial pyramid pooling, denotes the mean of the pooling features of the fused image features of the low-quality image sequence at the l-th layer.

[0028] According to some embodiments of the present invention, the final loss function is calculated by the following formula:

[0029] L tatal = L N + λ C L C

[0030] where, L tatal denotes the final loss function, and λ C denotes the weight of the second loss function.

[0031] In a second aspect, an embodiment of the present invention provides a fundus image enhancement system based on feature consistency, and the fundus image enhancement system includes:

[0032] A low-quality fundus image acquisition module, configured to acquire a low-quality fundus image sequence;

[0033] A Laplacian pyramid module, configured to input the low-quality fundus image sequence into the Laplacian pyramid to obtain multi-layer pyramid features of the low-quality fundus image sequence;

[0034] An image enhancement network module, configured to construct an image enhancement network, input each layer of the pyramid features into the image enhancement network in descending order of resolution, obtain the fused image features of each layer of the pyramid features in the image enhancement network, and obtain an enhanced fundus image sequence according to the fused image features; construct a first loss function through the enhanced fundus image sequence; calculate the pooling features of the fused image features of each layer of the pyramid features in the image enhancement network, and construct a second loss function according to the pooling features;

[0035] The final image enhancement network module is used to calculate a final loss function through the first loss function and the second loss function, and supervise the learning of the image enhancement network through the final loss function until convergence to obtain a final image enhancement network; the final image enhancement network is used to perform image enhancement on low-quality fundus images.

[0036] In a third aspect, an embodiment of the present invention provides an electronic device, including at least one control processor and a memory communicatively connected to the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the feature-consistency-based fundus image enhancement method as described in the first aspect.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the feature-consistency-based fundus image enhancement method as described in the first aspect.

[0038] It should be noted that the beneficial effects of the second to fourth aspects of the present invention compared with the prior art are the same as those of the feature-consistency-based fundus image enhancement method in the first aspect, which will not be elaborated here.

[0039] Other features and advantages of the present invention will be described in the subsequent specification, and some will become obvious from the specification or be understood by implementing the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0041] Figure 1 is a flowchart of a feature-consistency-based fundus image enhancement method provided by an embodiment of the present invention;

[0042] Figure 2 is a flowchart of a method for obtaining a fused image provided by an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of a feature-consistency-based fundus image enhancement method provided by an embodiment of the present invention;

[0044] Figure 4 is a structural diagram of a feature-consistency-based fundus image enhancement system provided by an embodiment of the present invention;

[0045] Figure 5It is a structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0046] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0047] In the description of the present invention, if the first, second, etc. are described, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0048] In the description of the present invention, it should be understood that for the orientation description, such as up, down, etc., the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0049] In the description of the present invention, it should be noted that unless otherwise clearly defined, words such as setting, installation, connection, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0050] Refer to Figure 1 , in some embodiments of the present invention, a fundus image enhancement method based on feature consistency is provided. The fundus image enhancement method includes:

[0051] Step S100: Obtain a sequence of low-quality fundus images.

[0052] Step S200: Input the sequence of low-quality fundus images into a Laplacian pyramid to obtain multi-layer pyramid features of the sequence of low-quality fundus images.

[0053] Step S300: Construct an image enhancement network, and input each layer of pyramid features into the image enhancement network in descending order of resolution to obtain the fused image features of each layer of pyramid features in the image enhancement network, and obtain a sequence of enhanced fundus images according to the fused image features; construct a first loss function through the sequence of enhanced fundus images; calculate the pooled features of the fused image features of each layer of pyramid features in the image enhancement network, and construct a second loss function according to the pooled features.

[0054] Step S400: Calculate the final loss function through the first loss function and the second loss function, and supervise the learning of the image enhancement network by the result of the final loss function until convergence to obtain the final image enhancement network; the final image enhancement network is used to perform image enhancement on low-quality fundus images.

[0055] In step S200 of the embodiment of this method, first input the low-quality fundus image sequence into the Laplacian pyramid to obtain multi-layer pyramid features. The Laplacian pyramid can more effectively preserve image features, and has low memory requirements and is easy to deploy. Then, through step S300, input the pyramid features into the image enhancement network to obtain an enhanced fundus image sequence. Training the image enhancement network can extract invariant features of low-quality fundus images. Step S400 uses the final loss function obtained from the first loss function constructed by the enhanced fundus image sequence and the second loss function constructed by the mean of the pooled features to supervise the image enhancement network until the image enhancement network converges to obtain the final image enhancement network. At this time, the obtained final image enhancement network enables the enhanced image to effectively retain and enhance the key fundus information therein, and introduces the loss of the pooled features to guide the network to extract key features in low-quality fundus images, ultimately realizing the enhancement of low-quality fundus images.

[0056] In some embodiments of the present invention, the image enhancement network is a U-shaped network, and the number of layers of the enhancement network is the same as the number of layers of the pyramid features.

[0057] It should be noted that the input size and depth of each layer of the network are different. For example, when the image enhancement network is a 5-layer network, the input sizes and depths of the image enhancement network from shallow to deep are {256, 128, 64, 32, 16} and {64, 128, 256, 512, 1024} respectively. Here, no specific restrictions are imposed on the input size and depth of each layer of the network.

[0058] The U-shaped network has a contracting path for capturing context information and a symmetric expansion path for allowing precise localization, which enables the network to propagate context information to higher layer resolutions and adapt to multi-layer pyramid features, improving robustness.

[0059] Refer to Figure 2 , in some embodiments of the present invention, input each layer of pyramid features into the image enhancement network in order from largest to smallest resolution, and obtain the enhanced fundus image sequence in the following manner:

[0060] Step S301: Input the first layer of pyramid features into the corresponding first layer of the image enhancement network to obtain the image features extracted by the first layer of the image enhancement network.

[0061] Step S302: Input the second-layer pyramid features into the corresponding second-layer image enhancement network to obtain the image features extracted by the second-layer image enhancement network; and fuse the image features extracted by the second-layer image enhancement network and the image features extracted by the first-layer image enhancement network through downsampling to obtain the fused image features output by the second-layer image enhancement network.

[0062] Step S303: Input the third-layer pyramid features into the corresponding third-layer image enhancement network to obtain the image features extracted by the third-layer image enhancement network; and fuse the image features extracted by the third-layer image enhancement network and the fused image features output by the second-layer image enhancement network through downsampling to obtain the fused image features output by the third-layer image enhancement network.

[0063] Step S304: By analogy, until the L-layer pyramid features are input into the corresponding L-layer image enhancement network to obtain the image features extracted by the L-layer image enhancement network; and fuse the image features extracted by the L-layer image enhancement network and the fused image features output by the L-1 layer image enhancement network through downsampling to obtain the fused image features output by the L-layer image enhancement network; Upsample the fused image features output by each layer of the image enhancement network in turn from the L layer until the upsampling of the fused image features output by the first-layer image enhancement network ends, to obtain an enhanced fundus image sequence, where the pyramid features have a total of L layers.

[0064] It should be noted that for upsampling, a skip connection method is used. The L-layer image enhancement network directly performs upsampling. The L-1 layer first performs splicing of the fused image features and then performs upsampling until the upsampling of the first-layer image enhancement network is completed, to obtain an enhanced fundus image sequence.

[0065] Through the image enhancement network, feature extraction and fusion are performed from shallow to deep on the pyramid features, ensuring that the enhancement network can extract the features of low-quality fundus images as much as possible. And by extracting pyramid features with different resolutions, the robustness during feature extraction is ensured, and the error rate is reduced.

[0066] In some embodiments of the present invention, the fused image features are obtained through the following formula:

[0067]

[0068] Where represents the fused image feature of the k-th image in the (l + 1)-th layer of fused image features output by the image enhancement network, Conv(·) represents the convolution operation, [·] represents the feature splicing operation, represents the pyramid feature of the k-th image in the (l + 1)-th layer of pyramid features, represents the fused image feature of the k-th image of the l-th layer of fused image features output by the image enhancement network, and

[0069] The fused image is obtained by feature splicing and convolution, effectively maintaining and enhancing the key information of fundus images.

[0070] In some embodiments of the present invention, the first loss function is:

[0071]

[0072] where L N represents the first loss function, K represents the number of low-quality fundus images in the low-quality fundus image sequence, I represents the high-quality fundus image, represents the enhanced fundus image obtained after enhancing the k-th low-quality fundus image.

[0073] The first loss function calculates the loss using the content consistency implicit in the low-quality fundus image sequence, and learns low-quality image enhancement through the first loss function.

[0074] In some embodiments of the present invention, the formula for constructing the second loss function by the mean of the pooling features corresponding to each layer of pyramid features includes:

[0075]

[0076]

[0077] where L C represents the second loss function, represents the consistency constraint loss of the l-th layer of fused image features, represents the pooling feature after spatial pyramid pooling, represents the mean of the pooling features of the fused image features of the low-quality image sequence at the l-th layer.

[0078] The second loss function constructed by the mean of the pooling features corresponding to the pyramid features avoids the problem of over-strong feature constraints, calculates the consistency constraint loss of each layer of pyramid features using cosine similarity, and constrains the feature consistency and avoids the model being too sensitive to the magnitude of numerical values.

[0079] In some embodiments of the present invention, the final loss function is calculated by the following formula:

[0080] L tatal = L N + λ C L C

[0081] where Ltatal represents the final loss function, λ C represents the weight of the second loss function.

[0082] By calculating the first loss function and the second loss function to obtain the final loss function, and supervising the training of the entire image enhancement network through the final loss function, guiding the network to extract key features in low-quality fundus images, finally realizing the enhancement of low-quality fundus images in clinical scenarios and effectively maintaining and enhancing the key information of fundus images.

[0083] Referring to Figure 3 , for the convenience of those skilled in the art to understand, a specific embodiment of the present invention provides a clustering and integration method based on seed multi-features, including the following steps:

[0084] The first step: Collect a small number of high-quality fundus images for constructing a training set and a validation set. For each high-quality fundus image I, using a low-quality fundus image simulation model, according to the common image noises in fundus images, including blur, artifacts, and uneven illumination, perform random noise combination and random noise intensity selection to generate K simulated low-quality fundus images, so as to obtain a low-quality fundus image sequence D(I) = {I′ k | k = 1, 2,..., K}.

[0085] The second step: Based on the U-shaped network in deep learning, construct an image enhancement network N with 5 layers; use the Laplacian pyramid to decompose the low-quality fundus image into a linearly reversible multi-scale image pyramid; for the input image I, its Laplacian pyramid feature is P(I) = {p l | l = 0, 1,..., L}, where L = 4. According to the order from shallow to deep, the above-extracted pyramid feature P(I) is first used for feature splicing and convolution, and the features of each layer of P(I) are respectively fused and extracted with the corresponding features of the enhancement network N to form the input of the next layer of features of the enhancement network N; that is, the output of the l-th layer of the image enhancement network N is denoted as Then the feature output of the next layer of the image enhancement network N is:

[0086]

[0087] where, represents the fused image feature of the k-th image in the fused image feature of the (l + 1)-th layer output by the image enhancement network, Conv(·) represents the convolution operation, [·] represents the feature splicing operation, represents the pyramid feature of the k-th image in the (l + 1)-th layer pyramid feature, represents the fused image feature of the k-th image in the fused image feature of the l-th layer output by the image enhancement network, and

[0088] Step 3: Utilize the content consistency implicit in the low-quality fundus image sequence D(I), and use the first loss function to learn low-quality fundus image enhancement:

[0089]

[0090] Among them, L N represents the first loss function, K represents the number of low-quality fundus images in the low-quality fundus image sequence, I represents the high-quality fundus image, represents the enhanced fundus image obtained after enhancing the k-th low-quality fundus image;

[0091] Meanwhile, utilize the spatial pyramid pooling operation to transform the image features of the l-th layer into pooled features That is where σ(·) is the spatial pyramid pooling operation; then calculate the mean value of the pooled features of the l-th layer of the image sequence Calculate the between each degraded image at the l-th layer and

[0092]

[0093]

[0094] Among them, L C represents the second loss function, represents the consistency constraint loss of the fused image features of the l-th layer, represents the pooled features after spatial pyramid pooling, represents the mean value of the pooled features of the fused image features of the low-quality image sequence at the l-th layer.

[0095] Finally, calculate the final loss function:

[0096] L tatal = L N + λ C L C

[0097] Among them, L tatal represents the final loss function, λ C represents the weight of the second loss function.

[0098] Step 4: Supervise the learning of the image enhancement network through the result of the final loss function until convergence to obtain the final image enhancement network; perform image enhancement on the low-quality fundus images through the final image enhancement network.

[0099] Referring to Figure 4 , an embodiment of the present invention further provides a fundus image enhancement system based on feature consistency, including a low-quality fundus image acquisition module 1001, a Laplacian pyramid module 1002, an image enhancement network module 1003, and a final image enhancement network module 1004, where:

[0100] The low-quality fundus image acquisition module 1001 is used to acquire a sequence of low-quality fundus images.

[0101] The Laplacian pyramid module 1002 is used to input the sequence of low-quality fundus images into the Laplacian pyramid to obtain multi-layer pyramid features of the sequence of low-quality fundus images.

[0102] The image enhancement network module 1003 is used to construct an image enhancement network, input each layer of pyramid features into the image enhancement network in order from largest to smallest resolution, obtain the fused image features of each layer of pyramid features in the image enhancement network, and obtain a sequence of enhanced fundus images based on the fused image features; construct a first loss function through the sequence of enhanced fundus images; calculate the pooled features of the fused image features of each layer of pyramid features in the image enhancement network, and construct a second loss function based on the pooled features.

[0103] The final image enhancement network module 1004 is used to calculate the final loss function through the first loss function and the second loss function, supervise the learning of the image enhancement network through the final loss function until convergence to obtain the final image enhancement network; the final image enhancement network is used to perform image enhancement on the low-quality fundus images.

[0104] It should be noted that since the fundus image enhancement system based on feature consistency in this embodiment and the above-mentioned fundus image enhancement method based on feature consistency are based on the same inventive concept, the corresponding content in the method embodiment also applies to this device embodiment and will not be elaborated here.

[0105] Refer to Figure 5 , another embodiment of the present invention further provides an electronic device. The electronic device 6000 can be any type of intelligent terminal, such as a mobile phone, a tablet computer, a personal computer, etc.

[0106] Specifically, the electronic device 6000 includes one or more control processors 6001 and a memory 6002. Figure 5Taking a control processor 6001 and a memory 6002 as an example, the control processor 6001 and the memory 6002 can be connected through a bus or other means. Figure 5 Taking the connection through the bus as an example.

[0107] The memory 6002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to an electronic device in an embodiment of the present invention.

[0108] The control processor 6001 executes various functional applications and data processing of a fundus image enhancement method based on feature consistency by running the non-transitory software programs, instructions, and modules stored in the memory 6002, that is, implements a fundus image enhancement method based on feature consistency in the above method embodiment.

[0109] The memory 6002 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created by using a fundus image enhancement method based on feature consistency, etc. In addition, the memory 6002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 6002 may optionally include a memory remotely set relative to the control processor 6001, and these remote memories can be connected to the electronic device 6000 through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0110] When one or more modules are stored in the memory 6002 and executed by the one or more control processors 6001, a fundus image enhancement method based on feature consistency in the above method embodiment is executed, for example, executing the Figure 1 and Figure 2 method steps.

[0111] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0112] It should be noted that since an electronic device in this embodiment and the above-mentioned fundus image enhancement method based on feature consistency are based on the same inventive concept, the corresponding content in the method embodiment also applies to this device embodiment and will not be elaborated here.

[0113] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions for executing: the fundus image enhancement method based on feature consistency as in the above embodiment.

[0114] It should be noted that since a computer-readable storage medium in this embodiment and the above-mentioned fundus image enhancement method based on feature consistency are based on the same inventive concept, the corresponding content in the method embodiment also applies to this device embodiment and will not be elaborated here.

[0115] Those of ordinary skill in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and their appropriate combinations. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing data, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassettes, tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired data and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any data delivery medium.

[0116] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0117] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A fundus image enhancement method based on feature consistency, characterized in that The fundus image enhancement method includes: Obtaining a sequence of low-quality fundus images; Inputting the sequence of low-quality fundus images into a Laplacian pyramid to obtain multi-layer pyramid features of the sequence of low-quality fundus images; Constructing an image enhancement network, and inputting each layer of the pyramid features into the image enhancement network in order from the largest to the smallest resolution to obtain fused image features of each layer of the pyramid features in the image enhancement network, and obtaining a sequence of enhanced fundus images according to the fused image features; constructing a first loss function through the sequence of enhanced fundus images; calculating pooling features of the fused image features of each layer of the pyramid features in the image enhancement network, and constructing a second loss function according to the pooling features; Calculating a final loss function through the first loss function and the second loss function, and supervising the learning of the image enhancement network by the result of the final loss function until convergence to obtain a final image enhancement network; the final image enhancement network is used for enhancing the low-quality fundus images; The fused image features are obtained through the following formula: Among them, represents the fused image feature of the k-th image in the (l + 1)-th layer of fused image features output by the image enhancement network. Conv(·) represents the convolution operation, and [·] represents the feature concatenation operation. represents the pyramid feature of the k-th image in the (l + 1)-th layer of pyramid features. represents the fused image feature of the k-th image in the l-th layer of fused image features output by the image enhancement network, and The first loss function is: Among them, L N represents the first loss function, K represents the number of low-quality fundus images in the low-quality fundus image sequence, I represents the high-quality fundus image, represents the enhanced fundus image obtained after enhancing the k-th low-quality fundus image; The formula for constructing the second loss function through the mean value of the pooling features corresponding to each layer of the pyramid features includes: Among them, L C represents the second loss function, represents the consistency constraint loss of the fused image features in the l-th layer, represents the pooled features after spatial pyramid pooling, represents the mean value of the pooled features of the fused image features of the low-quality image sequence in the l-th layer, and L represents the total number of Laplacian pyramid features.

2. The method for enhancing fundus images based on feature consistency according to claim 1, characterized in that The image enhancement network is a U-shaped network, and the number of layers of the enhancement network is the same as the number of layers of the pyramid features.

3. The fundus image enhancement method based on feature consistency according to claim 2, characterized in that Inputting each layer of the pyramid features into the image enhancement network in order from the largest to the smallest resolution, and obtaining the sequence of enhanced fundus images through the following method: Inputting the first layer of the pyramid features into the corresponding first layer of the image enhancement network to obtain image features extracted by the first layer of the image enhancement network; Inputting the second layer of the pyramid features into the corresponding second layer of the image enhancement network to obtain image features extracted by the second layer of the image enhancement network; and fusing the image features extracted by the second layer of the image enhancement network with the image features extracted by the first layer of the image enhancement network through downsampling to obtain fused image features output by the second layer of the image enhancement network; Inputting the third layer of the pyramid features into the corresponding third layer of the image enhancement network to obtain image features extracted by the third layer of the image enhancement network; and fusing the image features extracted by the third layer of the image enhancement network with the fused image features output by the second layer of the image enhancement network through downsampling to obtain fused image features output by the third layer of the image enhancement network; And so on, until the Lth layer of the pyramid features is input into the corresponding Lth layer of the image enhancement network to obtain image features extracted by the Lth layer of the image enhancement network; And fusing the image features extracted by the Lth layer of the image enhancement network with the fused image features output by the (L-1)th layer of the image enhancement network through downsampling to obtain fused image features output by the Lth layer of the image enhancement network; The fused image features output by each layer of the image enhancement network are successively upsampled starting from the L-th layer until the upsampling of the fused image features output by the first layer of the image enhancement network is completed, obtaining the enhanced fundus image sequence, where the pyramid features have a total of L layers.

4. The fundus image enhancement method based on feature consistency according to claim 1, wherein The final loss function is calculated by the following formula: L tatal = L N + λ C L C Among them, L tatal represents the final loss function, and λ C represents the weight of the second loss function.

5. A fundus image enhancement system based on feature consistency, characterized in that, The fundus image enhancement system includes: A low-quality fundus image acquisition module, configured to acquire a low-quality fundus image sequence; A Laplacian pyramid module, configured to input the low-quality fundus image sequence into a Laplacian pyramid to obtain multi-layer pyramid features of the low-quality fundus image sequence; An image enhancement network module, configured to construct an image enhancement network, and input each layer of the pyramid features into the image enhancement network in order from largest to smallest resolution, obtaining the fused image features of each layer of the pyramid features in the image enhancement network, obtaining an enhanced fundus image sequence according to the fused image features; constructing a first loss function through the enhanced fundus image sequence; calculating the pooling features of the fused image features of each layer of the pyramid features in the image enhancement network, and constructing a second loss function according to the pooling features; A final image enhancement network module, configured to calculate a final loss function through the first loss function and the second loss function, supervise the learning of the image enhancement network through the final loss function until convergence, obtaining a final image enhancement network; the final image enhancement network is used to perform image enhancement on low-quality fundus images; the fused image features are obtained by the following formula: Among them, represents the fused image feature of the k-th image in the (l + 1)-th layer of fused image features output by the image enhancement network, Conv(·) represents the convolution operation, and [·] represents the feature concatenation operation. represents the pyramid feature of the k-th image in the (l + 1)-th layer of pyramid features. represents the fused image feature of the k-th image in the l-th layer of fused image features output by the image enhancement network, and The first loss function is: Among them, L N represents the first loss function, K represents the number of low-quality fundus images in the low-quality fundus image sequence, and I represents the high-quality fundus image. represents the enhanced fundus image obtained after enhancing the k-th low-quality fundus image; The formula for constructing the second loss function through the mean value of the pooling features corresponding to each layer of the pyramid features includes: Among them, L C represents the second loss function, represents the consistency constraint loss of the fused image features at the l-th layer, represents the pooled features after spatial pyramid pooling, represents the mean of the pooled features of the fused image features of the low-quality image sequence at the l-th layer, and L represents the total number of Laplacian pyramid features.

6. An electronic device, characterized in that: Including at least one control processor and a memory for communicatively connecting with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the feature-consistency-based fundus image enhancement method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the feature-consistency-based fundus image enhancement method according to any one of claims 1 to 5.

Images