Method, device, electronic device and readable storage medium for fundus image processing

Abstract

The present application relates to the field of image processing technology, and discloses a method, device, electronic device and readable storage medium for fundus image processing, the method comprising obtaining a fundus image to be processed; using a trained brightness enhancement model to perform brightness enhancement processing on the fundus image to be processed, and obtaining a processed target fundus image; wherein the brightness enhancement model is obtained based on training of a generative adversarial network and an image quality loss function. The embodiment method of the present application obtains a brightness enhancement model through training of a generative adversarial network and an image quality loss function, and can improve the image quality of the fundus image obtained by the brightness enhancement model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device and readable storage medium for fundus image processing. Background Art

[0002] With the development of Internet technology, people have higher and higher requirements for images. In some application scenarios, in order to meet user needs, it is usually necessary to enhance the brightness of images with lower brightness.

[0003] In the prior art, algorithms such as deep learning are usually used to enhance the brightness of fundus images to obtain brightened fundus images.

[0004] However, the fundus image obtained in this way usually has a lot of information loss, so that the brightened fundus image cannot meet the actual needs of users.

[0005] Therefore, when brightening the fundus image, how to reduce the information loss of the fundus image is a technical problem that needs to be solved. Summary of the invention

[0006] Embodiments of the present application provide a method, an apparatus, an electronic device, and a readable storage medium for processing fundus images, so as to reduce information loss of fundus images when brightening the fundus images.

[0007] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0008] Obtain the fundus image to be processed.

[0009] The trained brightness enhancement model is used to perform brightness enhancement processing on the fundus image to be processed to obtain the processed target fundus image; wherein the brightness enhancement model is obtained by training based on a generative adversarial network and an image quality loss function.

[0010] In the above implementation process, the trained brightness enhancement model is used to perform brightness enhancement processing on the fundus images to be processed. This method can perform brightness enhancement on batches of fundus images to be processed, thereby improving the image processing efficiency, reducing the workload of manual image processing, and saving the cost of image processing. In addition, the adversarial generative network is trained through the image quality loss function to obtain the brightness enhancement model, which enhances the accuracy of parameter adjustment during the model training process, thereby reducing the loss of texture detail features of the fundus images after brightness enhancement, and further improving the image quality of the fundus images after brightness enhancement.

[0011] In combination with the first aspect, in one implementation, before using the trained brightness enhancement model to perform brightness enhancement processing on the fundus image to be processed, the method further includes:

[0012] Get a collection of sample fundus images.

[0013] The sample fundus images in the sample fundus image set are input into the adversarial generative network to obtain the corresponding reconstructed sample fundus images.

[0014] The target loss function is used to determine the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images.

[0015] According to the overall loss value, the parameters in the adversarial generative network are adjusted to obtain a trained brightness enhancement model.

[0016] In the above implementation process, the adversarial generative network is trained through sample fundus images and target loss functions to obtain a trained brightness enhancement model, thereby realizing the training of the brightness enhancement model.

[0017] In combination with the first aspect, in one implementation, obtaining a sample fundus image set includes:

[0018] An initial image set is acquired, wherein the initial image set includes a first initial image set and a second initial image set, the first initial image set includes at least one first initial image, and the second initial image set includes at least one second initial image.

[0019] The brightness value of each first initial image in the first initial image set is within a first brightness value range, the brightness value of each second initial image in the second initial image set is within a second brightness value range, and the maximum brightness value in the first brightness value range is less than the minimum brightness value in the second brightness value range.

[0020] The black edges of each first initial image in the first initial image set are trimmed, features are extracted, and the size is adjusted to obtain a first preprocessed initial image set, wherein the first preprocessed initial image set includes the first preprocessed initial images corresponding to each first initial image.

[0021] The black edges of each second initial image in the second initial image set are trimmed, features are extracted, and the size is adjusted to obtain a second preprocessed initial image set, wherein the second preprocessed initial image set includes the second preprocessed initial images corresponding to each second initial image.

[0022] At least one first preprocessed initial image in the first preprocessed initial image set is randomly cropped and flipped to obtain a first sample fundus image set, wherein the first sample fundus image set includes a plurality of first sample fundus images.

[0023] At least one second preprocessed initial image in the second preprocessed initial image set is randomly cropped and flipped to obtain a second sample fundus image set, wherein the second sample fundus image set includes a plurality of second sample fundus images.

[0024] The first sample fundus image set and the second sample fundus image set are both sample fundus image sets, and the plurality of first sample fundus images and the plurality of second sample fundus images are both sample fundus images.

[0025] In the above implementation process, the acquired initial image is preprocessed and data expanded, thereby obtaining a sample fundus image for training a brightness enhancement model.

[0026] In combination with the first aspect, in one implementation, the generative adversarial network includes a first generator and a second generator.

[0027] Inputting a sample fundus image in the sample fundus image set into the adversarial generation network to obtain a corresponding reconstructed sample fundus image, including:

[0028] The first sample fundus image in the first sample fundus image set is input into the first generator to obtain a first reconstructed bright image.

[0029] Inputting the first reconstructed bright image into the second generator to obtain a first reconstructed dark image;

[0030] The second sample fundus image in the second sample fundus image set is input into the second generator to obtain a second reconstructed dark image.

[0031] Inputting the second reconstructed dark image into the first generator to obtain a second reconstructed bright image;

[0032] The first reconstructed dark image and the second reconstructed bright image are both reconstructed sample fundus images.

[0033] In the above implementation process, the first sample fundus image and the second sample fundus image are reconstructed respectively by the first generator and the second generator, so as to facilitate parameter adjustment of the first generator and the second generator according to the reconstructed image and the original sample fundus image, so as to train the first generator and the second generator to improve the accuracy of the brightness enhancement model.

[0034] In combination with the first aspect, in one implementation, the target loss function includes an adversarial loss function, a cycle consistency loss function, and an image quality loss function, and the adversarial generative network also includes a first discriminator and a second discriminator;

[0035] Using the target loss function, the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images is determined, including:

[0036] Using an image quality loss function, determine a first image quality loss value between a first sample fundus image in the first sample fundus image set and a corresponding first reconstructed dark image;

[0037] Using a cycle consistency loss function, determine a first cycle loss value between a first sample fundus image in the first sample fundus image set and a corresponding first reconstructed dark image;

[0038] Using a first discriminator and an adversarial loss function, determine a first discriminant value between a first sample fundus image and a second reconstructed dark image in a first sample fundus image set;

[0039] Using an image quality loss function, determine a second image quality loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image;

[0040] Using a cycle consistency loss function, determine a second cycle loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image;

[0041] Using a second discriminator and an adversarial loss function, determining a second discriminant value between a second sample fundus image in a second sample fundus image set and the first reconstructed bright image;

[0042] Among them, the overall loss value is the sum of the first image quality loss value, the first cycle loss value, the first discrimination value, the second image quality loss value, the second cycle loss value and the second discrimination value.

[0043] In the above implementation process, the overall loss value between the sample fundus image and the reconstructed image is calculated through the image quality loss function, the cycle consistency loss function and the adversarial loss function, so as to further adjust the parameters of the generative adversarial network according to the overall loss value, thereby improving the accuracy of the image generated by the generative adversarial network.

[0044] In combination with the first aspect, in one implementation, the target loss function is represented by formula (1):

[0045] L(G, F, D x , D y )=L GAN (G,D y ,X,Y)+L GAN (F,D x , Y, X)+λL cyc (G, F)+L identity (G, F)+L ssim (G, F) (1)

[0046] Among them, L ssim(G, F) = 1-|ssim(X, F(G(X)))| + 1-|ssim(Y, F(G(Y)))| = 2-|ssim(X, F(G(X)))| - |ssim(Y, F(G(Y)))|, where G represents the first generator, F represents the second generator, and D represents the x represents the first discriminator, D y represents the second discriminator, X represents the first sample fundus image, Y represents the second sample fundus image, L(G, F, D x , D y ) represents the target loss function, L GAN (G,D y , X, Y) represents the adversarial loss function between the first generator and the second discriminator, L GAN (F,D x , Y, X) represents the adversarial loss function between the second generator and the first discriminator, L cyc (G, F) represents the cycle consistency loss function between the first generator and the second generator, λ is the coefficient of the cycle consistency loss function, L identity (G, F) represents the nearly identical loss function between the first generator and the second generator, L ssim (G, F) represents the image quality loss function between the first generator and the second generator.

[0047] In combination with the first aspect, in one implementation, according to the overall loss value, the parameters in the adversarial generative network are adjusted to obtain a trained brightness enhancement model, including:

[0048] Determine whether the overall loss value meets the preset training conditions;

[0049] If yes, the first generator is determined as the trained brightness enhancement model;

[0050] If not, the parameters of the first generator are adjusted through the Ranger optimizer, and the parameters of the second generator are adjusted until the overall loss value meets the preset training conditions.

[0051] In the above implementation process, the parameters in the first generator and the second generator are adjusted by the optimizer, so as to enhance the stability of the generative adversarial network and improve the convergence speed of the generative adversarial network.

[0052] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0053] The acquisition module is used to acquire the fundus image to be processed.

[0054] The processing module is used to use the trained brightness enhancement model to perform brightness enhancement processing on the fundus image to be processed to obtain the processed target fundus image; wherein the brightness enhancement model is obtained by training the adversarial generation network based on the image quality loss function.

[0055] In conjunction with the second aspect, in one implementation, the processing module is further configured to:

[0056] Get a collection of sample fundus images.

[0057] The sample fundus images in the sample fundus image set are input into the adversarial generative network to obtain the corresponding reconstructed sample fundus images.

[0058] The target loss function is used to determine the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images.

[0059] According to the overall loss value, the parameters in the adversarial generative network are adjusted to obtain a trained brightness enhancement model.

[0060] In conjunction with the second aspect, in one implementation, the processing module is specifically configured to:

[0061] An initial image set is acquired, wherein the initial image set includes a first initial image set and a second initial image set, the first initial image set includes at least one first initial image, and the second initial image set includes at least one second initial image.

[0062] The brightness value of each first initial image in the first initial image set is within a first brightness value range, the brightness value of each second initial image in the second initial image set is within a second brightness value range, and the maximum brightness value in the first brightness value range is less than the minimum brightness value in the second brightness value range.

[0063] The black edges of each first initial image in the first initial image set are trimmed, features are extracted, and the size is adjusted to obtain a first preprocessed initial image set, wherein the first preprocessed initial image set includes the first preprocessed initial images corresponding to each first initial image.

[0064] The black edges of each second initial image in the second initial image set are trimmed, features are extracted, and the size is adjusted to obtain a second preprocessed initial image set, wherein the second preprocessed initial image set includes the second preprocessed initial images corresponding to each second initial image.

[0065] At least one first preprocessed initial image in the first preprocessed initial image set is randomly cropped and flipped to obtain a first sample fundus image set, wherein the first sample fundus image set includes a plurality of first sample fundus images.

[0066] At least one second preprocessed initial image in the second preprocessed initial image set is randomly cropped and flipped to obtain a second sample fundus image set, wherein the second sample fundus image set includes a plurality of second sample fundus images.

[0067] The first sample fundus image set and the second sample fundus image set are both sample fundus image sets, and the plurality of first sample fundus images and the plurality of second sample fundus images are both sample fundus images.

[0068] In combination with the second aspect, in one implementation, the generative adversarial network includes a first generator and a second generator.

[0069] The processing module is specifically used to: input the first sample fundus image in the first sample fundus image set into the first generator to obtain a first reconstructed bright image.

[0070] The first reconstructed bright image is input into the second generator to obtain the first reconstructed dark image.

[0071] The second sample fundus image in the second sample fundus image set is input into the second generator to obtain a second reconstructed dark image.

[0072] The second reconstructed dark image is input into the first generator to obtain the second reconstructed bright image.

[0073] The first reconstructed dark image and the second reconstructed bright image are both reconstructed sample fundus images.

[0074] In conjunction with the second aspect, in one implementation, the target loss function includes an adversarial loss function, a cycle consistency loss function, and an image quality loss function, and the adversarial generative network also includes a first discriminator and a second discriminator;

[0075] The processing module is specifically used for:

[0076] Using the target loss function, the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images is determined, including:

[0077] An image quality loss function is used to determine a first image quality loss value between a first sample fundus image in the first sample fundus image set and a corresponding first reconstructed dark image.

[0078] A cycle consistency loss function is used to determine a first cycle loss value between a first sample fundus image in the first sample fundus image set and a corresponding first reconstructed dark image.

[0079] A first discriminant and an adversarial loss function are used to determine a first discriminant value between a first sample fundus image and a second reconstructed dark image in a first sample fundus image set.

[0080] An image quality loss function is used to determine a second image quality loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image.

[0081] A cycle consistency loss function is used to determine a second cycle loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image.

[0082] A second discriminant and an adversarial loss function are used to determine a second discriminant value between a second sample fundus image in a second sample fundus image set and the first reconstructed bright image.

[0083] Among them, the overall loss value is the sum of the first image quality loss value, the first cycle loss value, the first discrimination value, the second image quality loss value, the second cycle loss value and the second discrimination value.

[0084] In conjunction with the second aspect, in one implementation, the target loss function is represented by formula (1):

[0085] L(G, F, D x , D y )=L GAN (G,D y ,X,Y)+L GAN (F,D x , Y, X)+λL cyc (G, F)+L identity (G, F)+L ssim (G, F) (1)

[0086] Among them, L ssim (G, F)=1-|ssim(X, F(G(X)))|+1-|ssim(Y, F(G(Y)))|=2-|ssim(X, F(G(X)))|-|ssim(Y, F(G(Y)))|, G represents the first generator, F represents the second generator, D x represents the first discriminator, D y represents the second discriminator, X represents the first sample fundus image, Y represents the second sample fundus image, L(G, F, D x , D y ) represents the objective loss function, L GAN (G,D y , X, Y) represents the adversarial loss function between the first generator and the second discriminator, L GAN (F,D x , Y, X) represents the adversarial loss function between the second generator and the first discriminator, L cyc(G, F) represents the cycle consistency loss function between the first generator and the second generator, λ is the coefficient of the cycle consistency loss function, L identity (G, F) represents the nearly identical loss function between the first generator and the second generator, L ssim (G, F) represents the image quality loss function between the first generator and the second generator.

[0087] In conjunction with the second aspect, in one implementation, the processing module is specifically configured to:

[0088] Determine whether the overall loss value meets the preset training conditions.

[0089] If so, the first generator is determined to be the trained brightness enhancement model.

[0090] If not, the parameters of the first generator are adjusted through the Ranger optimizer, and the parameters of the second generator are adjusted until the overall loss value meets the preset training conditions.

[0091] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0092] A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the method provided by any implementation of the first aspect above.

[0093] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method provided in any implementation of the first aspect described above are executed.

[0094] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0096] Figure 1 A flowchart of a method for processing fundus images provided in an embodiment of the present application;

[0097] Figure 2 A brightness enhancement model training flow chart provided in an embodiment of the present application;

[0098] Figure 3 A schematic diagram of a generative adversarial network structure provided in an embodiment of the present application;

[0099] Figure 4 A schematic diagram of fundus image brightness enhancement provided in an embodiment of the present application;

[0100] Figure 5 A structural block diagram of a fundus image processing device provided in an embodiment of the present application;

[0101] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0102] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0103] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0104] First, some terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0105] Terminal device: can be a mobile terminal, a fixed terminal or a portable terminal, such as a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system device, a personal navigation device, a personal digital assistant, an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the terminal device can support any type of interface for the user (such as a wearable device), etc.

[0106] Server: It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0107] The fundus is composed of the retina, fundus blood vessels, optic nerve head, optic nerve fibers, macula on the retina, and choroid behind the retina. Fundus images are images of the fundus area acquired using a fundus camera. In the medical field, fundus images are usually used to screen and detect fundus lesions. However, due to retinal pathology or imaging configuration, some fundus images acquired by fundus cameras have very low brightness. These fundus images cannot accurately screen and detect fundus lesions, affecting the accuracy of fundus lesion detection.

[0108] In the related art, the brightness of fundus images is enhanced through a deep learning network model. However, due to the low accuracy of training parameters during the training of the deep learning network model, the resolution of the generated brightness-enhanced fundus images is low, which affects the accurate screening and detection of fundus lesions.

[0109] Therefore, improving the parameter training accuracy of the deep learning network model so as to obtain high-resolution fundus images through the trained deep learning network model is a problem that needs to be solved.

[0110] See also Figure 1 , Figure 1 A flowchart of a method for fundus image processing provided in an embodiment of the present application. In the embodiment of the present application, the executor of the method may be an electronic device. Optionally, the electronic device may be a server or a terminal device, but the present application is not limited thereto.

[0111] As an example, Figure 1 The specific implementation process of the method shown is as follows:

[0112] Step 101: Acquire a fundus image to be processed.

[0113] Specifically, when executing step 101, the following steps may be adopted:

[0114] Step 1: Obtain the original fundus image to be processed.

[0115] As an embodiment, the original fundus image to be processed is an original low-brightness fundus image, and the electronic device further acquires the original low-brightness fundus image.

[0116] Step 2: Cut the black edges, extract features and adjust the size of the original fundus image to be processed to obtain the fundus image to be processed.

[0117] Furthermore, the original fundus image to be processed is preprocessed to obtain the fundus image to be processed.

[0118] Specifically, the black edges of the original low-brightness fundus image are trimmed, features are extracted, and the size is adjusted to obtain a preprocessed low-brightness fundus image, that is, the fundus image to be processed is the preprocessed low-brightness fundus image.

[0119] It should be noted that feature extraction may be performed by adding a field of view mask to an image, or by extracting structural features of an image, which is not limited here.

[0120] In the above implementation process, the original fundus image is preprocessed by cutting the black edges, extracting features, and adjusting the size of the original fundus image, thereby improving the resolution of the generated brightness-enhanced image.

[0121] Step 102: Use the trained brightness enhancement model to perform brightness enhancement processing on the fundus image to be processed to obtain a processed target fundus image.

[0122] Among them, the brightness enhancement model is obtained by training the adversarial generation network based on the image quality loss function.

[0123] Specifically, the image quality loss function is represented by formula (2):

[0124] ssim loss(x,y)=1-SSIM(x,y) (2)

[0125] Where SSIM(x, y) = [l(x, y) α *c(x,y) β *s(x,y) γ], x represents the sample fundus image, y represents the reconstructed sample fundus image corresponding to the sample fundus image, ssim loss(x, y) represents the image quality loss value of the image quality loss function, l(x, y) represents the brightness difference between the sample fundus image and the corresponding reconstructed sample fundus image, c(x, y) represents the contrast difference between the sample fundus image and the corresponding reconstructed sample fundus image, s(x, y) represents the feature difference between the sample fundus image and the corresponding reconstructed sample fundus image, α, β and γ represent the exponents of brightness difference, contrast difference and feature difference respectively, and α, β and γ are arbitrary constants.

[0126] The brightness difference between the sample fundus image x and the corresponding reconstructed sample fundus image y is expressed by formula (3):

[0127]

[0128] Wherein, l(x, y) represents the brightness difference between the sample fundus image x and the corresponding reconstructed sample fundus image y; μ x represents the average grayscale of the sample fundus image x, μ y represents the average grayscale of the reconstructed sample fundus image y corresponding to the sample fundus image; c1 represents a constant for maintaining stability, for example, c1 = (k1L) 2 , k1 can be 0.01 or 0.02, and there is no limitation here. L represents the dynamic range of pixel values.

[0129] The contrast difference between the sample fundus image x and the corresponding reconstructed sample fundus image y is expressed by formula (4):

[0130]

[0131] Where c(x, y) represents the contrast difference between the sample fundus image x and the corresponding reconstructed sample fundus image y; σ x represents the standard deviation of each pixel value of the sample fundus image x, σ y represents the standard deviation of each pixel value of the reconstructed sample fundus image y; c2 represents a constant to maintain stability, for example, c2 = (k2L) 2 , k2 can be 0.02 or 0.03, and there is no limitation here. L represents the dynamic range of pixel values.

[0132] The first characteristic difference between the sample fundus image x and the corresponding reconstructed sample fundus image y is expressed by formula (5):

[0133]

[0134] Where s(x, y) represents the characteristic difference between the sample fundus image x and the corresponding reconstructed sample fundus image y; σ xy represents the pixel covariance between the sample fundus image x and the corresponding reconstructed sample fundus image y, and c3 is a constant. For example, the value of c3 can be:

[0135] Specifically, before executing step 102, a trained brightness enhancement model is first obtained, such as Figure 2 As shown, Figure 2 A flowchart of a brightness enhancement model training is provided in an embodiment of the present application. Specifically, when training the brightness enhancement model, the following steps can be adopted:

[0136] S1021: Acquire a set of sample fundus images.

[0137] Specifically, when executing S1021, the following steps may be adopted:

[0138] Step a: Get an initial image set.

[0139] The initial image set includes a first initial image set and a second initial image set, the first initial image set includes at least one first initial image, and the second initial image set includes at least one second initial image.

[0140] The brightness value of each first initial image in the first initial image set is within a first brightness value range, the brightness value of each second initial image in the second initial image set is within a second brightness value range, and the maximum brightness value in the first brightness value range is less than the minimum brightness value in the second brightness value range.

[0141] Specifically, multiple fundus images are collected, and the multiple fundus images are screened according to a first brightness value range and a second brightness value range to obtain a low-brightness fundus image set and a high-brightness fundus image set, wherein the low-brightness fundus image set contains at least one low-brightness fundus image, and the high-brightness fundus image set contains at least one high-brightness fundus image, and the low-brightness fundus image set and the high-brightness fundus image set constitute an initial image set.

[0142] As an embodiment, the first brightness value range is [25, 40], the second brightness value range is [85, 100], the brightness value of each fundus image in the multiple fundus images collected is determined, the fundus images whose brightness values ​​are within the first brightness value range are taken as images in the low-brightness fundus image set, the fundus images whose brightness values ​​are within the second brightness value range are taken as images in the high-brightness fundus image set, and the fundus images whose brightness values ​​are neither within the first brightness value range nor within the second brightness value range are discarded, thereby forming a low-brightness fundus image set and a high-brightness fundus image set.

[0143] That is, the low-brightness fundus image set is the first initial image set, and the high-brightness fundus image set is the second initial image set.

[0144] It should be noted that in the embodiments of the present application, only the first brightness value range of [25, 40] and the second brightness value range of [85, 100] are used as examples for illustration. In actual applications, the first brightness value range can also be [30, 60] and the second brightness value range can also be [70, 90]. The first brightness value range and the second brightness value range can be set according to actual conditions and are not limited here.

[0145] In the above implementation process, the multiple collected fundus images are screened according to the first brightness value range and the second brightness value range to obtain a low-brightness fundus image set and a high-brightness fundus image set, thereby achieving classification of the fundus images.

[0146] Step b: perform black border trimming, feature extraction and size adjustment on each first initial image in the first initial image set to obtain a first preprocessed initial image set, wherein the first preprocessed initial image set includes first preprocessed initial images corresponding to each first initial image; perform black border trimming, feature extraction and size adjustment on each second initial image in the second initial image set to obtain a second preprocessed initial image set, wherein the second preprocessed initial image set includes second preprocessed initial images corresponding to each second initial image.

[0147] Furthermore, the images in the first initial image set and the images in the second initial image set are preprocessed respectively.

[0148] Specifically, the black edges of each low-brightness fundus image in the low-brightness fundus image set are clipped, features are extracted, and the size is adjusted to generate a preprocessed low-brightness fundus image set; the black edges of each high-brightness fundus image in the high-brightness fundus image set are clipped, features are extracted, and the size is adjusted to generate a preprocessed high-brightness fundus image set.

[0149] Step c: randomly cropping and flipping at least one first preprocessed initial image in the first preprocessed initial image set to obtain a first sample fundus image set, wherein the first sample fundus image set includes multiple first sample fundus images; randomly cropping and flipping at least one second preprocessed initial image in the second preprocessed initial image set to obtain a second sample fundus image set, wherein the second sample fundus image set includes multiple second sample fundus images.

[0150] The first sample fundus image set and the second sample fundus image set are both sample fundus image sets, and the plurality of first sample fundus images and the plurality of second sample fundus images are both sample fundus images.

[0151] Furthermore, data expansion is performed on the preprocessed low-brightness fundus image and high-brightness fundus image respectively.

[0152] As an embodiment, the images in the preprocessed low-brightness fundus image set are randomly cropped and flipped to generate a low-brightness fundus image sample set; the images in the preprocessed high-brightness fundus image set are randomly cropped and flipped to generate a high-brightness fundus image sample set.

[0153] Among them, the low-brightness fundus sample fundus images in the low-brightness fundus image sample set and the high-brightness fundus sample fundus images in the high-brightness fundus image sample set are both sample fundus images for training the brightness enhancement model.

[0154] It should be noted that, in the embodiments of the present application, only random cropping and flipping processing are used as the data expansion method for explanation.

[0155] As an embodiment, if the size of the fundus images in the preprocessed low-brightness fundus image set or the size of the images in the preprocessed high-brightness fundus image set does not meet the size after random cropping, it is necessary to randomly crop the corresponding images so that the corresponding images meet the cropped size. Furthermore, after randomly cropping the images in the preprocessed low-brightness fundus image set or the fundus images in the preprocessed high-brightness fundus image set, the randomly cropped images can also be flipped to obtain low-brightness sample fundus images. Alternatively, depending on the actual application, the randomly cropped images may not be flipped and the cropped images can be directly used as sample fundus images.

[0156] As an embodiment, if the size of the images in the preprocessed low-brightness fundus image set or the images in the preprocessed high-brightness fundus image set meets the size after random cropping, the images in the preprocessed low-brightness fundus image set that meet the size after random cropping and the images in the preprocessed high-brightness fundus image set that meet the size after random cropping can be directly flipped to obtain sample fundus images, or the images that meet the size after random cropping can be directly used as sample fundus images according to actual applications.

[0157] It should be noted that the number of images in the high-brightness fundus image set after data expansion is the same as the number of images in the low-brightness fundus image set.

[0158] It should be noted that the size of the fundus image after random cropping is 1 / 4 of the size of the original fundus image. The size after random cropping can also be set to other sizes according to actual applications, and there is no limitation here.

[0159] In the above implementation process, the acquired initial image is preprocessed and data expanded, thereby obtaining a sample fundus image for training a brightness enhancement model.

[0160] S1022: Inputting a sample fundus image in the sample fundus image set into a generative adversarial network to obtain a corresponding reconstructed sample fundus image.

[0161] like Figure 3 As shown, Figure 3 A schematic diagram of a generative adversarial network structure provided in an embodiment of the present application is shown in FIG. Figure 3 The adversarial generation network shown in includes a first generator 303, a second generator 306, a first discriminator 305, and a second discriminator 310.

[0162] As an embodiment, the first generator is an initial brightness enhancement generator, and the second generator is an initial brightness reduction generator.

[0163] It should be noted that in the embodiment of the present application, the network structure of the first generator and the second generator is an attention mechanism adversarial generation network (ResBlock+CBAM).

[0164] Specifically, when executing S1022, the following steps may be adopted:

[0165] Step 1: Input the first sample fundus image in the first sample fundus image set into the first generator to obtain a first reconstructed bright image; input the second sample fundus image in the second sample fundus image set into the second generator to obtain a second reconstructed dark image.

[0166] Step 2: Input the first reconstructed bright image into the second generator to obtain the first reconstructed dark image; input the second reconstructed dark image into the first generator to obtain the second reconstructed bright image.

[0167] The first reconstructed dark image and the second reconstructed bright image are both reconstructed sample fundus images.

[0168] As an example, Figure 3 As shown, a low-brightness sample fundus image 301 in a low-brightness fundus image sample set is input into a first generator 303 to obtain a first reconstructed bright image 304, and a high-brightness sample fundus image 309 in a high-brightness fundus image sample set is input into a second generator 306 to obtain a second reconstructed dark image 308.

[0169] The first reconstructed bright image 304 is input into the second generator 306 to obtain the first reconstructed dark image 307 , and the second reconstructed dark image 308 is input into the first generator 303 to obtain the second reconstructed bright image 302 .

[0170] In the above implementation process, the low-brightness fundus sample fundus image and the high-brightness fundus sample fundus image are reconstructed by the first generator and the second generator respectively, so as to facilitate parameter adjustment of the first generator and the second generator according to the reconstructed image and the original sample fundus image, so as to train the first generator and the second generator.

[0171] S1023: Determine the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images using the target loss function.

[0172] Specifically, the target loss function is represented by formula (1):

[0173] L(G, F, D x , D y )=L GAN (G,D y ,X,Y)+L GAN (F,D x , Y, X)+λL cyc (G, F)+L identity (G, F)+L ssim (G, F) (1)

[0174] Among them, L ssim (G, F) = 1-| ssim (X, F(G(X)))|+1-|ssim(Y, F(G(Y)))|=2-|ssim(X, F(G(X)))|-|ssim(Y, F(G(Y)))|, G represents the first generator, F represents the second generator, D x represents the first discriminator, D y represents the second discriminator, X represents the first sample fundus image, Y represents the second sample fundus image, and λ is L(G, F, D x , D y ) represents the target loss function, L GAN (G,D y , X, Y) represents the adversarial loss function between the first generator and the second discriminator, L GAN (F,D x , Y, X) represents the adversarial loss function between the second generator and the first discriminator, L cyc (G, F) represents the cycle consistency loss function between the first generator and the second generator, λ is the coefficient of the cycle consistency loss function, L identity (G, F) represents the nearly identical loss function between the first generator and the second generator, L ssim (G, F) represents the image quality loss function between the first generator and the second generator.

[0175] Among them, LGAN (G,D y , X, Y) is to generate a more realistic image, L GAN (G,D y , X, Y) = E y～Pdata(y) [logD y (y)+E x～Pdata(x) [log(1-D y (G(x)),L cyc (G, F) = E x～Pdata(x) [||F(G(x)-x)||1]+E y～Pdata(y) [||G(F(y)-y)||1], G's input is x, which is the first generator used to generate the fake image y, and F's input is y, which is the second generator used to generate the fake image x. When x is sent to G, a fake y image is obtained, and this fake y image is sent to F to obtain a more fake x image. Ideally, the more fake x image should be almost the same as the original x image. L identity (G, F) = E y～Pdata(y) [||G(F(y)-y)||1]+||F(x)-x)||1], L identity (G, F) is to use the first generator G to generate y-style images. Then, if y is sent into G, y should still be generated. Only in this way can it be proved that G has the ability to generate y-style images.

[0176] Among them, the target loss function includes an adversarial loss function, a cycle consistency loss function and an image quality loss function, and the adversarial generation network also includes a first discriminator and a second discriminator.

[0177] As an embodiment, the target loss function may also be an adversarial loss function, a cycle consistency loss function, an image quality loss function, and a nearly identical loss function.

[0178] It should be noted that the first discriminator and the second discriminator in the embodiment of the present application can be a Markov discriminator (PatchGAN).

[0179] Specifically, when executing S1023, the following steps may be adopted:

[0180] Step 1: Use the image quality loss function to determine the first image quality loss value between the first sample fundus image in the first sample fundus image set and the corresponding first reconstructed dark image, and use the image quality loss function to determine the second image quality loss value between the second sample fundus image in the second sample fundus image set and the corresponding second reconstructed bright image.

[0181] Specifically, the first brightness difference between the low-brightness sample fundus image 301 and the corresponding first reconstructed dark image 307 is determined according to the average grayscale of the low-brightness sample fundus image 301 and the average grayscale of the corresponding first reconstructed dark image 307. The first contrast difference between the low-brightness sample fundus image 301 and the corresponding first reconstructed dark image 307 is determined according to the standard deviation of each pixel value of the low-brightness sample fundus image 301 and the standard deviation of each pixel value of the corresponding first reconstructed dark image 307. The first characteristic difference between the low-brightness sample fundus image 301 and the corresponding first reconstructed dark image 307 is determined according to the covariance of the low-brightness sample fundus image 301 and the corresponding first reconstructed dark image 307.

[0182] A first image quality loss value is determined according to the first brightness difference value, the first contrast difference value, and the first feature difference value.

[0183] As an embodiment, if the low-brightness sample fundus image is x1, the first reconstructed dark image corresponding to the low-brightness sample fundus image x1 is y1.

[0184] Then the first brightness difference between the low brightness sample fundus image x1 and the corresponding first reconstructed dark image y1 is expressed as:

[0185]

[0186] Wherein, l(x1, y1) represents the first brightness difference between the low brightness sample fundus image x1 and the corresponding first reconstructed dark image y1; represents the average grayscale of the low-brightness sample fundus image x1, represents the average grayscale of the first reconstructed dark image y1 corresponding to the low-brightness sample fundus image.

[0187] The first contrast difference between the low-brightness sample fundus image and the corresponding first reconstructed dark image is expressed as:

[0188]

[0189] Wherein, c(x1, y1) represents the first contrast difference between the low-brightness sample fundus image x1 and the corresponding first reconstructed dark image y1; represents the standard deviation of each pixel value of the low-brightness sample fundus image x1, represents the standard deviation of each pixel value of the first reconstructed dark image y1 corresponding to the low-brightness sample fundus image.

[0190] The first characteristic difference between the low-brightness sample fundus image and the corresponding first reconstructed dark image is expressed as:

[0191]

[0192] Wherein, s(x1, y1) represents the first characteristic difference between the low-brightness sample fundus image x1 and the first reconstructed dark image y1; represents the pixel covariance between the low-brightness sample fundus image x1 and the corresponding first reconstructed dark image y1, c3 is a constant, for example, the value of c3 can be:

[0193] Furthermore, the first image quality loss value 311 between the low-brightness sample fundus image 301 and the corresponding first reconstructed dark image 307 is:

[0194] ssim loss(x1,y1)=1-SSIM(x1,y1) (9)

[0195] Among them, SSIM(x1,y1)=[l(x1,y1) α *c(x1,y1) β *s(x1,y1) γ ], α, β and γ represent the exponents of the first brightness, the first contrast and the first characteristic difference respectively, α, β and γ can be arbitrary constants, when α=β=γ, SSIM(x1,y1)=[l(x1,y1)*c(x1,y1)*s(x1,y1)]; ssim loss(x1,y1) represents the image quality loss value of the first image quality loss function.

[0196] Similarly, the process of using the image quality loss function to determine the second image quality loss value between the high-brightness sample fundus image 309 in the high-brightness fundus image sample set and the corresponding second reconstructed dark image 302 is as follows:

[0197] The second brightness difference between the high-brightness sample fundus image 309 and the second reconstructed dark image 302 corresponding to the high-brightness sample fundus image 309 is determined according to the average grayscale of the high-brightness sample fundus image 309 and the average grayscale of the corresponding second reconstructed dark image 302 .

[0198] The second contrast difference between the high-brightness sample fundus image 309 and the corresponding second reconstructed dark image 302 is determined according to the standard deviation of each pixel value of the high-brightness sample fundus image 309 and the standard deviation of each pixel value of the corresponding second reconstructed dark image 302 .

[0199] The first characteristic difference between the high-brightness sample fundus image 309 and the corresponding second reconstructed dark image 302 is determined according to the covariance of the high-brightness sample fundus image 309 and the corresponding second reconstructed dark image 302 .

[0200] A second image quality loss value is determined according to the second brightness difference value, the second contrast difference value, and the second feature difference value.

[0201] It should be noted that the feature difference may also be a structural difference between the sample fundus image and the corresponding reconstructed image, but the present application is not limited thereto.

[0202] As an embodiment, it is assumed that x2 is a high-brightness sample fundus image, and y2 is a second reconstructed dark image corresponding to the high-brightness sample fundus image.

[0203] Then the second brightness difference between the high brightness sample fundus image x2 and the corresponding second reconstructed dark image y2 is expressed as:

[0204]

[0205] Wherein, l(x2, y2) represents the second brightness difference between the high-brightness sample fundus image x2 and the corresponding second reconstructed dark image y2; represents the average grayscale of the high-brightness sample fundus image x2, represents the average grayscale of the second reconstructed dark image y2 corresponding to the high-brightness sample fundus image.

[0206] The second contrast difference between the high-brightness sample fundus image x2 and the corresponding second reconstructed dark image y2 is expressed as:

[0207]

[0208] Wherein, c(x2, y2) represents the second contrast difference between the high-brightness sample fundus image x2 and the second reconstructed dark image y2; represents the standard deviation of each pixel value of the high-brightness sample fundus image x2, represents the standard deviation of each pixel value of the second reconstructed dark image y2 corresponding to the high-brightness sample fundus image.

[0209] The second characteristic difference between the high-brightness sample fundus image x2 and the corresponding second reconstructed dark image y2 is expressed as:

[0210]

[0211] Wherein, s(x2, y2) represents the second characteristic difference between the high-brightness sample fundus image x2 and the corresponding second reconstructed dark image y2; represents the pixel covariance between the high-brightness sample fundus image x2 and the corresponding second reconstructed dark image y2.

[0212] Furthermore, the second image quality loss value 312 between the high-brightness sample fundus image 309 and the corresponding second reconstructed dark image 302 is:

[0213] ssim loss(x2,y2)=1-SSIM(x2,y2) (13)

[0214] Among them, SSIM(x2,y2)=[l(x2,y2) α *c(x2,y2) β *s(x2,y2) γ ], α, β and γ represent the indexes of the second brightness, the second contrast and the second characteristic difference respectively, α, β and γ can be arbitrary constants, when α=β=γ, SSIM(x2, y2)=[l(x2, y2)*c(x2, y2)*s(x2, y2)]; ssim loss(x2, y2) represents the image quality loss value of the second image quality loss function.

[0215] Step 2: Use a cycle consistency loss function to determine a first cycle loss value between a first sample fundus image in a first sample fundus image set and a corresponding first reconstructed dark image; use the cycle consistency loss function to determine a second cycle loss value between a second sample fundus image in a second sample fundus image set and a corresponding second reconstructed bright image.

[0216] Specifically, a cycle consistency loss function is used to determine a first cycle loss value 313 between a low-brightness sample fundus image 301 in a low-brightness fundus image sample set and a corresponding first reconstructed dark image 307. A cycle consistency loss function is used to determine a second cycle loss value 314 between a high-brightness sample fundus image 309 in a high-brightness fundus image sample set and a corresponding second reconstructed bright image 302.

[0217] It should be noted that the cycle consistency loss function is the L1 norm loss function.

[0218] Step three: Use the first discriminator and the adversarial loss function to determine the first discriminant value between the first sample fundus image in the first sample fundus image set and the second reconstructed dark image; use the second discriminator and the adversarial loss function to determine the second discriminant value between the second sample fundus image in the second sample fundus image set and the first reconstructed bright image.

[0219] As an embodiment, a first discriminator 310 and an adversarial loss function are used to determine a first discriminant value 315 between a low-brightness sample fundus image 301 in a low-brightness fundus image sample set and the corresponding second reconstructed dark image 308. At the same time, a second discriminator 305 and an adversarial loss function are used to determine a second discriminant value 316 between a high-brightness sample fundus image 309 in a high-brightness fundus image sample set and the corresponding first reconstructed bright image 304.

[0220] It should be noted that the adversarial loss function can be one or any combination of the square error loss function, the cross entropy loss function, the perceptual loss function, the 0-1 loss function and the regularized loss function, and is not limited here.

[0221] As an embodiment, a first nearly identical loss value between a low-brightness sample fundus image 301 and the corresponding first reconstructed dark image 307 in a low-brightness fundus image sample set is calculated by a nearly identical loss function, and a second nearly identical loss value between a high-brightness sample fundus image 309 and the corresponding second reconstructed dark image 302 is calculated by a nearly identical loss function.

[0222] Step 4: Determine the sum of the first image quality loss value, the first cycle loss value, the first discrimination value, the second image quality loss value, the second cycle loss value and the second discrimination value as the overall loss value.

[0223] Among them, the overall loss value is the sum of the first image quality loss value, the first cycle loss value, the first discrimination value, the second image quality loss value, the second cycle loss value and the second discrimination value.

[0224] As an embodiment, the sum of the first image quality loss value, the first cycle loss value, the first discrimination value, the first quasi-identical loss value, the second image quality loss value, the second cycle loss value, the second discrimination value and the second quasi-identical loss value is determined as the overall loss value.

[0225] In the above implementation process, the overall loss value between the sample fundus image and the reconstructed image is calculated through the image quality loss function, the cycle consistency loss function and the adversarial loss function, so as to further adjust the parameters of the generative adversarial network according to the overall loss value, thereby reducing the loss of texture detail features of the brightness enhanced image obtained by using the trained network model.

[0226] S1024: According to the overall loss value, the parameters in the adversarial generative network are adjusted to obtain a trained brightness enhancement model.

[0227] Specifically, when executing S1024, the following methods may be used:

[0228] Method 1: Determine whether the overall loss value meets the preset training conditions. If so, determine the first generator as the trained brightness enhancement model. If not, adjust the parameters of the first generator through the Ranger optimizer, and adjust the parameters of the second generator until the overall loss value meets the preset training conditions.

[0229] Specifically, determine whether the overall loss value is not greater than a preset threshold.

[0230] If the overall loss value is not greater than the preset threshold, the first generator is determined as the trained brightness enhancement model.

[0231] If the overall loss value is greater than the preset threshold, the parameters of the first generator are adjusted through the Ranger optimizer, and the parameters of the second generator are adjusted until the overall loss value is no greater than the preset threshold.

[0232] It should be noted that the Ranger optimizer in the embodiment of the present application is a combination of LookAhead and RAdam.

[0233] In the above implementation process, in the process of adjusting the parameters of the first generator, the parameters of the first generator are adjusted through the Ranger optimizer, which can improve the stability of the first generator model training, as well as the speed and accuracy of model convergence during the first generator model training process. Furthermore, the low-brightness fundus image is input into the trained first generator, and the generated high-brightness fundus image can retain the texture structure of the image, that is, the accuracy of the fundus image generated by the adversarial generative network is improved.

[0234] After obtaining the trained brightness enhancement model, step 102 is performed.

[0235] Specifically, the trained brightness enhancement model is used to perform brightness enhancement processing on the preprocessed low-brightness fundus image to obtain a high-brightness fundus image.

[0236] As an example, please refer to Figure 4 , Figure 4 A schematic diagram of fundus image brightness enhancement provided in an embodiment of the present application is provided, where a low-brightness fundus image 401 is input into a ResBlock+CBAM network 402 of a first generator, and a high-brightness fundus image 403 is generated.

[0237] It should be noted that during the image processing process, the size of the low-brightness fundus image is the same as the size of the original low-brightness fundus image. That is to say, during the training process, the size of the fundus image input into the adversarial generation network is 1 / 4 of the size of the low-brightness fundus image in the image processing process.

[0238] In the above implementation process, the trained brightness enhancement model, namely the first generator, is used to perform brightness enhancement processing on the preprocessed low-brightness fundus image, thereby achieving brightness enhancement of the low-brightness fundus image.

[0239] In the above implementation process, the trained brightness enhancement model is used to perform brightness enhancement processing on the fundus images to be processed. This method can perform brightness enhancement on batches of fundus images to be processed, thereby improving the image processing efficiency, reducing the workload of manual image processing, and saving the cost of image processing. In addition, the brightness enhancement model is obtained through adversarial generative network and image quality loss function training, which enhances the accuracy of parameter adjustment during model training, thereby reducing the loss of texture detail features of the image after brightness enhancement.

[0240] In the above implementation process, the parameters of the adversarial generative network in the training process are adjusted through the target loss function composed of the adversarial loss function, the cycle consistency loss function and the image quality loss function, so as to improve the accuracy of the adversarial generative network training, and the parameters of the first generator and the second generator are adjusted through the Ranger optimizer combined with LookAhead and RAdam, which can improve the stability of the adversarial generative network and the convergence efficiency in the training process, so as to reduce the image quality loss of the fundus image after brightening, and the trained adversarial generative network can process high-resolution fundus images, thereby improving the applicability of the adversarial generative network.

[0241] See also Figure 5 As shown, Figure 5 A block diagram of a fundus image processing device provided in an embodiment of the present application is provided. Figure 5 The device 500 shown is Figure 1 methods, including those that can be implemented Figure 1 Each functional module of the method.

[0242] In one embodiment, Figure 5 The illustrated apparatus 500 comprises:

[0243] The acquisition module 501 is used to acquire the fundus image to be processed.

[0244] The processing module 502 is used to use the trained brightness enhancement model to perform brightness enhancement processing on the fundus image to be processed to obtain a processed target fundus image, wherein the brightness enhancement model is obtained by training based on a generative adversarial network and an image quality loss function.

[0245] In one implementation, the acquisition module 501 is specifically used to:

[0246] Obtain the original fundus image to be processed.

[0247] The black edges of the original fundus image to be processed are trimmed, features are extracted, and the size is adjusted to obtain the fundus image to be processed.

[0248] In one implementation, the processing module 502 is further configured to:

[0249] Get a collection of sample fundus images.

[0250] The sample fundus images in the sample fundus image set are input into the adversarial generative network to obtain the corresponding reconstructed sample fundus images.

[0251] The target loss function is used to determine the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images.

[0252] According to the overall loss value, the parameters in the adversarial generative network are adjusted to obtain a trained brightness enhancement model.

[0253] In one embodiment, the objective loss function is represented by formula (1):

[0254] L(G, F, D x , D y )=L GAN (G,D y ,X,Y)+L GAN (F,D x , Y, X)+λL cyc (G, F)+L identity (G, F)+L ssim (G, F) (1)

[0255] Among them, L ssim (G, F) = 1-|ssim(X, F(G(X)))| + 1-|ssim(Y, F(G(Y)))| = 2-|ssim(X, F(G(X)))| - |ssim(Y, F(G(Y)))|, where G represents the first generator, F represents the second generator, and D represents the x represents the first discriminator, D y represents the second discriminator, X represents the first sample fundus image, Y represents the second sample fundus image, L(G, F, D x , D y ) represents the target loss function, L GAN (G,D y , X, Y) represents the adversarial loss function between the first generator and the second discriminator, L GAN (F,D x , Y, X) represents the adversarial loss function between the second generator and the first discriminator, L identity (G, F) represents the nearly identical loss function between the first generator and the second generator, L ssim (G, F) represents the image quality loss function between the first generator and the second generator.

[0256] In one implementation, the processing module 502 is specifically configured to:

[0257] An initial image set is acquired, wherein the initial image set includes a first initial image set and a second initial image set, the first initial image set includes at least one first initial image, and the second initial image set includes at least one second initial image.

[0258] The brightness value of each first initial image in the first initial image set is within a first brightness value range, the brightness value of each second initial image in the second initial image set is within a second brightness value range, and the maximum brightness value in the first brightness value range is less than the minimum brightness value in the second brightness value range.

[0259] The black edges of each first initial image in the first initial image set are trimmed, features are extracted, and the size is adjusted to obtain a first preprocessed initial image set, wherein the first preprocessed initial image set includes the first preprocessed initial images corresponding to each first initial image.

[0260] The black edges of each second initial image in the second initial image set are trimmed, features are extracted, and the size is adjusted to obtain a second preprocessed initial image set, wherein the second preprocessed initial image set includes the second preprocessed initial images corresponding to each second initial image.

[0261] At least one first preprocessed initial image in the first preprocessed initial image set is randomly cropped and flipped to obtain a first sample fundus image set, wherein the first sample fundus image set includes a plurality of first sample fundus images.

[0262] At least one second preprocessed initial image in the second preprocessed initial image set is randomly cropped and flipped to obtain a second sample fundus image set, wherein the second sample fundus image set includes a plurality of second sample fundus images.

[0263] The first sample fundus image set and the second sample fundus image set are both sample fundus image sets, and the plurality of first sample fundus images and the plurality of second sample fundus images are both sample fundus images.

[0264] In one embodiment, the adversarial generative network includes a first generator and a second generator.

[0265] The processing module 502 is specifically used to: input the first sample fundus image in the first sample fundus image set into the first generator to obtain a first reconstructed bright image.

[0266] The first reconstructed bright image is input into the second generator to obtain the first reconstructed dark image.

[0267] The second sample fundus image in the second sample fundus image set is input into the second generator to obtain a second reconstructed dark image.

[0268] The second reconstructed dark image is input into the first generator to obtain the second reconstructed bright image.

[0269] The first reconstructed dark image and the second reconstructed bright image are both reconstructed sample fundus images.

[0270] In one embodiment, the target loss function includes an adversarial loss function, a cycle consistency loss function, and an image quality loss function, and the adversarial generation network also includes a first discriminator and a second discriminator.

[0271] The processing module 502 is specifically used to: determine a first image quality loss value between a first sample fundus image in the first sample fundus image set and a corresponding first reconstructed dark image by using an image quality loss function.

[0272] A cycle consistency loss function is used to determine a first cycle loss value between a first sample fundus image in the first sample fundus image set and a corresponding first reconstructed dark image.

[0273] A first discriminant and an adversarial loss function are used to determine a first discriminant value between a first sample fundus image and a second reconstructed dark image in a first sample fundus image set.

[0274] An image quality loss function is used to determine a second image quality loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image.

[0275] A cycle consistency loss function is used to determine a second cycle loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image.

[0276] A second discriminant and an adversarial loss function are used to determine a second discriminant value between a second sample fundus image in a second sample fundus image set and the first reconstructed bright image.

[0277] Among them, the overall loss value is the sum of the first image quality loss value, the first cycle loss value, the first discrimination value, the second image quality loss value, the second cycle loss value and the second discrimination value.

[0278] In one implementation, the processing module 502 is specifically configured to:

[0279] Determine whether the overall loss value meets the preset training conditions.

[0280] If so, the first generator is determined to be the trained brightness enhancement model.

[0281] If not, the parameters of the first generator are adjusted through the Ranger optimizer, and the parameters of the second generator are adjusted until the overall loss value meets the preset training conditions.

[0282] It should be noted that Figure 5 The device 500 shown can achieve Figure 1 The operations and / or functions of the various modules in the apparatus 500 are respectively to implement Figure 1 For details, please refer to the description in the above method embodiment. To avoid repetition, detailed description is appropriately omitted here.

[0283] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 6 The electronic device 600 shown may include: at least one processor 610, such as a CPU, at least one communication interface 620, at least one memory 630 and at least one communication bus 640. Among them, the communication bus 640 is used to realize the direct connection and communication of these components. Among them, the communication interface 620 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 630 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 630 can optionally also be at least one storage device located away from the aforementioned processor. The memory 630 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 610, the electronic device executes the above-mentioned Figure 1 The method process is shown.

[0284] The present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a server, Figure 1 The method process shown.

[0285] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system embodiments described above are only illustrative. For example, the division of the system devices is only a logical function division. There may be other division methods in actual implementation. For example, multiple devices or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0286] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0287] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing fundus images, characterized in that: include: Acquiring a fundus image to be processed; Using the trained brightness enhancement model, the to-be-processed fundus image is subjected to brightness enhancement processing to obtain a processed target fundus image; Wherein, the brightness enhancement model is obtained by training the adversarial generation network based on the image quality loss function; Before using the trained brightness enhancement model to perform brightness enhancement processing on the fundus image to be processed, the method further includes: obtaining a sample fundus image set; inputting the sample fundus images in the sample fundus image set into the adversarial generation network to obtain corresponding reconstructed sample fundus images; using the target loss function, determining the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images; adjusting the parameters in the adversarial generation network according to the overall loss value to obtain the trained brightness enhancement model; The method of acquiring a sample fundus image set includes: acquiring an initial image set, wherein the initial image set includes a first initial image set and a second initial image set, the first initial image set includes at least one first initial image, and the second initial image set includes at least one second initial image; the brightness value of each first initial image in the first initial image set is within a first brightness value range, the brightness value of each second initial image in the second initial image set is within a second brightness value range, and the maximum brightness value in the first brightness value range is less than the minimum brightness value in the second brightness value range; performing black border trimming, feature extraction, and size adjustment on each first initial image in the first initial image set to obtain a first preprocessed initial image set, wherein the first preprocessed initial image set includes the first preprocessed initial image corresponding to each first initial image; performing preprocessing on each second initial image in the second initial image set The image is subjected to the black border trimming, feature extraction and size adjustment to obtain a second preprocessed initial image set, wherein the second preprocessed initial image set includes second preprocessed initial images corresponding to each second initial image; at least one first preprocessed initial image in the first preprocessed initial image set is randomly cropped and flipped to obtain a first sample fundus image set, wherein the first sample fundus image set includes a plurality of first sample fundus images; at least one second preprocessed initial image in the second preprocessed initial image set is randomly cropped and flipped to obtain a second sample fundus image set, wherein the second sample fundus image set includes a plurality of second sample fundus images; wherein the first sample fundus image set and the second sample fundus image set are both sample fundus image sets, and the plurality of first sample fundus images and the plurality of second sample fundus images are both sample fundus images; The adversarial generative network includes a first generator and a second generator; the step of inputting the sample fundus images in the sample fundus image set into the adversarial generative network to obtain the corresponding reconstructed sample fundus images includes: inputting the first sample fundus image in the first sample fundus image set into the first generator to obtain a first reconstructed bright image; inputting the first reconstructed bright image into the second generator to obtain a first reconstructed dark image; inputting the second sample fundus image in the second sample fundus image set into the second generator to obtain a second reconstructed dark image; inputting the second reconstructed dark image into the first generator to obtain a second reconstructed bright image; wherein the first reconstructed dark image and the second reconstructed bright image are both reconstructed sample fundus images; The target loss function includes an adversarial loss function, a cycle consistency loss function and the image quality loss function, and the adversarial generation network also includes a first discriminator and a second discriminator; the use of the target loss function to determine the overall loss value between the sample fundus images in the sample fundus image set and the corresponding reconstructed sample fundus images includes: using the image quality loss function to determine the first image quality loss value between the first sample fundus image in the first sample fundus image set and the corresponding first reconstructed dark image; using the cycle consistency loss function to determine the first cycle loss value between the first sample fundus image in the first sample fundus image set and the corresponding first reconstructed dark image; using the first discriminator and the adversarial loss function to determine the first sample fundus image in the first sample fundus image set. a first discriminant value between a sample fundus image and the second reconstructed dark image; using the image quality loss function to determine a second image quality loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image; using the cycle consistency loss function to determine a second cycle loss value between a second sample fundus image in the second sample fundus image set and a corresponding second reconstructed bright image; using the second discriminator and the adversarial loss function to determine a second discriminant value between a second sample fundus image in the second sample fundus image set and the first reconstructed bright image; wherein the overall loss value is the sum of the first image quality loss value, the first cycle loss value, the first discriminant value, the second image quality loss value, the second cycle loss value and the second discriminant value; The objective loss function is represented by formula (1): L(G,F,D x ,D y )=L GAN (G,D y ,X,Y)+L GAN (F,D x ,Y,X)+λL cyc (G,F)+L identity (G,F)+L ssim (G,F)(1) in, L ssim (G,F)=1-|yes(X,F(G(X)))|+1-|yes(Y,F(G(Y)))| =2-|ssim(X,F(G(X)))|-|ssim(Y,F(G(Y)))|| G represents the first generator, F represents the second generator, and D x represents the first discriminator, D y represents the second discriminator, X represents the first sample fundus image, Y represents the second sample fundus image, L(G, F, D x ,D y ) represents the objective loss function, L GAN (G,D y ,X,Y) represents the adversarial loss function between the first generator and the second discriminator, L GAN (F,D x ,Y,X) represents the adversarial loss function between the second generator and the first discriminator, L cyc (G, F) Cycle consistency loss function, λ is the coefficient of the cycle consistency loss function, L identity (G, F) represents the nearly identical loss function between the first generator and the second generator, L ssim (G, F) represents the image quality loss function between the first generator and the second generator.

2. The method according to claim 1, characterized in that The step of adjusting the parameters in the generative adversarial network according to the overall loss value to obtain a trained brightness enhancement model includes: Determine whether the overall loss value meets the preset training conditions; If yes, the first generator is determined to be the trained brightness enhancement model; If not, the parameters of the first generator are adjusted through the Ranger optimizer, and the parameters of the second generator are adjusted until the overall loss value meets the preset training conditions.

3. A fundus image processing device, characterized in that: The device is used to perform the method according to claim 1, comprising: An acquisition module, used for acquiring the fundus image to be processed; A processing module, used to use a trained brightness enhancement model to perform brightness enhancement processing on the fundus image to be processed to obtain a processed target fundus image; Wherein, the brightness enhancement model is obtained by training the adversarial generation network based on the image quality loss function.

4. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images