Eye fundus color photo enhanced segmentation method and device based on Diffusion

Abstract

The invention relates to the field of fundus photograph enhancement and segmentation, and provides a fundus photograph enhancement segmentation method and device based on Diffusion, and the method comprises the steps: obtaining a fundus photograph, inputting the fundus photograph into a pre-trained Diffusion-based combined image enhancement segmentation model, and obtaining a fundus photograph enhancement image and a fundus photograph retinal blood vessel segmentation image; the model comprises a Diffusion-based image enhancement branch, an image segmentation branch and a symbiotic information interaction module, and the symbiotic information interaction module is used for realizing information flow between the image enhancement branch and the image segmentation branch, so that the image enhancement branch generates a fundus color photograph enhancement image based on a fundus color photograph and information from the image segmentation branch. Meanwhile, the image segmentation branch generates a fundus color photoretinal blood vessel segmentation image based on the fundus color photograph and information from the image enhancement branch. According to the invention, the understanding and analysis of the model on the image can be improved, and accurate blood vessel segmentation is provided while the image quality is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of fundus color photograph enhancement and segmentation, and particularly relates to a fundus color photograph enhancement and segmentation method and device based on Diffusion. BACKGROUND

[0002] Medical images taken under suboptimal conditions may have quality problems such as blurring, poor lighting, low resolution, and noise, which may lead to misdiagnosis. However, advanced medical image enhancement techniques are difficult to improve image resolution quality while maintaining clear local anatomical features. Typically, image enhancement and image segmentation are developed and implemented independently, and each task needs to be processed separately.

[0003] In recent years, an image generation model called StillGAN has appeared in the prior art. StillGAN is a Generative Adversarial Network (GAN) based image generation model that proposes an innovative and universal bidirectional GAN, mainly used to learn the latent structure and texture distribution of a scene from a single static image and generate high-quality, diverse perspective change images, which can be used to enhance medical image quality. Researchers are also increasingly interested in the ability of the powerful generative model Stable Diffusion to improve repair performance. StableSR is a Diffusion model based image Super-Resolution (SR) algorithm, mainly used to improve the resolution and quality of images. StableSR uses low-quality images as control conditions to guide stable diffusion. These methods highlight the great potential of generative in image enhancement tasks. However, using only low-quality image information without introducing additional semantic control can lead to incorrect content reconstruction. The Synchronous Edge Alignment and Memory Auxiliary Learning (SEAMAL) framework explores effective segmentation in noisy labels. Although directly fine-tuning the segmentation network on low-quality images can enable the segmentation network to some extent to obtain the ability to perceive degradation, the performance improvement is still limited. Although low-quality medical image enhancement and image segmentation have made some progress, research on their interdependence is still limited.

[0004] CHLNet can be applied to high-resolution medical image enhancement. The high-resolution auxiliary quality enhancement network in CHLNet is used to eliminate global low-quality factors in high-resolution images, thereby improving the overall image quality. The high-quality semantic guidance in CHLNet is used to capture semantic knowledge of high-quality images from the output of the quality enhancement network, thereby ensuring that detailed structures are retained. In addition, due to its lightweight design, CHLNet can be easily deployed on medical edge devices. However, the above-mentioned existing method has limited research on the relationship between the medical image enhancement task and the image segmentation task, which first enhances the medical image and then segments the image, which is disconnected from the relationship between image enhancement and image segmentation.

[0005] Another prior art first uses a denoising network to alleviate noise degradation, then a segmentation network learns instance-level affinity first, encodes important spatial structure information, and finally proposes a new instance-aware embedding module in the fusion network to utilize important spatial structure information to extract denoising features from segmentation features, thereby enhancing the denoising effect. However, the above-mentioned prior art is also not comprehensive enough in studying the relationship between the two tasks of medical image enhancement and image segmentation, only studying the effect of image segmentation on image enhancement, and not fully utilizing the relationship between the two tasks to gain from each other. SUMMARY

[0006] The present disclosure aims to at least solve one of the problems existing in the prior art, and provides a Diffusion-based fundus color photograph enhancement segmentation method and device.

[0007] In one aspect of the present disclosure, a Diffusion-based fundus color photograph enhancement segmentation method is provided, which comprises:

[0008] Obtaining a fundus color photograph;

[0009] Inputting the fundus color photograph into a pre-trained Diffusion-based joint image enhancement segmentation model to obtain a fundus color photograph enhanced image and a fundus color photograph retinal blood vessel segmentation image;

[0010] The Diffusion-based joint image enhancement and segmentation model comprises a Diffusion-based image enhancement branch, an image segmentation branch, and a coexistence information interaction module. The coexistence information interaction module is configured to realize information flow between the Diffusion-based image enhancement branch and the image segmentation branch, so that the Diffusion-based image enhancement branch generates the fundus color image enhancement map based on the fundus color image and information from the image segmentation branch, and at the same time, the image segmentation branch generates the fundus color image retinal blood vessel segmentation map based on the fundus color image and information from the Diffusion-based image enhancement branch.

[0011] Optionally, the Diffusion-based image enhancement branch is configured to perform an image enhancement forward diffusion process and an image enhancement reverse diffusion process in a latent space.

[0012] The image enhancement forward diffusion process is configured to add Gaussian noise to the fundus color image to generate a noise image.

[0013] The image enhancement reverse diffusion process is configured to generate the fundus color image enhancement map based on the noise image, input image encoding features obtained by performing convolution operation on the fundus color image, and semantic control features generated by the coexistence information interaction module based on information from the image segmentation branch.

[0014] Optionally, the image enhancement reverse diffusion process is configured to generate the fundus color image enhancement map based on the noise image, input image encoding features obtained by performing convolution operation on the fundus color image, and semantic control features generated by the coexistence information interaction module based on information from the image segmentation branch, comprising:

[0015] The image enhancement reverse diffusion process is configured to: based on the input image encoding features and the noise image, systematically remove noise in the noise image by using an image denoiser to obtain a corresponding denoising result, while maintaining alignment between the denoising result and the input fundus color image, and iteratively output the fundus color image enhancement map by combining the denoising result with the semantic control features generated by the coexistence information interaction module based on information from the image segmentation branch.

[0016] Optionally, the image enhancement reverse diffusion process is implemented by using latent diffusion models (LDMs).

[0017] Optionally, the image segmentation branch is configured to perform an image segmentation forward diffusion process and an image segmentation reverse diffusion process.

[0018] The image segmentation forward diffusion process is configured to adjust the encoding size with a real segmentation mask, introduce noise into the input fundus photo based on the adjusted encoding, iteratively add noise to a current state of the fundus photo to a next state, and generate a low-quality image representation;

[0019] The image segmentation reverse diffusion process is configured to generate a low-quality latent embedding conditioned on the low-quality image representation, and iteratively output a fundus photo retinal vessel segmentation map through a U-Net segmentation module based on the low-quality latent embedding, guide features generated by the coexistence information interaction module based on information from the Diffusion-based image enhancement branch, and input image encoding features.

[0020] Optionally, the coexistence information interaction module is configured to generate the semantic control features through a semantic control feature generation process, wherein the semantic control feature generation process comprises:

[0021] The denoising result is obtained from the Diffusion-based image enhancement branch as first image enhancement input features, and the low-quality latent embedding is obtained from the image segmentation branch as first image segmentation input features;

[0022] The first image enhancement input features and the first image segmentation input features are sequentially subjected to a Flatten operation and a full connection operation, respectively, to obtain corresponding first enhancement full connection results and first segmentation full connection results;

[0023] The first enhancement full connection results and the first segmentation full connection results are operated using a multi-scale point multiplication attention, and a full connection operation is performed on the operation results to obtain first attention enhancement features;

[0024] The first attention enhancement features are connected in residual with the first image enhancement input features to obtain the semantic control features.

[0025] Optionally, the coexistence information interaction module is configured to generate the guide features through a guide feature generation process, wherein the guide feature generation process comprises:

[0026] The denoising result is obtained from the Diffusion-based image enhancement branch as second image enhancement input features, and the low-quality latent embedding is obtained from the image segmentation branch as second image segmentation input features;

[0027] The second image enhancement input features and the second image segmentation input features are sequentially subjected to a Flatten operation and a full connection operation, respectively, to obtain corresponding second enhancement full connection results and second segmentation full connection results;

[0028] performing an operation on the second enhanced full connection result and the second segmentation full connection result by using multi-scale point multiplication attention, and performing a full connection operation on an operation result to obtain a second attention enhanced feature;

[0029] performing a residual connection on the second attention enhanced feature and the second image segmentation input feature to obtain the guidance feature.

[0030] Optionally, in the training process of the Diffusion-based image enhancement branch, the final result of image enhancement is supervised by a restoration loss.

[0031] Optionally, in the training process of the image segmentation branch, a weighted binary cross-entropy loss is used to optimize the final result of image segmentation.

[0032] Another aspect of the present disclosure provides a Diffusion-based fundus color photograph enhancement segmentation device, which comprises:

[0033] an acquisition module configured to acquire a fundus color photograph;

[0034] an image enhancement segmentation module configured to input the fundus color photograph into a pre-trained Diffusion-based joint image enhancement segmentation model to obtain a fundus color photograph enhanced image and a fundus color photograph retinal blood vessel segmentation image;

[0035] The Diffusion-based joint image enhancement segmentation model comprises a Diffusion-based image enhancement branch, an image segmentation branch, and a co-occurrence information interaction module. The co-occurrence information interaction module is configured to realize information flow between the Diffusion-based image enhancement branch and the image segmentation branch, so that the Diffusion-based image enhancement branch generates the fundus color photograph enhanced image based on the fundus color photograph and information from the image segmentation branch, and simultaneously, the image segmentation branch generates the fundus color photograph retinal blood vessel segmentation image based on the fundus color photograph and information from the Diffusion-based image enhancement branch.

[0036] Compared with the existing technology, the present disclosure obtains a fundus color photograph and inputs the fundus color photograph into a pre-trained joint image enhancement and segmentation model based on diffusion to obtain a fundus color photograph enhanced image and a fundus color photograph retinal vessel segmentation map. This not only improves the model's understanding of the shape, edge, and texture features in the image, thereby improving the image quality and making the fundus color photograph enhanced image clearer, but also makes the segmentation information in the fundus color photograph retinal vessel segmentation map more accurate, which can better assist ophthalmologists in completing the analysis of fundus color photograph image features and provide assistance for the accurate diagnosis of ophthalmic diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0038] Figure 1 This is a flowchart of a diffusion-based fundus color photography enhancement and segmentation method provided in one embodiment of the present disclosure;

[0039] Figure 2 A schematic diagram of a framework of a joint image enhancement segmentation model based on Diffusion provided in another embodiment of the present disclosure;

[0040] Figure 3 A schematic diagram of a color fundus photograph provided in another embodiment of the present disclosure;

[0041] Figure 4 for Figure 3 The fundus color photograph shown corresponds to the enhanced fundus color photograph;

[0042] Figure 5 for Figure 3 The retinal blood vessel segmentation diagram of the fundus color photograph corresponding to the fundus color photograph shown;

[0043] Figure 6 A schematic diagram of an image enhancement reverse diffusion process provided by another embodiment of the present disclosure;

[0044] Figure 7 A schematic diagram of an image segmentation inverse diffusion process provided by another embodiment of the present disclosure;

[0045] Figure 8 A schematic diagram of a semantic control feature generation process and a guidance feature generation process performed by a symbiotic information interaction module according to another embodiment of the present disclosure;

[0046] Figure 9A structure diagram of a Diffusion-based fundus color photograph enhancement segmentation device according to another embodiment of the present disclosure is provided. DETAILED DESCRIPTION

[0047] To make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the drawings. However, it can be understood by those skilled in the art that, in the embodiments of the present disclosure, many technical details are presented in order to make the readers better understand the present disclosure. However, the technical solutions claimed by the present disclosure can be implemented even without these technical details and based on various changes and modifications of the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific embodiments of the present disclosure, and the embodiments can be combined with each other and cited to each other without contradiction.

[0048] One embodiment of the present disclosure relates to a Diffusion-based fundus color photograph enhancement segmentation method, the flow of which is shown in Figure 1 as including steps S110 and S120.

[0049] Step S110, obtaining a fundus color photograph.

[0050] Specifically, the fundus is the inner surface of the posterior part of the eyeball, including key structures such as the retina, optic nerve, and macula. The fundus color photograph can be obtained by photographing the fundus by a fundus photographing device (such as a fundus camera).

[0051] Step S120, inputting the fundus color photograph into a pre-trained Diffusion-based joint image enhancement segmentation model to obtain a fundus color photograph enhancement map and a fundus color photograph retinal blood vessel segmentation map. The Diffusion-based joint image enhancement segmentation model includes a Diffusion-based image enhancement branch, an image segmentation branch, and a co-occurrence information interaction module. The co-occurrence information interaction module is used to realize the information flow between the Diffusion-based image enhancement branch and the image segmentation branch, so as to make the Diffusion-based image enhancement branch generate the fundus color photograph enhancement map based on the fundus color photograph and information from the image segmentation branch, and at the same time make the image segmentation branch generate the fundus color photograph retinal blood vessel segmentation map based on the fundus color photograph and information from the Diffusion-based image enhancement branch.

[0052] Specifically, in combination with Figure 2, step S120 can input the fundus photograph into the Diffusion-based image enhancement branch and the image segmentation branch of the Diffusion-based joint image enhancement and segmentation model respectively, so as to obtain a fundus photograph enhanced image corresponding to the fundus photograph through the Diffusion-based image enhancement branch, and obtain a fundus photograph retinal blood vessel segmentation image corresponding to the fundus photograph through the image segmentation branch. The image segmentation branch can provide information from the fundus photograph to the Diffusion-based image enhancement branch after processing by the co-occurrence information interaction module, so that the Diffusion-based image enhancement branch generates the fundus photograph enhanced image based on the fundus photograph and the information from the image segmentation branch. The Diffusion-based image enhancement branch can provide information from the input fundus photograph to the image segmentation branch after processing by the co-occurrence information interaction module, so that the image segmentation branch generates the fundus photograph retinal blood vessel segmentation image based on the fundus photograph and the information from the Diffusion-based image enhancement module.

[0053] The image segmentation branch is mainly used to segment the retinal blood vessel region from the fundus photograph. It should be noted that although the retinal blood vessel region usually includes the retinal artery region and the retinal vein region, the image segmentation branch does not distinguish between the retinal artery region and the retinal vein region when completing the image segmentation task, but regards the retinal artery region and the retinal vein region as the retinal blood vessel region. For example, the fundus photograph shown in Figure 3 is input into the pre-trained Diffusion-based joint image enhancement and segmentation model, and the obtained fundus photograph enhanced image and fundus photograph retinal blood vessel segmentation image are as shown in Figure 4 , Figure 5 . Figure 5 In the above-mentioned fundus photograph enhanced image and fundus photograph retinal blood vessel segmentation image, the black part represents the background, and the white part is the retinal blood vessel region segmented from the fundus photograph shown in Figure 3 .

[0054] By using the co-occurrence information interaction module to realize the information flow between the Diffusion-based image enhancement branch and the image segmentation branch, the information can be exchanged between the Diffusion-based image enhancement branch and the image segmentation branch, so that the two branches can benefit from each other when completing their respective tasks.

[0055] For example, the Diffusion-based image enhancement branch is used to perform an image enhancement forward diffusion process and an image enhancement reverse diffusion process in the latent space.

[0056] Specifically, inspired by Denoising Diffusion Probabilistic Models (DDPMs), the embodiment uses a Diffusion-based image enhancement branch to perform an image enhancement forward diffusion process and an image enhancement reverse diffusion process within the latent space, and adds information from the image segmentation branch to the image enhancement reverse diffusion process, so that the Diffusion-based image enhancement branch benefits from the image enhancement reverse diffusion process.

[0057] The image enhancement forward diffusion process is used to add Gaussian noise to the fundus photograph to generate a noisy image.

[0058] Specifically, the image enhancement forward diffusion process can utilize a pre-trained variational autoencoder, in particular, an image encoder and an image decoder in the variational autoencoder, to generate a noisy image by adding Gaussian noise to the fundus photograph based on a given input image, i.e., the fundus photograph obtained in step S110, and a scalar noise level. In the initial stage of the image enhancement forward diffusion process, the fundus photograph is encoded and embedded, and then Gaussian noise can be gradually added to the fundus photograph according to the scalar noise level, so as to generate latent noise and generate a noisy image.

[0059] The image enhancement reverse diffusion process is used to generate an enhanced fundus photograph based on the noisy image, the input image encoding features obtained by performing convolution operation on the fundus photograph, and the semantic control features generated by the co-occurrence information interaction module based on the information from the image segmentation branch.

[0060] Illustratively, the image enhancement reverse diffusion process is implemented by a latent diffusion model (LDM).

[0061] Specifically, the latent diffusion model (LDM) is a new high-resolution image synthesis method proposed in CVPR 2022. The latent diffusion model (LDM) achieves the goal of training a high-quality image synthesis model under limited computing resources by applying a diffusion model in the latent space of a pre-trained autoencoder.

[0062] Illustratively, the image enhancement reverse diffusion process is specifically used to: based on the input image encoding features and the noisy image, systematically remove the noise in the noisy image using an image denoiser to obtain a corresponding denoising result, while maintaining the alignment between the denoising result and the input fundus photograph, and iteratively performing the denoising result with the semantic control features generated by the co-occurrence information interaction module based on the information from the image segmentation branch, to output an enhanced fundus photograph.

[0063] That is, in combination with Figure 6The image enhancement reverse diffusion process first uses the image denoiser to systematically remove noise from the noisy image based on the input image encoding features obtained by convolution operation on the fundus color photo, and obtains the corresponding denoising result while maintaining the alignment between the denoising result and the input fundus color photo. After that, the denoising result is iterated multiple times with the semantic control features output by the symbiotic information interaction module to output the fundus color photo enhancement map. The number of iterations of the denoising result and the semantic control features can be set to T. When the image enhancement reverse diffusion process is implemented by the latent diffusion model LDMs, Figure 6 The noise prediction module in refers to the latent diffusion model LDMs.

[0064] By using the Diffusion-based image enhancement branch to perform image enhancement forward diffusion process and image enhancement reverse diffusion process in the latent space, the image quality can be effectively improved, making the final enhanced fundus color photograph clearer.

[0065] Exemplarily, the image segmentation branch is used to perform an image segmentation forward diffusion process and an image segmentation reverse diffusion process.

[0066] The image segmentation forward diffusion process is used to adjust the code size using the true segmentation mask. Based on the adjusted code, noise is gradually introduced into the input fundus color photograph. The current state of the fundus color photograph is iteratively noised to the next state, generating a low-quality image representation. Specifically, during the image segmentation forward diffusion process, the true segmentation mask is first simply resized. Then, based on the adjusted code, noise is gradually introduced into the input fundus color photograph. The current state of the fundus color photograph is iteratively noised to the next state, generating a low-quality image representation.

[0067] The image segmentation inverse diffusion process is used to generate low-quality latent embeddings based on low-quality image representation. The low-quality latent embeddings, the guiding features generated by the symbiotic information interaction module based on the information from the diffusion-based image enhancement branch, and the input image encoding features obtained by convolution operation on the fundus color photograph are gradually iterated through the U-Net segmentation module to output the retinal vascular segmentation map of the fundus color photograph.

[0068] Specifically, combined with Figure 7 During the image segmentation backward diffusion process, the image segmentation branch first generates a low-quality latent embedding based on the low-quality image representation generated by the image segmentation forward diffusion process. Subsequently, the low-quality latent embedding, the guidance features output by the symbiotic information interaction module, and the input image encoding features obtained by convolution of the fundus color image are iterated step by step using the U-Net segmentation module. After the iteration is complete, a retinal vascular segmentation map of the fundus color image is output. The number of iterations of the step-by-step iteration using the U-Net segmentation module can also be set to T. Figure 7The segmentation prediction module in the image segmentation branch refers to a U-Net segmentation module.

[0069] By using the image segmentation branch to perform the image segmentation forward diffusion process and the image segmentation reverse diffusion process, the initial segmentation result of the image segmentation branch can be gradually refined, the initial segmentation result is more closely aligned with the distribution of the real segmentation mask, and the accuracy of the segmentation information in the fundus color photograph retinal blood vessel segmentation graph output by the image segmentation branch is effectively improved.

[0070] Illustratively, the symbiotic information interaction module is configured to generate semantic control features through a semantic control feature generation process.

[0071] In combination Figure 8 The semantic control feature generation process includes: obtaining the denoising result as the first image enhancement input feature from the Diffusion-based image enhancement branch, and obtaining the low-quality latent embedding as the first image segmentation input feature from the image segmentation branch. The first image enhancement input feature and the first image segmentation input feature are sequentially subjected to a Flatten operation and a full connection operation, respectively, to obtain corresponding first enhancement full connection results and first segmentation full connection results. The first enhancement full connection results and the first segmentation full connection results are operated using a multi-scale point multiplication attention, and a full connection operation is performed on the operation results to obtain a first attention enhancement feature. The first attention enhancement feature is connected in residual with the first image enhancement input feature to obtain the semantic control feature.

[0072] Specifically, the first enhancement full connection result refers to a result obtained by performing a Flatten operation on the first image enhancement input feature and performing a full connection operation on the corresponding Flatten operation result. The first segmentation full connection result refers to a result obtained by performing a Flatten operation on the first image segmentation input feature and performing a full connection operation on the corresponding Flatten operation result. For example, as shown in Figure 8 In the semantic control feature generation process, after performing a Flatten operation on the first image enhancement input feature, two full connection operations can be performed on the Flatten operation result to obtain two first enhancement full connection results. Then, the two first enhancement full connection results and the first segmentation full connection result are operated using a multi-scale point multiplication attention to obtain a corresponding operation result. After performing a full connection operation on the operation result, a first attention enhancement feature is obtained. Finally, the first attention enhancement feature is connected in residual with the first image enhancement input feature to obtain the semantic control feature.

[0073] Illustratively, the symbiotic information interaction module is configured to generate guide features through a guide feature generation process.

[0074] In combination Figure 8The guided feature generation process includes: obtaining the denoising result from the diffusion-based image enhancement branch as the second image enhancement input feature, and obtaining the low-quality latent embedding from the image segmentation branch as the second image segmentation input feature. Flattening and fully connecting the second image enhancement input feature and the second image segmentation input feature are performed in sequence to obtain the corresponding second enhanced fully connected result and second segmentation fully connected result. Multi-scale point product attention is used to operate the second enhanced fully connected result and the second segmentation fully connected result, and the operation result is fully connected to obtain the second attention enhanced feature. The second attention enhanced feature is residually connected with the second image segmentation input feature to obtain the guided feature.

[0075] Specifically, the second enhanced fully connected result refers to the result obtained by performing a Flatten operation on the second image enhanced input feature and performing a fully connected operation on the corresponding Flatten operation result. The second segmentation fully connected result refers to the result obtained by performing a Flatten operation on the second image segmentation input feature and performing a fully connected operation on the corresponding Flatten operation result. For example, Figure 8 As shown in the figure, during the guide feature generation process, after the second image enhancement input feature is flattened, two fully connected operations can be performed on the Flatten operation results respectively, thereby obtaining two second enhanced fully connected results. Afterwards, the two second enhanced fully connected results and the second segmentation fully connected result are operated using multi-scale point product attention to obtain the corresponding operation results. After the operation results are fully connected, the second attention enhancement feature is obtained. Finally, the second attention enhancement feature is residually connected with the second image segmentation input feature to obtain the guide feature.

[0076] In particular, the first image enhancement input feature and the second image enhancement input feature in the above embodiment both refer to the denoising results obtained by the image enhancement branch based on Diffusion in the image enhancement inverse diffusion process. Here, only the first image enhancement input feature is used to refer to the denoising result obtained by the image enhancement branch based on Diffusion as the image enhancement input feature in the semantic control feature generation process, and the second image enhancement input feature is used to refer to the denoising result obtained by the image enhancement branch based on Diffusion as the image enhancement input feature in the guide feature generation process.

[0077] Similarly, the first image segmentation input feature and the second image segmentation input feature in the above embodiments both refer to the low-quality latent embedding generated by the image segmentation branch in the image segmentation reverse diffusion process. Here, the first image segmentation input feature refers to the low-quality latent embedding generated by the image segmentation branch as the image segmentation input feature in the semantic control feature generation process, and the second image segmentation input feature refers to the low-quality latent embedding generated by the image segmentation branch as the image segmentation input feature in the guide feature generation process.

[0078] For example, to train the Diffusion-based joint image enhancement segmentation model, a public fundus dataset can be selected as the model dataset. By dividing the model dataset into a training set and a test set according to actual needs at a preset ratio, the training set can be used to train the Diffusion-based joint image enhancement segmentation model, and the test set can be used to test the trained Diffusion-based joint image enhancement segmentation model. During the training process, the final result of image enhancement output by the Diffusion-based image enhancement branch can be supervised by the restoration loss, and the final result of image segmentation of the predicted mask in the image segmentation branch can be optimized by using the weighted binary cross-entropy loss.

[0079] In particular, when training the Diffusion-based joint image enhancement segmentation model, a joint training strategy can be used to train the Diffusion-based image enhancement branch, the image segmentation branch, and the symbiotic information interaction module in the Diffusion-based joint image enhancement segmentation model simultaneously using an overall target loss function. The overall target loss function combines the restoration loss and the weighted binary cross-entropy loss, so that the final result of image enhancement is supervised by the restoration loss during the training process of the Diffusion-based image enhancement branch, and the final result of image segmentation is optimized by using the weighted binary cross-entropy loss during the training process of the image segmentation branch, thereby effectively improving the understanding of the model for shape, edge, and texture features in the image.

[0080] The Diffusion-based fundus color photograph enhancement segmentation method provided by the embodiments of the present disclosure improves the understanding of the model for shape, edge, and texture features in the image, improves the image quality, makes the fundus color photograph enhancement image clearer, and also makes the segmentation information in the fundus color photograph retinal blood vessel segmentation image more accurate, which can better assist ophthalmologists in analyzing the image features of the fundus color photograph and provide help for precise diagnosis of ophthalmic diseases.

[0081] Another implementation of the present disclosure relates to a Diffusion-based fundus photograph enhancement segmentation device, as shown, comprising an acquisition module 910 and an image enhancement segmentation module 920. Figure 9 The acquisition module 910 is configured to acquire a fundus photograph.

[0082] The acquisition module 910 is configured to acquire a fundus photograph.

[0083] The image enhancement segmentation module 920 is configured to input the fundus photograph into a pre-trained Diffusion-based joint image enhancement segmentation model to obtain a fundus photograph enhancement image and a fundus photograph retinal blood vessel segmentation image. The Diffusion-based joint image enhancement segmentation model comprises a Diffusion-based image enhancement branch, an image segmentation branch, and a symbiotic information interaction module. The symbiotic information interaction module is configured to realize information flow between the Diffusion-based image enhancement branch and the image segmentation branch, so that the Diffusion-based image enhancement branch generates the fundus photograph enhancement image based on the fundus photograph and information from the image segmentation branch, and at the same time, the image segmentation branch generates the fundus photograph retinal blood vessel segmentation image based on the fundus photograph and information from the Diffusion-based image enhancement branch.

[0084] The Diffusion-based fundus photograph enhancement segmentation device provided by the implementation of the present disclosure can be implemented by the Diffusion-based fundus photograph enhancement segmentation method provided by the implementation of the present disclosure, which is described above and will not be repeated here.

[0085] The Diffusion-based fundus photograph enhancement segmentation device provided by the implementation of the present disclosure can be implemented by the Diffusion-based fundus photograph enhancement segmentation method provided by the implementation of the present disclosure, which is described above and will not be repeated here.

[0086] Another implementation of the present disclosure relates to an electronic device comprising at least one processor and a memory connected to the at least one processor in communication. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the Diffusion-based fundus photograph enhancement segmentation method described in the above implementation.

[0087] The bus can include any number of interconnecting buses and bridges, allowing for a variety of configurations of peripheral devices, processors, and memories. The bus also can include various other circuits and devices such as power management, clock, and other control circuits, which are well known in the art, and therefore, will not be described further. A bus interface provides an interface between the bus and a transceiver. The transceiver can be a single device or a plurality of devices, such as a plurality of receivers and transmitters, which provide for communication with various other devices over a transmission medium. Processors process data, which can be transmitted over the bus. Further, the processors can receive data from the transceiver and transmit data to the transceiver.

[0088] The processor is responsible for managing the bus and general processing, including the execution of software stored on the memory. The memory can be used for storing data used by the processor, such as intermediate or final results.

[0089] Another embodiment of the present disclosure relates to a computer readable storage medium, storing a computer program, which, when executed by a processor, implements the Diffusion-based fundus color photograph enhancement segmentation method according to the above embodiment.

[0090] That is, those skilled in the art can understand that all or part of the steps of the method according to the above embodiment can be completed by a program instructing relevant hardware, and the program is stored in a storage medium, including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method according to the embodiments of the present disclosure. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0091] Another embodiment of the present disclosure relates to a computer program product, including a computer program, which, when executed by a processor, implements the Diffusion-based fundus color photograph enhancement segmentation method according to the above embodiment.

[0092] Those skilled in the art can understand that the above embodiments are specific embodiments of the present disclosure, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present disclosure.

Claims

1. A Diffusion-based fundus color photograph enhancement segmentation method, characterized in that, The Diffusion-based fundus color photograph enhancement segmentation method comprises: obtaining a fundus color photograph; inputting the fundus color photograph into a pre-trained Diffusion-based joint image enhancement segmentation model to obtain a fundus color photograph enhancement image and a fundus color photograph retinal blood vessel segmentation image; The Diffusion-based joint image enhancement segmentation model comprises a Diffusion-based image enhancement branch, an image segmentation branch, and a symbiotic information interaction module, which is used to realize information flow between the Diffusion-based image enhancement branch and the image segmentation branch, so that the Diffusion-based image enhancement branch generates the fundus color photograph enhancement image based on the fundus color photograph and information from the image segmentation branch, and at the same time, the image segmentation branch generates the fundus color photograph retinal blood vessel segmentation image based on the fundus color photograph and information from the Diffusion-based image enhancement branch.

2. The Diffusion-based fundus color photograph enhancement segmentation method according to claim 1, wherein, The Diffusion-based image enhancement branch is used to perform an image enhancement forward diffusion process and an image enhancement reverse diffusion process in a latent space; wherein The image enhancement forward diffusion process is used to add Gaussian noise to the fundus color photograph to generate a noise image; The image enhancement reverse diffusion process is used to generate the fundus color photograph enhancement image based on the noise image, input image encoding features obtained by performing convolution operation on the fundus color photograph, and semantic control features generated by the symbiotic information interaction module based on information from the image segmentation branch.

3. The Diffusion-based fundus color photograph enhancement segmentation method according to claim 2, wherein, The image enhancement reverse diffusion process is used to generate the fundus color photograph enhancement image based on the noise image, input image encoding features obtained by performing convolution operation on the fundus color photograph, and semantic control features generated by the symbiotic information interaction module based on information from the image segmentation branch, comprising: The image enhancement reverse diffusion process is used to: based on the input image encoding features and the noise image, systematically remove noise in the noise image by using an image denoiser, obtain a corresponding denoising result, maintain alignment between the denoising result and the input fundus color photograph, and perform multiple iterations on the denoising result and the semantic control features generated by the symbiotic information interaction module based on information from the image segmentation branch, and output the fundus color photograph enhancement image.

4. The Diffusion-based fundus color photograph enhancement segmentation method of claim 3, wherein, The image enhancement reverse diffusion process is realized by a latent diffusion model (LDM).

5. The Diffusion based fundus color enhancement segmentation method according to claim 3, wherein, The image segmentation branch is used to perform an image segmentation forward diffusion process and an image segmentation reverse diffusion process; The image segmentation forward diffusion process is used to adjust the encoding size by using a real segmentation mask, gradually introduce noise into the input fundus color photograph based on the adjusted encoding, iteratively add noise to the current state of the fundus color photograph to generate a low-quality image representation; The image segmentation reverse diffusion process is configured to generate a low-quality latent embedding conditioned on the low-quality image representation, and the low-quality latent embedding, the symbiotic information interaction module, a guiding feature generated based on information from the Diffusion-based image enhancement branch, and the input image encoding feature are iteratively processed by a U-Net segmentation module to output the fundus color photograph retinal blood vessel segmentation map.

6. The Diffusion-based fundus color photograph enhancement segmentation method according to claim 5, wherein, The symbiotic information interaction module is configured to generate the semantic control feature by a semantic control feature generation process, wherein the semantic control feature generation process comprises: The symbiotic information interaction module is configured to generate the guiding feature by a guiding feature generation process, wherein the guiding feature generation process comprises: The symbiotic information interaction module is configured to generate the guiding feature by a guiding feature generation process, wherein the guiding feature generation process comprises: The symbiotic information interaction module is configured to generate the guiding feature by a guiding feature generation process, wherein the guiding feature generation process comprises: The symbiotic information interaction module is configured to generate the guiding feature by a guiding feature generation process, wherein the guiding feature generation process comprises:

7. The Diffusion based fundus color enhancement segmentation method of claim 5, wherein, In the training process of the Diffusion-based image enhancement branch, the final result of image enhancement is supervised by a restoration loss. In the training process of the image segmentation branch, a weighted binary cross-entropy loss is used to optimize the final result of image segmentation. The Diffusion-based fundus color photograph enhancement segmentation device comprises: An acquisition module is configured to acquire a fundus color photograph. An image enhancement segmentation module is configured to input the fundus color photograph into a pre-trained Diffusion-based joint image enhancement segmentation model to obtain a fundus color photograph enhanced image and a fundus color photograph retinal blood vessel segmentation map.

8. The Diffusion-based fundus color photograph enhancement segmentation method according to any one of claims 1 to 7, characterized in that, An acquisition module is configured to acquire a fundus color photograph.

9. The Diffusion-based fundus color photograph enhancement segmentation method according to any one of claims 1 to 7, characterized in that, An image enhancement segmentation module is configured to input the fundus color photograph into a pre-trained Diffusion-based joint image enhancement segmentation model to obtain a fundus color photograph enhanced image and a fundus color photograph retinal blood vessel segmentation map.

10. A Diffusion-based fundus color photograph enhancement segmentation device, characterized by, ​ ​ ​ The Diffusion-based joint image enhancement and segmentation model comprises a Diffusion-based image enhancement branch, an image segmentation branch and a symbiotic information interaction module, the symbiotic information interaction module is used for realizing information flow between the Diffusion-based image enhancement branch and the image segmentation branch, so that the Diffusion-based image enhancement branch generates the fundus color image enhancement map based on the fundus color image and information from the image segmentation branch, and at the same time, the image segmentation branch generates the fundus color image retinal blood vessel segmentation map based on the fundus color image and information from the Diffusion-based image enhancement branch.

Images