Cross-modal eye fundus image generation method and system based on generative adversarial network

Abstract

The invention discloses a cross-modal eye fundus image generation method and system based on a generative adversarial network, relates to the technical field of medical image processing, and constructs an eye fundus focus perception and edge consistency generative adversarial network by taking a cyclic consistency generative adversarial network as a baseline. The core of the method is that a lesion perception mixed attention module is embedded in a bottleneck layer of a generator so as to strengthen the extraction capability of fine features of a lesion area; an edge information extraction module is designed, and key edge features are accurately extracted in combination with Roberts edge detection, wavelet transform and non-local mean denoising; and a joint loss function containing edge consistency loss is constructed, and the semantic consistency of a focus structure during cross-modal generation is ensured by minimizing the feature difference between the source image and the generated image. According to the method, the problems of disordered content, inconsistent structure and unstable training of the generated image in the prior art are effectively solved, and the simulation degree and clinical availability of the generated image are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a cross-modal fundus image generation method and system based on a generative adversarial network. BACKGROUND

[0002] Diabetic retinopathy (hereinafter referred to as "sugar net") is a retinal injury caused by diabetes and is one of the common complications of diabetes.

[0003] To support early diagnosis and treatment of sugar net, multi-modal fundus imaging technologies such as Color Fundus Photography (CFP), Optical Coherence Tomography (OCT), and Optical Coherence Tomography Angiography (OCTA) have become important imaging bases. The lesions of sugar net among different modal fundus images have relevance and complementarity, which can provide multi-angle imaging information for precise auxiliary diagnosis of sugar net and help clinicians fully grasp the characteristics of the lesions. However, due to the high cost of equipment procurement and maintenance, uneven distribution of clinical medical resources, and personal factors of patients, patients often can only obtain a certain modality or non-paired multi-modal fundus data when they undergo ophthalmic disease examination, making it difficult to fully play the auxiliary diagnostic value of multi-modal data. Therefore, designing a cross-modal fundus image generation method that can utilize the relevance and complementarity between different modal fundus images has important practical significance and application value for auxiliary clinical diagnosis and improving the performance of artificial intelligence fundus lesion analysis models.

[0004] In recent years, deep learning (DL) based technical solutions have been widely applied in the field of medical image cross-modal generation, and generative adversarial networks (GANs) have shown great application potential in medical image generation tasks such as fundus images. The quality and content simulation of medical images generated by GANs have been significantly improved, providing a technical direction for solving the problem of multi-modal fundus data acquisition.

[0005] Since the generative adversarial network (GAN) was proposed in 2014, it has undergone many improvements and innovations. With the advantages of low training cost and high quality of generated images, it has gained wide recognition among researchers in the field. The core goal of this model is to learn features from the inherent structure and distribution of the data, enabling the model to generate new samples similar to real data. In the network structure of GAN, the generator and the discriminator form an adversarial relationship: the goal of the generator is to generate as realistic fake samples as possible to deceive the discriminator, while the goal of the discriminator is to accurately determine whether the input sample is a real sample or a fake sample generated by the generator; both of them learn collaboratively in the training process, and finally reach a dynamic balance, so that the generator can achieve excellent sample generation performance. However, the GAN model has inherent technical defects, such as poor training process stability, easy occurrence of gradient explosion and mode collapse, which restricts its application in complex medical image generation scenarios.

[0006] To improve the above-mentioned defects of GAN, the cycle-consistent generative adversarial network (CycleGAN) is proposed as an improved image generation model based on GAN. The core design idea of CycleGAN is to learn the bidirectional mapping relationship between two image domains through two symmetrically set generators and two corresponding discriminators without the need for paired training data. At the same time, by introducing cycle consistency loss, the translation process of unpaired data is constrained to improve the content consistency of generated images and source images. However, CycleGAN still has obvious technical limitations: on the one hand, when the difference between the two image domains is large, the cycle consistency loss is difficult to effectively play a constraint role, leading to a decrease in the structural relevance of generated images and source images; on the other hand, compared to the basic GAN model, the training process of CycleGAN needs to more finely balance the performance of the generator and the discriminator. Even with the introduction of cycle consistency loss to guide the model to generate reasonable content, it is still prone to mode collapse and unstable training problems, which cannot meet the high-precision requirements of cross-modal fundus image generation.

[0007] Although existing GAN-based technical solutions have made some progress in the field of cross-modal fundus image generation, there are still three significant shortcomings in the generation task of diabetic retinopathy-related fundus images: 1. Insufficient content simulation CycleGAN only relies on cycle consistency loss function, which is difficult to make the model fully learn the content consistency relationship between different modal fundus images. This defect is particularly prominent when generating OCTA and other complex modal images: the generated fundus blood vessels are prone to unexpected deformation, fracture or blurred edges, and cannot accurately reproduce the fundus structure characteristics that should be in the target modal; in addition, the model often cannot correctly generate the sugar net lesion area, resulting in misinterpretation, confusion or even loss of key pathological features, which seriously reduces the simulation and clinical credibility of the generated images, and cannot meet the needs of auxiliary diagnosis.

[0008] 2. Poor model training stability The generative adversarial network itself has the characteristics of unstable training, and the fundus medical image contains rich fine structures, with high data complexity, which further amplifies this characteristic. The existing technical solutions generally face the risk of mode collapse: the generator tends to generate a few single types of fundus images, which cannot cover the rich pathological manifestations in the target modal, resulting in lack of diversity in the generated data, which is difficult to support the generalization training of subsequent artificial intelligence models.

[0009] 3. Weak model generalization ability The existing technical solutions have strong dependence on the distribution of the training data set: when the model is applied to fundus images from different acquisition devices, or to fundus images of rare diseases, its generalization ability decreases significantly-the generated images are prone to serious artifacts, or the image quality is sharply reduced, which cannot adapt to the clinical diversified image input scene.

[0010] Therefore, it is urgent to propose a new method that can enhance content consistency, improve training stability and improve generalization ability in the process of cross-modal fundus image generation. SUMMARY

[0011] To this end, the embodiment of the present application provides a cross-modal fundus image generation method and system based on a generative adversarial network, which is used to solve the generation disorder problem of the generative adversarial network applied to cross-modal fundus image generation in the prior art and the problem that the unsupervised generation network cannot maintain the cross-modal consistency of the lesion content.

[0012] In order to solve the above technical problems, the embodiment of the present application provides a cross-modal fundus image generation method based on a generative adversarial network, which comprises: constructing a fundus lesion perception and edge consistency generative adversarial network, the fundus lesion perception and edge consistency generative adversarial network taking a cycle consistency generative adversarial network as a baseline network, including two generators and two discriminators, for realizing bidirectional cross-modal mapping of two different modal fundus images; The bottleneck layer of the generator is embedded with a lesion perception hybrid attention module, which is used to strengthen the extraction ability of the generator for subtle texture and morphological features of the lesion area in the fundus image. An edge information extraction module is designed to extract key edge features of the source domain image / target domain image and the corresponding generated image, respectively. A joint loss function is defined, which includes a basic loss of a cycle consistency generative adversarial network and an edge consistency loss, wherein the basic loss of the cycle consistency generative adversarial network is composed of an adversarial loss, a cycle consistency loss and an identity loss, and the edge consistency loss is obtained by calculating the similarity of the edge features extracted by the edge information extraction module from the source domain / target domain image and the corresponding generated image. The fundus lesion perception and edge consistency generative adversarial network is trained using a multi-modal fundus dataset, and the parameters of the generator and the discriminator are optimized by minimizing the joint loss function. The trained fundus lesion perception and edge consistency generative adversarial network is used to input any modality of fundus image, and the corresponding generator is used to output the cross-modal fundus image of another modality.

[0013] Preferably, the generator includes an encoder, a bottleneck layer and a decoder: The encoder includes three sequentially connected down-sampling layers, each of which is composed of a CIL layer, each CIL layer is composed of a 3x3 convolution, an instance normalization and a Leaky ReLU activation function, and the input image is input into the encoder after being filled with boundary reflection, the encoder is used to extract image features layer by layer and reduce the size of the feature map; The bottleneck layer includes nine sequentially stacked residual blocks, each of which is composed of two cascaded reflection padding, two CIR layers and a residual connection, and the lesion perception hybrid attention module is embedded between the fourth residual block and the fifth residual block; The decoder includes two sequentially connected up-sampling layers, a reflection padding operation and an output layer, the up-sampling layer is composed of a 3x3 transpose convolution, an instance normalization and a ReLU activation function, and the output layer is composed of a 3x3 convolution and a Tanh activation function, which is used to restore the high-dimensional features to the original input image size.

[0014] Preferably, the lesion perception hybrid attention module includes channel attention, spatial attention and local importance attention, and the specific working process is as follows: The input feature map is The channel attention operation is performed to obtain the output feature map of the channel attention : The output feature map of the channel attention is The input spatial attention branch obtains an output feature map of spatial attention The output feature map of channel attention The input local importance attention branch obtains an output feature map of local importance attention ; The output feature map of spatial attention is fused with the output feature map of local importance attention to obtain a final output feature map of the lesion perception hybrid attention module .

[0015] Preferably, the input feature map is subjected to a channel attention operation to obtain an output feature map of channel attention , specifically including: The input feature map is subjected to global average pooling and global maximum pooling respectively in the spatial dimension to obtain two different channel description vectors; The two different channel description vectors are input into a weight-shared multi-layer perception machine for nonlinear transformation, and the transformed result is subjected to an addition operation; The addition result is subjected to a Sigmoid activation function to generate a channel attention weight vector; The channel attention weight vector is multiplied with the input feature map channel by channel to obtain an output feature map of channel attention .

[0016] Preferably, the output feature map of channel attention is input into a spatial attention branch to obtain an output feature map of spatial attention , specifically including: The output feature map of channel attention is subjected to global average pooling and global maximum pooling respectively in the channel dimension to obtain two single-channel feature maps; The two single-channel feature maps are spliced in the channel dimension to obtain a double-channel feature map; The double-channel feature map is subjected to 7x7 convolution for spatial information fusion, and then subjected to a Sigmoid activation function to generate a spatial attention weight mask; The spatial attention weight mask is multiplied with the output feature map of channel attention point by point to obtain an output feature map of spatial attention .

[0017] Preferably, the output feature map of channel attention Input the local importance attention branch to obtain the output feature map of the local importance attention. Specifically, it includes: Output feature map of channel attention Perform 1×1 convolutions to reduce the number of channels; The SoftPooling operation is performed on the dimensionality-reduced feature map to downsample it and preserve local feature information; The downsampling results are subjected to two 3×3 convolutions in sequence. The first convolution is used to extract high-dimensional features, and the second convolution is used to adjust the number of channels. A weight map is generated from the feature map after adjusting the number of channels using the Sigmoid activation function; The weight map is upsampled using bilinear interpolation to make its resolution similar to the output feature map of the channel attention. Consistent; The upsampled weight map is compared with the output feature map of channel attention. Element-wise multiplication yields the output feature map of the local importance attention. .

[0018] Preferably, the edge information extraction module includes two identical branches, one above the other, and the working process of each branch is as follows: Roberts edge detection is performed on the input image to calculate the gray-level difference between adjacent pixels in the image to initially extract edge information; Perform wavelet transform on the edge detection results to decompose and extract the edge feature components in the horizontal and vertical directions, and filter out low-frequency noise; The nonlocal mean denoising enhancement operation is performed on the feature components of the wavelet transform output to suppress noise while preserving the edge structure, thus obtaining the final edge features.

[0019] Preferably, the formula for calculating the edge consistency loss is: ; in, For edge consistency loss, , These represent the source domain image and the target domain image, respectively. express After generator The generated image, express After generator The generated image, EIEM(·) represents the edge information obtained after being extracted by the edge information extraction module. Describing the L1 norm, and Both represent mathematical expectation operations.

[0020] The formula for calculating the joint loss function is as follows: ; in, For the joint loss function, Generate the basic loss for the adversarial network to achieve cycle consistency. For edge consistency loss, represents the weighting coefficient for edge consistency loss.

[0021] This invention also provides a cross-modal fundus image generation system based on generative adversarial networks (GANs). This system is used to implement the aforementioned cross-modal fundus image generation method based on GANs, specifically including: The network construction module is used to construct a generative adversarial network for fundus lesion perception and edge consistency. The generative adversarial network for fundus lesion perception and edge consistency uses a cyclic consistency generative adversarial network as the baseline network and includes two generators and two discriminators to achieve bidirectional cross-modal mapping of fundus images of two different modalities. An attention embedding module is used to embed a lesion-aware hybrid attention module into the bottleneck layer of the generator. The lesion-aware hybrid attention module is used to enhance the generator's ability to extract subtle textures and morphological features of lesion areas in fundus images. An edge extraction module design module is used to design an edge information extraction module, which is used to extract key edge features of the source domain image / target domain image and the corresponding generated image, respectively. The loss function definition module is used to define the joint loss function, which includes the base loss and edge consistency loss of the recurrent consistency generative adversarial network. The base loss of the recurrent consistency generative adversarial network consists of adversarial loss, recurrent consistency loss and identity loss. The edge consistency loss is obtained by calculating the similarity between the edge features extracted by the edge information extraction module of the source domain / target domain image and the corresponding generated image. The network training module is used to train the fundus lesion perception and edge consistency generative adversarial network using a multimodal fundus dataset, and optimizes the parameters of the generator and discriminator by minimizing the joint loss function; The image generation module is used to utilize a trained fundus lesion perception and edge consistency generative adversarial network to take a fundus image of any modality as input and output a cross-modal fundus image of another modality through the corresponding generator.

[0022] As can be seen from the above technical solutions, this invention application has the following beneficial effects: (1) Enhance the model’s ability to capture lesion features: The present invention designs an embedded lesion perception hybrid attention module in the bottleneck layer of the generator to deeply mine the subtle texture and morphological information of lesions in fundus images. This effectively solves the problems of vascular deformation, breakage and blurring that are easy to occur when generating complex modalities such as angiography in the existing technology. At the same time, it avoids the misinterpretation or loss of key pathological features in the lesion area and significantly improves the content simulation and credibility of the generated image in the clinical auxiliary diagnosis scenario.

[0023] (2) Strengthening semantic association and enhancing cross-modal consistency of generated images: Addressing the challenge of unsupervised generative networks failing to maintain cross-modal consistency of lesions, this invention designs an edge information extraction module and an edge consistency loss. The edge information extraction module accurately extracts key edge information from the source and generated images through Roberts edge detection, wavelet transform, and nonlocal mean denoising enhancement. By minimizing the edge consistency loss, the generated image retains the integrity of core structures such as the optic disc, blood vessels, and lesions while conforming to the target modal characteristics. Compared to existing CycleGANs that rely solely on cycle consistency loss and are prone to "erroneous generation but correct reconstruction," this invention ensures semantic consistency across modalities from a structural perspective, making the pathological association between the generated and source images closer.

[0024] (3) Enhanced model generalization ability and adaptability to image generation in multiple scenarios: This invention reduces the model's dependence on the distribution of a specific dataset by accurately capturing lesion features through the edge information extraction module, stably extracting structural information through the edge information extraction module, and strictly constraining the generation process through the joint loss function. When the input data comes from images collected by different devices, the model can still generate high-quality images, demonstrating excellent generalization ability.

[0025] (4) Assisting in AI model optimization and promoting the intelligentization of medical imaging: The high-quality multimodal fundus images generated by this invention can be used to expand the fundus lesion dataset, solve the problem of scarce medical data, and thus improve the performance of fundus lesion analysis models based on deep learning. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of a cross-modal fundus image generation method based on generative adversarial networks provided by the present invention; Figure 2This is a schematic diagram of the overall structure of the adversarial network for fundus lesion perception and edge consistency generation in this invention; Figure 3 This is a schematic diagram of the generator designed in this invention; Figure 4 This is a schematic diagram of the lesion perception hybrid attention module designed in this invention; Figure 5 This is a schematic diagram of the edge information extraction module designed in this invention; Figure 6 This is a block diagram of a cross-modal fundus image generation system based on generative adversarial networks provided by the present invention; Figure 7 The results are comparative experiments of the Enface OCT of diabetic retinopathy and fundus color images generated in this invention, where (a) is the model input; (b) is the CycleGAN output; (c) is the StyleGAN-V2 output; (d) is the CUT output; (e) is the CycleGAN-turbo output; and (f) is the ReLBGAN output of this invention. Figure 8 These are the comparative experimental results of cross-modal generation of diabetic retinopathy OCTA and fundus color images in this invention, where (a) is the model input; (b) is the CycleGAN output; (c) is the StyleGAN-V2 output; (d) is the CUT output; (e) is the CycleGAN-turbo output; and (f) is the ReLBGAN output of this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example 1: To address the generation disorder issues that arise when using generative adversarial networks (GANs) for cross-modal fundus image generation in existing technologies, and the inability of unsupervised generative networks to maintain cross-modal consistency of lesion content, such as... Figure 1 As shown, this invention proposes a cross-modal fundus image generation method based on generative adversarial networks, which includes: S1: Construct a generative adversarial network for fundus lesion perception and edge consistency. The generative adversarial network for fundus lesion perception and edge consistency uses a recurrent consistency generative adversarial network as the baseline network and includes two generators and two discriminators to achieve bidirectional cross-modal mapping of fundus images of two different modalities. S2: Embed a lesion-aware hybrid attention module in the bottleneck layer of the generator. The lesion-aware hybrid attention module is used to enhance the generator's ability to extract subtle textures and morphological features of lesion areas in fundus images. S3: Design an edge information extraction module, which is used to extract key edge features of the source domain image / target domain image and the corresponding generated image respectively; S4: Define the joint loss function, which includes the base loss and edge consistency loss of the recurrent consistency generative adversarial network. The base loss of the recurrent consistency generative adversarial network consists of adversarial loss, recurrent consistency loss and identity loss. The edge consistency loss is obtained by calculating the similarity between the edge features extracted by the edge information extraction module of the source / target domain image and the corresponding generated image. S5: A generative adversarial network for fundus lesion perception and edge consistency is trained using a multimodal fundus dataset. The parameters of the generator and discriminator are optimized by minimizing the joint loss function. S6: Using the trained fundus lesion perception and edge consistency generative adversarial network, input a fundus image of any modality, and output a cross-modal fundus image of another modality through the corresponding generator.

[0029] As can be seen from the above technical solution, this invention proposes a cross-modal fundus image generation method based on generative adversarial networks. This method first constructs a Retinal Lesion-Aware and Boundary-Consistent Generative Adversarial Network (ReLBGAN) with a CycleGAN as the baseline network. Through its symmetrical structure of two generators and two discriminators, it achieves bidirectional cross-modal mapping between two different modalities of fundus images, laying the network foundation for high-quality image generation. Subsequently, a Lesion-Aware Hybrid Attention Module (LAHAM) is embedded in the bottleneck layer of the generator, enhancing the ability to extract subtle textures and morphological features of lesion regions in fundus images, effectively improving the lesion simulation accuracy of the generated images. Furthermore, an edge information extraction module is designed. The ExtractionModule (EIEM) accurately extracts key edge features from source and target domain images and their corresponding generated images, providing reliable feature support for maintaining structural consistency. By defining a joint loss function that includes a base loss and an edge consistency loss, where the edge consistency loss is obtained by calculating the similarity of edge features extracted by the edge information extraction module between the source / target domain image and the generated image, the structural distortion problem in cross-modal generation is effectively solved. The network is trained using a multimodal fundus dataset, and the parameters are optimized by minimizing the joint loss function, which significantly improves the training stability and generalization ability of the model. Finally, the trained network is used to achieve cross-modal generation from any modality fundus image to another modality. The generated images not only maintain the key pathological structures of the source image, but also conform to the visual features of the target modality, providing high-quality multimodal data support for clinical auxiliary diagnosis and AI model training.

[0030] In step S1, a generative adversarial network for fundus lesion perception and edge consistency (ReLBGAN) is constructed. The overall structure of ReLBGAN is as follows: Figure 2 As shown, ReLBGAN is based on the symmetric structure of CycleGAN and contains two generators: generator A2B and generator B2A, and two discriminators: discriminator A and discriminator B, which are used to realize cross-modal generation between fundus images A and B of two different modalities.

[0031] Specifically, the ReLBGAN model input consists of two generation branches, each composed of two generators cascaded together. Taking the branch for network input A as an example, A is transformed into a fake image B by generator A2B, and then fake image B is further transformed into fake image A by generator B2A. The branch for network input B has a symmetrical structure following the same generation process. This invention embeds a lesion-aware hybrid attention module (LAHAM) in the bottleneck layer of the generator, allowing the model to focus more on the detailed information of lesion regions in deep features. In terms of loss function design, to enhance the realism of the generated images, discriminators A and B calculate adversarial losses based on the real and generated images respectively, thereby improving the image quality of the generator. To ensure that the structural features of the original image are preserved during image translation, a cycle consistency loss is calculated to ensure that the input image can be restored to its original state as much as possible after bidirectional mapping. Meanwhile, to further enhance the preservation of structural information, this invention designs an edge consistency loss based on the proposed edge information extraction module (EIEM). By calculating the similarity between the edge features extracted by the edge information extraction module of the real image and the generated image, the generated image is made to maintain the consistency of content with the source image.

[0032] The following is combined Figures 3 to 5 Provide a detailed explanation of the structure of the ReLBGAN model.

[0033] The discriminator of this invention uses the existing PatchGAN structure. For example... Figure 3As shown, the generator designed in this invention consists of three parts: an encoder, a bottleneck layer, and a decoder. The input image is filled with boundary reflections and then input to the encoder. The encoder contains three sequentially connected downsampling layers. Each downsampling layer consists of a CIL layer (3×3 convolution + instance normalization + Leaky ReLU activation function). The encoder's function is to extract image features layer by layer and reduce the size of the feature map to adapt to the structure and processing requirements of deep networks. The bottleneck layer contains nine sequentially stacked residual blocks. Each residual block consists of two cascaded reflection fillers, two CIR layers (3×3 convolution + instance normalization + ReLU activation function), and residual connections. The reflection filler performs mirror patching on the image based on boundary pixels, reducing the loss and blurring of image edge information. The CIR layers extract features using 3×3 convolutions, preserve structural information such as contours and shapes using instance normalization, and introduce non-linearity using the ReLU activation function to enhance the model's expressive power. To enable the model to focus more on the detailed information of lesion regions in deep features, this invention embeds a lesion-aware hybrid attention module (LAHAM) between the fourth and fifth residual blocks. The decoder consists of two sequentially connected upsampling layers, a reflection padding operation, and an output layer. Each upsampling layer consists of a CTIR layer, and each CTIR layer is composed of a 3×3 transposed convolution, instance normalization, and a ReLU activation function. After the high-dimensional features pass through the two upsampling layers, they undergo reflection padding and enter the output layer, which consists of a 3×3 convolution and a Tanh activation function, restoring the features to the size of the original input image and completing the image generation process.

[0034] Furthermore, in step S2, to enhance the generator's ability to extract and represent lesion information features in the ReLBGAN model, this invention designs a lesion-aware hybrid attention module (LAHAM). This module combines local importance attention, channel attention, and spatial attention, integrating the advantages of global context modeling and local feature enhancement to improve the representation ability of lesion feature information while maintaining computational efficiency. Figure 4As shown, LAHAM employs a channel attention mechanism to weight the channels of the input feature map, thereby filtering out important channels and suppressing redundant information. Then, the feature maps filtered by channel attention are input into the spatial attention branch and the local importance attention branch, respectively. The spatial attention branch enhances the feature representation of key regions and suppresses irrelevant background information, guiding the model to focus on important regions in the image, while the local importance attention branch focuses on extracting detailed features, particularly enhancing the detailed features of important fundus structures such as lesions and blood vessels. Finally, the outputs of the spatial attention branch and the local importance attention branch are fused through a multiplication operation to obtain the output features of LAHAM, enabling the generator to generate more refined structural features such as lesions based on the learned important regions of the image. The composition of this hybrid attention module will be described in detail below.

[0035] (1) Channel attention: for the input feature map The channel attention module first processes the input feature map. Global average pooling and global max pooling are performed in the spatial dimension to obtain two different channel description vectors, summarizing the importance of each channel from the perspectives of spatial average and spatial maximum response, respectively. Then, the two vectors are passed through a weight-shared multilayer perceptron to perform a non-linear transformation on the channel information, uncovering the complex relationships between channels. They are then added together, and a channel attention weight vector is generated using a sigmoid activation function. This vector has the same length as the number of channels in the feature map, with each element corresponding to the weight of a channel. Finally, the input feature map is... Each channel is multiplied by its corresponding channel attention weight to obtain the output feature map of the channel attention. Output feature map of channel attention Compared to the input feature map Channels containing important information are enhanced, while relatively unimportant channels are suppressed. This process can be represented by the following formula: ; in, Indicates average pooling. This indicates max pooling. This represents a multilayer perceptron. This represents the Sigmoid activation function.

[0036] (2) Spatial attention: The feature map of the input after passing through the channel attention output. The input feature map enters a dual-branch attention path, one branch of which is a spatial attention branch. This branch generates spatial location weights for each spatial location in the input feature map, guiding the model to focus on important regions. In the processing, firstly, the input feature map... Global average pooling and global max pooling are performed along the channel dimension to capture the salient features and average response of each location along the channel dimension, generating two single-channel feature maps. These two feature maps are then concatenated along the channel dimension to form a dual-channel feature map, preserving the spatial dimensions of the original feature maps. Next, a 7×7 convolution is used to fuse spatial information in the concatenated feature map. This convolution operation effectively integrates the contextual relationships between different spatial locations through a weight-sharing mechanism of local receptive fields. The dual-channel feature spatial attention map is mapped to a range of 0 to 1 using the Sigmoid function. These values ​​represent the importance weight of each spatial location; the closer the value is to 1, the more critical the location is to the current task. Finally, the generated weight mask is multiplied point-by-point with the feature map that has undergone channel attention to obtain the output feature map of the spatial attention module. This process strengthens important features and weakens unimportant features. It can be represented by the following formula: ; in, This indicates a splicing operation. This represents a 7×7 convolution.

[0037] (3) Local importance attention: Output feature map of channel attention The other approach is the local importance attention module. First, the output feature map of the channel attention... After a 1×1 convolution, the number of channels is reduced to decrease computation. Then, the feature map undergoes a soft-pooling operation to preserve local feature information as much as possible while downsampling. The downsampled result is further downsampled using a 3×3 convolution for feature extraction to obtain high-dimensional feature information. This high-dimensional feature information is then subjected to a 3×3 convolution to adjust the number of channels, and a weight map is generated using the Sigmoid activation function. The weight map is then upsampled using bilinear interpolation to ensure its resolution matches that of the input feature map. Finally, the upsampled weight map is multiplied element-wise with the input feature map to obtain the output feature map of the local importance attention module. This enhances information in key local areas. This process can be represented by the following formula: ; ; in, and These represent 1×1 convolution and 3×3 convolution, respectively. This indicates a bilinear interpolation operation.

[0038] Finally, the feature maps obtained from the two branches are added together to obtain the output feature map of LAHAM. The calculation formula is as follows: .

[0039] Furthermore, in step S3, the erroneously generated fundus image differs significantly from the source image in terms of content structure. Edge information, however, can better reflect structural information and assist the model in extracting important detailed features such as blood vessels and lesions. Therefore, this invention employs a strategy of weak supervision based on image edge detection technology, designing an Edge Information Extraction Module (EIEM) and an edge consistency loss function that calculates the similarity between the output features of the source and generated images after passing through the EIEM, to supervise the training of the ReLBGAN model of this invention. Weak supervision based on edge information can provide the model with more accurate structural content information, assisting the model in more stably learning the features of structures such as the optic disc, blood vessels, and lesions. The structure of the EIEM designed in this invention is as follows: Figure 5 As shown, EIEM consists of two branches with identical structures, each comprising three key parts: Roberts edge detection, wavelet transform, and nonlocal means denoising enhancement. The source image and the generated image (output from the generator) are input into the upper and lower branches of EIEM, respectively. Each branch first uses the Roberts edge detection operator to calculate the gray-level difference between adjacent pixels in the image to detect edges and initially extract edge information. Then, to address the redundancy caused by the Roberts operator's sensitivity to noise, wavelet transform is used to perform multi-scale decomposition on the edge detection result image, extracting horizontal and vertical components while capturing edge features in both directions and filtering low-frequency noise. Finally, the output features of the wavelet transform are denoised and enhanced using a nonlocal means denoising algorithm. While preserving the edge structure, the weighted average of similar regions in the image effectively suppresses noise while maintaining the key edge feature information of the image, obtaining the horizontal and vertical edge information of the image as the output features of EIEM.

[0040] Further, in step S4, a joint loss function is defined, which includes the base loss and edge consistency loss of the recurrent consistency generative adversarial network.

[0041] (1) Basic loss function of CycleGAN This invention adopts the basic loss function of CycleGAN. The model consists of adversarial loss, cycle consistency loss, and identity loss. These three factors work together to enable the model to learn image mapping relationships more accurately. Adversarial loss prompts the generator to produce realistic transfer images, cycle consistency maintains the consistency and stability of semantic content during image translation, and identity loss enables the generator to better capture the inherent mapping relationship between the source and target domains, thus improving the quality of the generated images.

[0042] (2) Marginal Consistency Loss Function The cycle consistency loss function in CycleGAN's basic loss function primarily aims to ensure that the generated image obtained from the source image through generators A2B and B2A achieves the same consistency as the source image. Figure 1 This invention aims to address the issue of content consistency in generated images. However, in the scenario of cross-modal fundus image generation, the cyclic consistency loss function exhibits a significant flaw: an image is generated incorrectly but then correctly reconstructed. This allows the cyclic consistency loss to be optimized to a lower value, but it fails to help update the generator parameters. Therefore, the cyclic consistency loss cannot meet the requirement of maintaining content consistency before and after cross-modal fundus image generation in this invention.

[0043] Based on EIEM, this invention designs an edge consistency loss. The similarity between edge detection feature components corresponding to the real image and the generated image is calculated to guide the model in maintaining content consistency across modal generated images. The calculation formula is as follows: ; in, For edge consistency loss, , These represent the source domain image and the target domain image, respectively. express After generator The generated image, express After generator The generated image, EIEM(·) represents the edge information obtained after being extracted by the edge information extraction module. Describing the L1 norm, and Both represent mathematical expectation operations.

[0044] Edge consistency loss focuses on edge information of image structure, while the basic loss function of CycleGAN is used for image features such as texture and color. Therefore, this invention adopts... and joint loss function : ; in, The weighting coefficient for edge consistency loss is 1 in this embodiment.

[0045] In step S5, the ReLBGAN is trained using a multimodal fundus dataset, and the parameters of the generator and discriminator are optimized by minimizing the joint loss function.

[0046] In step S6, the trained ReLBGAN is used to input a fundus image of any modality and output a cross-modal fundus image of another modality through the corresponding generator.

[0047] Example 2: Figure 6 As shown, this invention provides a cross-modal fundus image generation system based on generative adversarial networks (GANs). This system is used to implement the cross-modal fundus image generation method based on GANs described in Embodiment 1 above, and includes: The network construction module 100 is used to construct a generative adversarial network for fundus lesion perception and edge consistency. The generative adversarial network for fundus lesion perception and edge consistency uses a cyclic consistency generative adversarial network as the baseline network and includes two generators and two discriminators to achieve bidirectional cross-modal mapping of fundus images of two different modalities. Attention embedding module 200 is used to embed lesion perception hybrid attention module into the bottleneck layer of generator. The lesion perception hybrid attention module is used to enhance the generator's ability to extract subtle textures and morphological features of lesion areas in fundus images. The edge extraction module design module 300 is used to design the edge information extraction module, which is used to extract the key edge features of the source domain image / target domain image and the corresponding generated image respectively; The loss function definition module 400 is used to define the joint loss function, which includes the basic loss and edge consistency loss of the recurrent consistency generative adversarial network. The basic loss of the recurrent consistency generative adversarial network consists of adversarial loss, recurrent consistency loss and identity loss. The edge consistency loss is obtained by calculating the similarity between the edge features extracted by the edge information extraction module of the source domain / target domain image and the corresponding generated image. The network training module 500 is used to train a generative adversarial network for fundus lesion perception and edge consistency using a multimodal fundus dataset, and optimizes the parameters of the generator and discriminator by minimizing the joint loss function; The image generation module 600 is used to utilize a trained fundus lesion perception and edge consistency generative adversarial network to input a fundus image of any modality and output a cross-modal fundus image of another modality through the corresponding generator.

[0048] This embodiment provides a cross-modal fundus image generation system based on generative adversarial networks (GANs) to implement the aforementioned cross-modal fundus image generation method based on GANs. Therefore, the specific implementation of the cross-modal fundus image generation system based on GANs can be found in the previous section on the embodiments of the cross-modal fundus image generation method based on GANs. For example, the network construction module 100, attention embedding module 200, edge extraction module design module 300, loss function definition module 400, network training module 500, and image generation module 600 are respectively used to implement steps S1, S2, S3, S4, S5, and S6 in the aforementioned cross-modal fundus image generation method based on GANs. Therefore, the specific implementation can be referred to the descriptions of the corresponding embodiments. To avoid redundancy, further details are omitted here.

[0049] To further illustrate the advantages of the method of this invention, the following description is provided in conjunction with specific simulation verification. The experiment includes an introduction to the dataset, evaluation metrics, and experimental results. These are described separately below: 1. Dataset To comprehensively evaluate the generalization ability, robustness, and quality of the generated images, this invention employed four fundus image datasets. For the training and testing of the ReLBGAN model, two multimodal fundus image datasets were used: Dataset 1, containing OCTA and fundus color images, and Dataset 2, containing Enface OCT and fundus color images. In downstream experiments, Dataset 3, containing Enface OCT images of diabetic retinopathy, and the publicly available fundus color image dataset APTOS2019 were used.

[0050] Dataset 1 consists of 104 pairs of fundus images from two modalities: OCTA and fundus color photography. It includes 102 pairs of normal images and 104 pairs of diabetic retinopathy images, all with a resolution of 512×512. In the generation experiment, the dataset was divided as follows: the training set included 84 pairs of normal images and 84 pairs of diabetic retinopathy images; the test set included 18 pairs of normal images and 20 pairs of diabetic retinopathy images.

[0051] Dataset 2 consists of 104 pairs of fundus images from two modalities: Enface OCT and fundus photography. It includes 102 pairs of normal images and 104 pairs of diabetic retinopathy images, all with a resolution of 512×512. In the generation experiment, the dataset was divided as follows: the training set included 84 pairs of normal images and 84 pairs of diabetic retinopathy images; the test set included 18 pairs of normal images and 20 pairs of diabetic retinopathy images.

[0052] Dataset 3: Contains 100 Enface OCT images of diabetic retinopathy (DR), primarily used to test the generalization ability of the ReLBGAN model and provide data augmentation for downstream experiments.

[0053] The APTOS 2019 dataset is a diabetic retinopathy grading dataset released by Kaggle, containing 3662 fundus color images, divided into five grades from 0 to 4. In the diabetic retinopathy grading experiment of this invention, the dataset was divided into a training set of 2922 images, a validation set of 359 images, and a test set of 361 images.

[0054] 2. Evaluation Indicators This invention uses Learned Perceptual Image Patch Similarity (LPIPS) and Fréchet Inception Distance (FID) as objective indicators to evaluate the quality of generated images. The smaller the FID distance value, the closer the feature distribution of the generated image is to that of the real image, and the better the image quality and diversity; the lower the LPIPS similarity score, the higher the perceptual similarity of the images.

[0055] In downstream diabetic reticulum grading experiments, accuracy, F1 score, Kappa coefficient, and recall were used as objective evaluation indicators.

[0056] 3. Results (1) Comparative experiment To verify the effectiveness of the proposed ReLBGAN model, it was compared with widely used unsupervised generative models. The comparison models mainly included CUT, styleGAN-V2, the fusion-diffusion model and CycleGAN-turbo, as well as the baseline network CycleGAN. Two sets of comparative experiments were set up: (a) cross-modal generation of diabetic retinas using EnfaceOCT and fundus photography; (b) cross-modal generation of diabetic retinas using OCTA and fundus photography.

[0057] (a) Cross-modal generation of diabetic retinopathy Enface OCT and fundus color images: Table 1 and Figure 7 Objective metrics and visualization results of cross-modal generation comparison experiments between diabetic retinopathy Enface OCT and fundus color images are presented. Experimental results show that the ReLBGAN method of this invention outperforms other comparative methods in generating fundus color images from diabetic retinopathy Enface OCT, particularly in terms of color, consistency of optic disc, blood vessels, and lesion content. The generated Enface OCT structure is reasonable, lesions are clear, and content consistency is better than other models. LPIPS similarity and FID distance metrics are also superior to other comparative methods.

[0058] Table 1. Comparative experiment of transmodal generation of diabetic retinopathy enface OCT and fundus photography.

[0059] (b) Cross-modal generation of diabetic retinopathy OCTA and fundus color images: Table 2 and Figure 8 Objective metrics and visualization results of cross-modal generation comparison experiments of diabetic retinopathy OCTA and fundus color images are presented. Experimental results show that the ReLBGAN method of this invention generates fundus color images with a more reasonable overall structure compared to other models in the experiment of generating fundus color images from diabetic retinopathy OCTA; the generated OCTA is not abrupt compared to the content generated by other methods, which alleviates the problem of poor content consistency caused by unsupervised training strategies to a certain extent, and generates high-quality diabetic retinopathy OCTA images without the problem of modality collapse, demonstrating stronger robustness compared to other comparison models.

[0060] Table 2 Comparative experiment of diabetic retinopathy OCTA and fundus photography cross-modal generation

[0061] (2) Ablation test This invention uses CycleGAN as the baseline network to verify the edge consistency loss function designed in this invention. To demonstrate the effectiveness of the hybrid attention module (LAHAM) for improving model generation performance, an ablation experiment was conducted on dataset 2 to generate diabetic retinopathy enface OCT and fundus color images across modalities.

[0062] Table 3 presents the ablation experimental results of cross-modal generation of diabetic retinopathy Enface OCT and fundus color images. The results show that when the baseline network is simultaneously coupled with the edge consistency loss function... Compared with LAHAM, the generated fundus color images and Enface OCT images show a higher degree of similarity to real images, especially in the generated fundus color images where the lesion structure and color are basically consistent with the real images. In the generated Enface OCT images, although the lesion structure and texture details are basically consistent, there are still some errors in the generation of lesion grayscale, but the performance is better in all objective indicators.

[0063] Table 3 Ablation experiments of transmodal generation of diabetic retinopathy enface OCT and fundus photography

[0064] (3) Downstream sugar reticulum grading experiment This experiment aims to generate Enface OCT images from fundus photographs using the ReLBGAN model, serving as a new modality to augment the APTOS2019 dataset and improve retinopathy grading performance. The grading model used is RETFound. First, using the ReLBGAN model trained on dataset 2, Enface OCT images were generated from fundus photographs in the APTOS 2019 training set (2922 images). The generated Enface OCTs were augmented to 20%, 40%, 60%, 80%, and 100% of the number of fundus photographs in the original training set. The retinopathy grading performance of the RETFound model under different augmentation ratios was compared to verify the effectiveness of the cross-modal data augmentation method. Table 4 shows the experimental results of retinopathy grading using cross-modal data augmentation of Enface OCT images with different augmentation ratios. As the Enface OCT augmentation ratio increases, the ACC and F1 scores of diabetic reticulum grading are continuously improved. When the augmentation ratio reaches 100%, all indicators reach their optimal levels, demonstrating the effectiveness of the cross-modal data augmentation method and showing that the ReLBGAN model of this invention can generate high-quality Enface OCT images.

[0065] Table 4. Results of Enface OCT expanded diabetic reticulum grading experiment generated from fundus color images.

[0066] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0069] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for generating cross-modal fundus images based on generative adversarial networks, characterized in that, include: A generative adversarial network for fundus lesion perception and edge consistency is constructed. The generative adversarial network for fundus lesion perception and edge consistency is based on a cyclic consistency generative adversarial network and includes two generators and two discriminators to achieve bidirectional cross-modal mapping of fundus images of two different modalities. A lesion-aware hybrid attention module is embedded in the bottleneck layer of the generator. The lesion-aware hybrid attention module is used to enhance the generator's ability to extract subtle textures and morphological features of lesion areas in fundus images. Design an edge information extraction module, which is used to extract key edge features of the source domain image / target domain image and the corresponding generated image respectively; Define a joint loss function, which includes the base loss and edge consistency loss of the recurrent consistency generative adversarial network. The base loss of the recurrent consistency generative adversarial network consists of adversarial loss, recurrent consistency loss and identity loss. The edge consistency loss is obtained by calculating the similarity between the edge features extracted by the edge information extraction module of the source / target domain image and the corresponding generated image. The generative adversarial network for fundus lesion perception and edge consistency was trained using a multimodal fundus dataset, and the parameters of the generator and discriminator were optimized by minimizing the joint loss function. Using a trained fundus lesion perception and edge consistency generative adversarial network, a fundus image of any modality is input and a cross-modal fundus image of another modality is output through the corresponding generator.

2. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The generator includes an encoder, a bottleneck layer, and a decoder; The encoder includes three sequentially connected downsampling layers. Each downsampling layer consists of a CIL layer. Each CIL layer consists of a 3×3 convolution, instance normalization, and Leaky ReLU activation function. The input image is filled with boundary reflections and then input into the encoder. The encoder is used to extract image features layer by layer and reduce the size of the feature map. The bottleneck layer includes nine stacked residual blocks, each residual block consisting of two cascaded reflection fillers, two CIR layers and residual connections, and the lesion perception hybrid attention module is embedded between the fourth and fifth residual blocks. The decoder includes two sequentially connected upsampling layers, a reflection fill operation, and an output layer. The upsampling layer consists of a 3×3 transposed convolution, instance normalization, and a ReLU activation function. The output layer consists of a 3×3 convolution and a Tanh activation function, used to restore high-dimensional features to the original input image size.

3. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1 or 2, characterized in that, The lesion perception hybrid attention module includes channel attention, spatial attention, and local importance attention. Its specific operation process is as follows: For the input feature map Perform channel attention operations to obtain the output feature map of channel attention. : The output feature map of the channel attention Input spatial attention branch to obtain spatial attention output feature map ; The output feature map of the channel attention Input the local importance attention branch to obtain the output feature map of the local importance attention. ; The output feature map of the spatial attention Output feature map of the local importance attention Feature fusion is performed to obtain the final output feature map of the lesion perception hybrid attention module. .

4. The cross-modal fundus image generation method based on generative adversarial networks according to claim 3, characterized in that, The input feature map Perform channel attention operations to obtain the output feature map of channel attention. Specifically, it includes: For the input feature map Perform global average pooling and global max pooling in the spatial dimension to obtain two different channel description vectors; Two different channel description vectors are input into a weight-shared multilayer perceptron for nonlinear transformation, and the transformed results are added together. The summation result is used to generate a channel attention weight vector through a Sigmoid activation function; The channel attention weight vector is compared with the input feature map. Channel-by-channel multiplication yields the output feature map of channel attention. .

5. The cross-modal fundus image generation method based on generative adversarial networks according to claim 3, characterized in that, The output feature map of the channel attention Input spatial attention branch to obtain spatial attention output feature map Specifically, it includes: Output feature map of channel attention Global average pooling and global max pooling are performed on the channel dimension respectively to obtain two single-channel feature maps; Two single-channel feature maps are concatenated along the channel dimension to obtain a two-channel feature map; Spatial information fusion is performed by performing a 7×7 convolution on the dual-channel feature map, and then a spatial attention weight mask is generated by using the Sigmoid activation function; The spatial attention weight mask and the output feature map of channel attention are compared. By multiplying point by point, we obtain the output feature map of spatial attention. .

6. The cross-modal fundus image generation method based on generative adversarial networks according to claim 3, characterized in that, The output feature map of the channel attention Input the local importance attention branch to obtain the output feature map of the local importance attention. Specifically, it includes: Output feature map of channel attention Perform 1×1 convolutions to reduce the number of channels; The SoftPooling operation is performed on the dimensionality-reduced feature map to downsample it and preserve local feature information; The downsampling results are subjected to two 3×3 convolutions in sequence. The first convolution is used to extract high-dimensional features, and the second convolution is used to adjust the number of channels. A weight map is generated from the feature map after adjusting the number of channels using the Sigmoid activation function; The weight map is upsampled using bilinear interpolation to make its resolution similar to the output feature map of the channel attention. Consistent; The upsampled weight map is compared with the output feature map of channel attention. Element-wise multiplication yields the output feature map of the local importance attention. .

7. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The edge information extraction module includes two identical branches, one above the other, and the working process of each branch is as follows: Roberts edge detection is performed on the input image to calculate the gray-level difference between adjacent pixels in the image to initially extract edge information; Perform wavelet transform on the edge detection results to decompose and extract the edge feature components in the horizontal and vertical directions, and filter out low-frequency noise; The nonlocal mean denoising enhancement operation is performed on the feature components of the wavelet transform output to suppress noise while preserving the edge structure, thus obtaining the final edge features.

8. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The formula for calculating the edge consistency loss is as follows: ； in, For edge consistency loss, , These represent the source domain image and the target domain image, respectively. express After generator The generated image, express After generator The generated image, EIEM(·) represents the edge information obtained after being extracted by the edge information extraction module. Describing the L1 norm, and Both represent mathematical expectation operations.

9. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The formula for calculating the joint loss function is as follows: ； in, For the joint loss function, Generate the basic loss for the adversarial network to achieve cycle consistency. For edge consistency loss, represents the weighting coefficient for edge consistency loss.

10. A cross-modal fundus image generation system based on generative adversarial networks, characterized in that, The system is used to implement the cross-modal fundus image generation method based on generative adversarial networks as described in any one of claims 1 to 9, comprising: The network construction module is used to construct a generative adversarial network for fundus lesion perception and edge consistency. The generative adversarial network for fundus lesion perception and edge consistency uses a cyclic consistency generative adversarial network as the baseline network and includes two generators and two discriminators to achieve bidirectional cross-modal mapping of fundus images of two different modalities. An attention embedding module is used to embed a lesion-aware hybrid attention module into the bottleneck layer of the generator. The lesion-aware hybrid attention module is used to enhance the generator's ability to extract subtle textures and morphological features of lesion areas in fundus images. An edge extraction module design module is used to design an edge information extraction module, which is used to extract key edge features of the source domain image / target domain image and the corresponding generated image, respectively. The loss function definition module is used to define the joint loss function, which includes the base loss and edge consistency loss of the recurrent consistency generative adversarial network. The base loss of the recurrent consistency generative adversarial network consists of adversarial loss, recurrent consistency loss and identity loss. The edge consistency loss is obtained by calculating the similarity between the edge features extracted by the edge information extraction module of the source domain / target domain image and the corresponding generated image. The network training module is used to train the fundus lesion perception and edge consistency generative adversarial network using a multimodal fundus dataset, and optimizes the parameters of the generator and discriminator by minimizing the joint loss function; The image generation module is used to utilize a trained fundus lesion perception and edge consistency generative adversarial network to take a fundus image of any modality as input and output a cross-modal fundus image of another modality through the corresponding generator.

Images