Cross-modal fundus image generation method and system based on generative adversarial network

Abstract

The application discloses a cross-modal fundus image generation method and system based on a generative adversarial network, relates to the technical field of medical image processing, and takes a cyclic consistency generative adversarial network as a baseline to construct a fundus lesion perception and edge consistency generative adversarial network. The core lies in that a lesion perception hybrid attention module is embedded in a generator bottleneck layer to strengthen the extraction capability of subtle features in a lesion area; an edge information extraction module is designed to accurately extract key edge features in combination with Roberts edge detection, wavelet transform and non-local mean denoising; and a joint loss function containing edge consistency loss is constructed to minimize the difference between source image features and generated image features, so as to ensure the semantic consistency of lesion structures during cross-modal generation. The application effectively solves the problems of image content disorder, inconsistent structure and unstable training in the prior art, and significantly improves the simulation degree and clinical usability of the generated image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a cross-modal fundus image generation method and system based on a generative adversarial network. BACKGROUND

[0002] Diabetic retinopathy (hereinafter referred to as "sugar net") is a retinal injury caused by diabetes and is one of the common complications of diabetes.

[0003] To support early diagnosis and treatment of sugar net, multi-modal fundus imaging technologies such as Color Fundus Photography (CFP), Optical Coherence Tomography (OCT), and Optical Coherence Tomography Angiography (OCTA) have become important imaging bases. The lesions of sugar net among different modal fundus images have relevance and complementarity, which can provide multi-angle imaging information for precise auxiliary diagnosis of sugar net and help clinicians fully grasp the characteristics of the lesions. However, due to the high cost of equipment procurement and maintenance, uneven distribution of clinical medical resources, and personal factors of patients, patients often can only obtain a certain modality or non-paired multi-modal fundus data when they undergo ophthalmic disease examination, making it difficult to fully play the auxiliary diagnostic value of multi-modal data. Therefore, designing a cross-modal fundus image generation method that can utilize the relevance and complementarity between different modal fundus images has important practical significance and application value for auxiliary clinical diagnosis and improving the performance of artificial intelligence fundus lesion analysis models.

[0004] In recent years, deep learning (DL) based technical solutions have been widely applied in the field of medical image cross-modal generation, and generative adversarial networks (GANs) have shown great application potential in medical image generation tasks such as fundus images. The quality and content simulation of medical images generated by GANs have been significantly improved, providing a technical direction for solving the problem of multi-modal fundus data acquisition.

[0005] Since the generative adversarial network (GAN) was proposed in 2014, it has undergone many improvements and innovations. With the advantages of low training cost and high quality of generated images, it has gained wide recognition among researchers in the field. The core goal of this model is to learn features from the inherent structure and distribution of the data, enabling the model to generate new samples similar to real data. In the network structure of GAN, the generator and the discriminator form an adversarial relationship: the goal of the generator is to generate as realistic fake samples as possible to deceive the discriminator, while the goal of the discriminator is to accurately determine whether the input sample is a real sample or a fake sample generated by the generator; both of them learn collaboratively in the training process, and finally reach a dynamic balance, so that the generator can achieve excellent sample generation performance. However, the GAN model has inherent technical defects, such as poor training process stability, easy occurrence of gradient explosion and mode collapse, which restricts its application in complex medical image generation scenarios.

[0006] To improve the above-mentioned defects of GAN, the cycle-consistent generative adversarial network (CycleGAN) is proposed as an improved image generation model based on GAN. The core design idea of CycleGAN is to learn the bidirectional mapping relationship between two image domains through two symmetrically set generators and two corresponding discriminators without the need for paired training data. At the same time, by introducing cycle consistency loss, the translation process of unpaired data is constrained to improve the content consistency of generated images and source images. However, CycleGAN still has obvious technical limitations: on the one hand, when the difference between the two image domains is large, the cycle consistency loss is difficult to effectively play a constraint role, leading to a decrease in the structural relevance of generated images and source images; on the other hand, compared to the basic GAN model, the training process of CycleGAN needs to more finely balance the performance of the generator and the discriminator, even if the cycle consistency loss is introduced to guide the model to generate reasonable content, it is still prone to mode collapse and unstable training problems, which cannot meet the high-precision requirements of cross-modal fundus image generation.

[0007] Although the existing GAN-based technical solutions have made some progress in the field of cross-modal fundus image generation, there are still three significant shortcomings in the generation task of diabetic retinopathy-related fundus images:

[0008] 1. Insufficient content simulation

[0009] CycleGAN only relies on cycle consistency loss function, which is difficult to make the model fully learn the content consistency relationship between different modal fundus images. This defect is particularly prominent when generating OCTA and other complex modal images: the generated fundus blood vessels are prone to unexpected deformation, fracture or blurred edges, and cannot accurately reproduce the fundus structure characteristics that should be in the target modal; in addition, the model often cannot correctly generate the sugar net lesion area, resulting in misinterpretation, confusion or even loss of key pathological features, which seriously reduces the simulation and clinical credibility of the generated images, and cannot meet the needs of auxiliary diagnosis.

[0010] 2. Poor model training stability

[0011] The generative adversarial network itself has the characteristics of unstable training, and the fundus medical image contains rich fine structures, and the data complexity is high, which further amplifies this characteristic. The existing technical solutions generally face the risk of mode collapse: the generator tends to generate a few single types of fundus images, which cannot cover the rich pathological manifestations in the target modal, resulting in lack of diversity of the generated data, which is difficult to support the generalization training of subsequent artificial intelligence models.

[0012] 3. Weak model generalization ability

[0013] The existing technical solutions have strong dependence on the distribution of the training data set: when the model is applied to fundus images from different acquisition devices, or to fundus images of rare diseases, its generalization ability decreases significantly - the generated images are prone to serious artifacts, or the image quality is significantly reduced, which cannot adapt to the clinical diversified image input scene.

[0014] Therefore, it is urgent to propose a new method that can enhance content consistency, improve training stability and improve generalization ability in the process of cross-modal fundus image generation. SUMMARY

[0015] To this end, the embodiments of the present application provide a cross-modal fundus image generation method and system based on a generative adversarial network, which is used to solve the generation disorder problem of the generative adversarial network applied to cross-modal fundus image generation in the prior art and the problem that the unsupervised generation network cannot maintain the cross-modal consistency of the lesion content.

[0016] To solve the above technical problems, the embodiments of the present application provide a cross-modal fundus image generation method based on a generative adversarial network, which comprises:

[0017] An eye fundus lesion perception and edge consistency generative adversarial network is constructed, which takes a cycle consistency generative adversarial network as a baseline network, includes two generators and two discriminators, and is used to realize bidirectional cross-modal mapping of two different modal fundus images;

[0018] The bottleneck layer of the generator is embedded with a lesion perception hybrid attention module, which is used to enhance the ability of the generator to extract subtle texture and morphological features of the lesion area in the fundus image;

[0019] An edge information extraction module is designed to extract key edge features of the source domain image / target domain image and the corresponding generated image, respectively;

[0020] A joint loss function is defined, which includes a basic loss of a cycle consistency generative adversarial network and an edge consistency loss, wherein the basic loss of the cycle consistency generative adversarial network is composed of an adversarial loss, a cycle consistency loss and an identity loss, and the edge consistency loss is obtained by calculating the similarity of the edge features extracted by the edge information extraction module from the source domain / target domain image and the corresponding generated image;

[0021] The fundus lesion perception and edge consistency generative adversarial network is trained using a multi-modal fundus dataset, and the parameters of the generator and the discriminator are optimized by minimizing the joint loss function;

[0022] Using the trained fundus lesion perception and edge consistency generative adversarial network, any modality of fundus image is input, and another modality of cross-modal fundus image is output through the corresponding generator.

[0023] Preferably, the generator includes an encoder, a bottleneck layer and a decoder:

[0024] The encoder includes three sequentially connected down-sampling layers, each of which is composed of a CIL layer, each CIL layer is composed of a 3x3 convolution, an instance normalization and a Leaky ReLU activation function, and the input image is input into the encoder after being filled with boundary reflection, the encoder is used to extract image features layer by layer and reduce the size of the feature map;

[0025] The bottleneck layer includes nine sequentially stacked residual blocks, each of which is composed of two cascaded reflection padding, two CIR layers and a residual connection, and the lesion perception hybrid attention module is embedded between the fourth residual block and the fifth residual block;

[0026] The decoder includes two sequentially connected up-sampling layers, a reflection padding operation and an output layer, the up-sampling layer is composed of a 3x3 transpose convolution, an instance normalization and a ReLU activation function, and the output layer is composed of a 3x3 convolution and a Tanh activation function, which is used to restore the high-dimensional features to the original input image size.

[0027] Preferably, the lesion perception hybrid attention module includes channel attention, spatial attention and local importance attention, and the specific working process is:

[0028] input feature map performing a channel attention operation to obtain an output feature map of channel attention :

[0029] inputting the output feature map of channel attention inputting a spatial attention branch to obtain an output feature map of spatial attention ; inputting the output feature map of channel attention inputting a local importance attention branch to obtain an output feature map of local importance attention ;

[0030] inputting the output feature map of spatial attention and the output feature map of local importance attention performing feature fusion to obtain a final output feature map of the lesion perception hybrid attention module .

[0031] Preferably, the input feature map performing a channel attention operation to obtain an output feature map of channel attention , specifically comprising:

[0032] input feature map performing global average pooling and global maximum pooling respectively in the spatial dimension to obtain two different channel description vectors;

[0033] inputting the two different channel description vectors into a weight-shared multi-layer perceptron to perform nonlinear transformation, and performing an addition operation on the transformed result;

[0034] generating a channel attention weight vector by Sigmoid activation function on the addition result;

[0035] multiplying the channel attention weight vector and the input feature map channel by channel to obtain an output feature map of channel attention .

[0036] Preferably, the input feature map of channel attention inputting a spatial attention branch to obtain an output feature map of spatial attention , specifically comprising:

[0037] inputting the output feature map of channel attention performing global average pooling and global maximum pooling respectively in the channel dimension to obtain two single-channel feature maps;

[0038] splicing the two single-channel feature maps in the channel dimension to obtain a double-channel feature map;

[0039] performing 7*7 convolution on the double-channel feature map to fuse spatial information, and generating a spatial attention weight mask through a Sigmoid activation function;

[0040] multiplying the spatial attention weight mask with the output feature map of the channel attention to obtain the output feature map of the spatial attention .

[0041] Preferably, the output feature map of the channel attention is input into a local importance attention branch to obtain an output feature map of the local importance attention , and specifically includes:

[0042] performing 1*1 convolution on the output feature map of the channel attention to reduce the number of channels;

[0043] performing SoftPooling operation on the reduced-dimension feature map to down-sample and reserve local feature information;

[0044] performing 3*3 convolution twice on the down-sampled result, the first time for extracting high-dimensional features and the second time for adjusting the number of channels;

[0045] generating a weight map through a Sigmoid activation function on the feature map after adjusting the number of channels;

[0046] performing bilinear interpolation up-sampling on the weight map to make its resolution consistent with the output feature map of the channel attention ;

[0047] multiplying the up-sampled weight map with the output feature map of the channel attention element by element to obtain the output feature map of the local importance attention .

[0048] Preferably, the edge information extraction module includes an upper branch and a lower branch which are structurally identical, and the working process of each branch is as follows:

[0049] performing Roberts edge detection on the input image to calculate the gray difference between adjacent pixels in the image to preliminarily extract edge information;

[0050] performing wavelet transform on the edge detection result to decompose and extract edge feature components in the horizontal and vertical directions and filter low-frequency noise;

[0051] performing non-local mean denoising enhancement operation on the feature components output by the wavelet transform to suppress noise while preserving edge structure to obtain final edge features.

[0052] ​Preferably, the calculation formula of the edge consistency loss is:

[0053] ;

[0054] wherein, is the edge consistency loss, , respectively represent the source domain image and the target domain image, represents the image generated by the generator , represents the image generated by the generator , and EIEM(·) represents the edge information extracted by the edge information extraction module, represents the L1 norm, and both represent the mathematical expectation operation.

[0055] The calculation formula of the joint loss function is:

[0056] ;

[0057] wherein, is the joint loss function, is the basic loss of the cycle consistency generative adversarial network, is the edge consistency loss, is the weight coefficient of the edge consistency loss.

[0058] The embodiment of the application also provides a cross-modal fundus image generation system based on a generative adversarial network, which is used to realize the cross-modal fundus image generation method based on a generative adversarial network.

[0059] The network construction module is used to construct a fundus lesion perception and edge consistency generative adversarial network, and the fundus lesion perception and edge consistency generative adversarial network takes the cycle consistency generative adversarial network as a baseline network and includes two generators and two discriminators, and is used to realize bidirectional cross-modal mapping of two different modal fundus images.

[0060] The attention embedding module is used to embed a lesion perception hybrid attention module in the bottleneck layer of the generator, and the lesion perception hybrid attention module is used to strengthen the extraction ability of the generator to the subtle texture and morphological features of the lesion area in the fundus image.

[0061] The edge extraction module design module is used to design an edge information extraction module, and the edge information extraction module is used to extract key edge features of the source domain image / target domain image and the corresponding generated image, respectively.

[0062] The loss function definition module is configured to define a joint loss function, which includes a cycle consistency generative adversarial network basic loss and an edge consistency loss, wherein the cycle consistency generative adversarial network basic loss is composed of an adversarial loss, a cycle consistency loss and an identity loss, and the edge consistency loss is obtained by calculating the similarity of the edge features of the source domain / target domain image and the corresponding generated image extracted by the edge information extraction module.

[0063] The network training module is configured to train the fundus lesion perception and edge consistency generative adversarial network using a multi-modal fundus data set, and optimize the parameters of the generator and the discriminator by minimizing the joint loss function.

[0064] The image generation module is configured to input any modality of fundus image into the trained fundus lesion perception and edge consistency generative adversarial network, and output a cross-modal fundus image of another modality through the corresponding generator.

[0065] From the above technical solutions, the present application has the following beneficial effects:

[0066] (1) Improve the lesion feature capturing ability of the model: the present application designs an embedded lesion perception hybrid attention module in the generator bottleneck layer, deeply excavates the subtle texture and morphology information of the lesion in the fundus image, effectively solves the problems of blood vessel deformation, fracture and blur that are prone to occur when generating complex modalities such as angiography in the prior art, avoids misinterpretation or loss of key pathological features in the lesion area, and significantly improves the content simulation and credibility of the generated image in the clinical auxiliary diagnosis scene.

[0067] (2) Strengthen semantic association and enhance cross-modal consistency of generated images: in view of the pain point that unsupervised generative network is difficult to maintain cross-modal consistency of lesions, the present application designs an edge information extraction module and an edge consistency loss. The edge information extraction module accurately extracts the key edge information of the source image and the generated image through Roberts edge detection, wavelet transform and non-local mean denoising enhancement, and minimizes the edge consistency loss to make the generated image conform to the characteristics of the target modality while retaining the integrity of the core structures such as optic disc, blood vessels and lesions. Compared with the existing CycleGAN which only relies on the cycle consistency loss and is prone to the vulnerability of "wrong generation but correct reconstruction", the present application ensures the semantic consistency of cross-modal generation from the structural level, making the pathological association between the generated image and the source image closer.

[0068] (3) Enhance the generalization ability of the model and adapt to multiple scene image generation: The present application reduces the dependence of the model on the distribution of specific data sets through the precise capture of lesion characteristics by the edge information extraction module, the stable extraction of structural information by the edge information extraction module, and the strict constraint of the generation process by the joint loss function. When the input data comes from images collected by different devices, the model can still generate high-quality images and has excellent generalization ability.

[0069] (4) Assist AI model optimization and promote medical image intelligence: The high-quality multi-modal fundus images generated by the present application can be used to expand the fundus lesion data set and solve the problem of medical data scarcity, thereby improving the performance of fundus lesion analysis models based on deep learning. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly described below. The features and advantages of the present application can be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as any limitation on the present application. Those skilled in the art can obtain other drawings according to these drawings without creative labor. Among them:

[0071] Figure 1 is a flowchart of a cross-modal fundus image generation method based on a generative adversarial network provided by the present application;

[0072] Figure 2 is a schematic diagram of the overall structure of the fundus lesion perception and edge consistency generative adversarial network in the present application;

[0073] Figure 3 is a schematic diagram of the structure of the generator designed by the present application;

[0074] Figure 4 is a schematic diagram of the structure of the lesion perception hybrid attention module designed by the present application;

[0075] Figure 5 is a schematic diagram of the structure of the edge information extraction module designed by the present application;

[0076] Figure 6 is a block diagram of a cross-modal fundus image generation system based on a generative adversarial network provided by the present application;

[0077] Figure 7are the comparative experimental results of the sugar network Enface OCT and fundus color photograph cross-modal generation in the present application, wherein (a) is the model input; (b) is the CycleGAN output; (c) is the StyleGAN-V2 output; (d) is the CUT output; (e) is the CycleGAN-turbo output; and (f) is the ReLBGAN output of the present application.

[0078] Figure 8 are the comparative experimental results of the sugar network OCTA and fundus color photograph cross-modal generation in the present application, wherein (a) is the model input; (b) is the CycleGAN output; (c) is the StyleGAN-V2 output; (d) is the CUT output; (e) is the CycleGAN-turbo output; and (f) is the ReLBGAN output of the present application. DETAILED DESCRIPTION

[0079] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0080] Embodiment one: in order to solve the generation disorder problem of the generative adversarial network applied to cross-modal fundus image generation and the problem that the unsupervised generative network cannot maintain the cross-modal consistency of the lesion content in the prior art, as shown in Figure 1 The present application proposes a cross-modal fundus image generation method based on a generative adversarial network, which comprises the following steps:

[0081] S1: constructing a fundus lesion perception and edge consistency generative adversarial network, the fundus lesion perception and edge consistency generative adversarial network taking a cycle consistency generative adversarial network as a baseline network, comprising two generators and two discriminators, and being used for realizing bidirectional cross-modal mapping of two different modal fundus images;

[0082] S2: embedding a lesion perception hybrid attention module in the bottleneck layer of the generator, the lesion perception hybrid attention module being used for strengthening the extraction ability of the generator to the subtle texture and morphological features of the lesion area in the fundus image;

[0083] S3: designing an edge information extraction module, the edge information extraction module being used for respectively extracting key edge features of the source domain image / target domain image and the corresponding generated image;

[0084] S4: defining a joint loss function, the joint loss function comprising a cycle-consistency generative adversarial network basic loss and an edge consistency loss, wherein the cycle-consistency generative adversarial network basic loss is composed of an adversarial loss, a cycle consistency loss and an identity loss, and the edge consistency loss is obtained by calculating the similarity of the edge features extracted by the edge information extraction module from the source domain / target domain image and the corresponding generated image;

[0085] S5: training the fundus lesion perception and edge consistency generative adversarial network by using a multi-modal fundus data set, and optimizing the parameters of the generator and the discriminator by minimizing the joint loss function;

[0086] S6: using the trained fundus lesion perception and edge consistency generative adversarial network, inputting any modality of fundus image, and outputting another modality of cross-modal fundus image through the corresponding generator.

[0087] From the above technical scheme, the application proposes a cross-modal fundus image generation method based on a generative adversarial network. The method first constructs a fundus lesion-aware and boundary-consistent generative adversarial network (ReLBGAN) based on a cycle-consistent generative adversarial network (CycleGAN) as a baseline network. Through the two generators and two discriminators of the symmetric structure, bidirectional cross-modal mapping between two different modal fundus images is realized, laying the network foundation for high-quality image generation. Then, a lesion-aware hybrid attention module (LAHAM) is embedded in the generator bottleneck layer, which strengthens the extraction ability of subtle texture and morphological features in the lesion area of the fundus image, effectively improving the lesion simulation degree of the generated image. Further, an edge information extraction module (EIEM) is designed to accurately extract the key edge features of the source domain and target domain images and their corresponding generated images, providing reliable feature support for maintaining structural consistency. A joint loss function is defined, which includes a basic loss and an edge consistency loss. The edge consistency loss is obtained by calculating the similarity of the edge features extracted by the edge information extraction module from the source / target domain images and the generated images, effectively solving the structural distortion problem in cross-modal generation. A multi-modal fundus dataset is used to train the network and optimize the parameters by minimizing the joint loss function, significantly improving the training stability and generalization ability of the model. Finally, the trained network is used to realize cross-modal generation of any modal fundus image to another modal, and the generated image not only maintains the key pathological structure of the source image, but also conforms to the visual features of the target modal, providing high-quality multi-modal data support for clinical auxiliary diagnosis and AI model training.

[0088] In step S1, a fundus lesion-aware and boundary-consistent generative adversarial network (ReLBGAN) is constructed. The overall structure of ReLBGAN is shown in Figure 2 The ReLBGAN is based on the symmetric structure of CycleGAN and includes two generators: generator A2B and generator B2A, and two discriminators: discriminator A and discriminator B, which are used to realize cross-modal generation between two different modal fundus images A and B.

[0089] Specifically, the ReLBGAN model input is composed of two generation branches, and each branch is composed of two generators in series. Taking the branch of network input A as an example, A is converted into generated image fake B through generator A2B, and then image fake B is further converted into image fake A through generator B2A. The branch of network input B is a symmetric structure following the same generation process. The present application embeds a lesion-aware hybrid attention module (LAHAM) in the generator bottleneck layer, so that the model pays more attention to the details of the lesion area in the deep feature. In terms of loss function design, in order to enhance the authenticity of the generated image, discriminators A and B respectively calculate the adversarial loss according to the real image and the generated image to improve the image quality of the generator. In order to ensure the structure features of the original image are preserved in the image translation process, the cycle consistency loss is calculated to ensure that the input image can be restored to the original state as much as possible after bidirectional mapping. At the same time, in order to further strengthen the structure information, the present application designs an edge consistency loss based on the edge information extraction module (EIEM) proposed by the present application, which calculates the similarity between the edge features extracted from the real image and the generated image through the edge information extraction module, so as to promote the consistency between the generated image and the source image.

[0090] The structure of the ReLBGAN model will be described in detail below. Figures 3 to 5 The structure of the ReLBGAN model will be described in detail below.

[0091] The discriminator of the present application adopts the existing PatchGAN structure. As shown in FIG. 2, the discriminator is composed of a feature extraction layer and a PatchGAN layer. The feature extraction layer is used to extract the feature of the input image, and the PatchGAN layer is used to calculate the adversarial loss. Figure 3As shown, the generator designed by the application is composed of an encoder, a bottleneck layer and a decoder. The input image is input into the encoder after being filled by boundary reflection. The encoder includes three sequentially connected down-sampling layers, each of which is composed of a CIL layer (3x3 convolution + instance normalization + Leaky ReLU activation function). The role of the encoder is to extract the features of the image layer by layer and reduce the size of the feature map to adapt to the structure and processing requirements of the deep network. The bottleneck layer includes nine residual blocks stacked in turn, each of which is composed of two cascaded reflection filling and two CIR layers (3x3 convolution + instance normalization + ReLU activation function) and residual connection. Among them, the reflection filling is based on the boundary pixels to mirror patch filling of the image, which reduces the loss and blur of the image edge information. The CIR layer extracts features by 3x3 convolution, retains structure information such as outline and shape by instance normalization, and introduces nonlinearity by ReLU activation function to enhance the expression ability of the model. In order to make the model pay more attention to the detail information of the lesion area in the deep feature, the application designs a lesion-aware hybrid attention module (LAHAM) embedded between the fourth residual block and the fifth residual block. The decoder includes two sequentially connected up-sampling layers, a reflection filling operation and an output layer. Each up-sampling layer is composed of a CTIR layer, and each CTIR layer is composed of a 3x3 transpose convolution, an instance normalization and a ReLU activation function. After the high-dimensional features pass through the two up-sampling layers, they are filled by reflection, enter the output layer composed of 3x3 convolution and Tanh activation function, restore the features to the size of the original input image, and complete the image generation process.

[0092] Further, in step S2, in order to enhance the feature extraction and expression generation ability of the generator in the ReLBGAN model, the application designs a lesion-aware hybrid attention module (LAHAM), which is composed of local importance attention, channel attention and spatial attention. The module integrates the advantages of global context modeling and local feature enhancement, improves the lesion feature information representation ability while maintaining the computing efficiency. As shown in FIG. 4, the LAHAM is composed of local importance attention, channel attention and spatial attention. Figure 4As shown, LAHAM adopts a channel attention mechanism to weight the channels of the input feature map, thereby screening important channels and suppressing redundant information. Then, the feature map screened by the channel attention is input into the spatial attention branch and the local importance attention branch, respectively. The spatial attention branch enhances the feature expression of key regions and suppresses irrelevant background information, guiding the model to focus on important regions in the image, while the local importance attention branch focuses on the extraction of detailed features, especially in the important structure regions of the fundus such as lesions and blood vessels. Finally, the outputs of the spatial attention branch and the local importance attention branch are fused by multiplication operation to obtain the output feature of LAHAM, so that the generator can generate more detailed structure features such as lesions on the basis of learning the important regions of the image. Next, the composition of the hybrid attention module is described in detail.

[0093] (1) Channel attention: input feature map The channel attention module first performs global average pooling and global maximum pooling on the input feature map in the spatial dimension to obtain two different channel description vectors, which summarize the importance of each channel from the perspectives of spatial average and spatial maximum response, respectively. Then, the two vectors are nonlinearly transformed by a weight-shared multi-layer perceptron to mine the complex relationships between channels, and then added and activated by a Sigmoid function to generate a channel attention weight vector, which has the same length as the number of channels of the feature map, and each element corresponds to the weight of a channel. Finally, each channel of the input feature map is multiplied by the corresponding channel attention weight to obtain the output feature map of the channel attention. The output feature map of the channel attention contains important information compared to the input feature map , and the relatively unimportant channels are suppressed. This process can be represented by the following formula:

[0094] ;

[0095] wherein represents average pooling, represents maximum pooling, represents multi-layer perceptron, represents Sigmoid activation function.

[0096] (2) Spatial attention: input feature map into a dual-branch attention passage, one branch of which is a spatial attention branch that generates spatial position weights for each spatial position of the input feature map to guide the model to focus on important regions. During processing, first, the input feature map Global average pooling and global max pooling are performed on the channel dimension to capture the significant features and average responses of each position in the channel dimension, respectively, to generate two single-channel feature maps. Subsequently, the two feature maps are spliced in the channel dimension to form a dual-channel feature map that retains the spatial dimensions of the original feature map. Next, spatial information fusion is performed on the spliced feature map through a 7x7 convolution, which effectively integrates the context relationships between different spatial positions through the weight sharing mechanism of the local receptive field. The spatial attention map is mapped to a value range of 0 to 1 through a Sigmoid function, and these values represent the importance weights of each spatial position, with values closer to 1 indicating that the position is more critical to the current task. Finally, the generated weight mask is multiplied point-by-point with the feature map after channel attention to obtain the output feature map of the spatial attention module , which achieves the strengthening of important features and the weakening of unimportant features. This process can be represented by the following formula:

[0097] ;

[0098] wherein, represents a splicing operation, represents a 7x7 convolution.

[0099] (3) Local importance attention: the output feature map of the channel attention enters the other branch, which is the local importance attention module. First, the output feature map of the channel attention is subjected to a 1x1 convolution to reduce the number of channels and reduce the computational load. Subsequently, the feature map is subjected to a SoftPooling operation to downsample while preserving as much local feature information as possible. The downsampled result is further downsampled through a 3x3 convolution to extract high-dimensional feature information. The obtained high-dimensional feature information is subjected to a 3x3 convolution to adjust the number of channels, and a weight map is generated through a Sigmoid activation function . The weight map is then upsampled through bilinear interpolation to ensure that its resolution is consistent with that of the input feature map. Finally, the upsampled weight map is multiplied element-by-element with the input feature map to obtain the output feature map of the local importance attention module , which achieves the enhancement of local key region information. This process can be represented by the following formula:

[0100] ;

[0101] ;

[0102] wherein, and denote 1x1 convolution and 3x3 convolution respectively, denotes bilinear interpolation operation.

[0103] Finally, the feature maps obtained by the two branches are added to obtain the output feature map of LAHAM , and the calculation formula is:

[0104] .

[0105] Further, in step S3, the error generated fundus image has a large gap with the content structure of the source image itself, and the edge information can better reflect the structural information and can assist the model to better extract important detail features such as blood vessels and lesions. Therefore, the application adopts a strategy of weak supervision of the model based on image edge detection technology, designs an edge information extraction module (EIEM), and designs an edge consistency loss function for calculating the output features of the source image and the generated image after EIEM to supervise the training of the ReLBGAN model of the application. The weak supervision guided based on edge information can provide more accurate structural content information for the model, and assist the model to more stably learn the features of structures such as optic disc, blood vessels and lesions. The structure of the EIEM designed by the application is shown in Figure 5 , the EIEM is divided into upper and lower branches with the same structure, and each branch is composed of three key parts of Roberts edge detection, wavelet transform and non-local mean denoising enhancement. The source image and the generated image output by the generator are input into the upper and lower branches of the EIEM respectively, and each branch first calculates the gray difference of adjacent pixels in the image by the Roberts edge detection operator to detect the edge and preliminarily extract the edge information. Then, in view of the redundant information caused by the sensitivity of the Roberts operator to noise, wavelet transform is adopted to perform multi-scale decomposition on the edge detection result image, and the horizontal and vertical components are extracted to filter low-frequency noise while capturing the edge features in the two directions. Finally, the output features of the wavelet transform are denoised and enhanced by the non-local mean denoising algorithm, which effectively suppresses noise by using the weighted average of similar areas in the image while preserving the key edge feature information of the image, and obtains the horizontal and vertical edge information of the image as the output features of the EIEM.

[0106] Further, in step S4, a joint loss function is defined, including the basic loss of the cycle consistency generative adversarial network and the edge consistency loss.

[0107] (1) Basic loss function of CycleGAN

[0108] This invention adopts the basic loss function of CycleGAN. The model consists of adversarial loss, cycle consistency loss, and identity loss. These three factors work together to enable the model to learn image mapping relationships more accurately. Adversarial loss prompts the generator to produce realistic transfer images, cycle consistency maintains the consistency and stability of semantic content during image translation, and identity loss enables the generator to better capture the inherent mapping relationship between the source and target domains, thus improving the quality of the generated images.

[0109] (2) Marginal Consistency Loss Function

[0110] The cycle consistency loss function in CycleGAN's basic loss function primarily aims to ensure that the generated image obtained from the source image through generators A2B and B2A achieves the same consistency as the source image. Figure 1 This invention aims to address the issue of content consistency in generated images. However, in the scenario of cross-modal fundus image generation, the cyclic consistency loss function exhibits a significant flaw: an image is generated incorrectly but then correctly reconstructed. This allows the cyclic consistency loss to be optimized to a lower value, but it fails to help update the generator parameters. Therefore, the cyclic consistency loss cannot meet the requirement of maintaining content consistency before and after cross-modal fundus image generation in this invention.

[0111] Based on EIEM, this invention designs an edge consistency loss. The similarity between edge detection feature components corresponding to the real image and the generated image is calculated to guide the model in maintaining content consistency across modal generated images. The calculation formula is as follows:

[0112] ;

[0113] in, For edge consistency loss, , These represent the source domain image and the target domain image, respectively. express After generator The generated image, express After generator The generated image, EIEM(·) represents the edge information obtained after being extracted by the edge information extraction module. Describing the L1 norm, and Both represent mathematical expectation operations.

[0114] Edge consistency loss focuses on edge information of image structure, while the basic loss function of CycleGAN is used for image features such as texture and color. Therefore, this invention adopts... and a joint loss function :

[0115] ;

[0116] wherein, is a weight coefficient of the edge consistency loss, and in the embodiment, the value is 1.

[0117] In step S5, the ReLBGAN is trained by using the multi-modal fundus dataset, and the parameters of the generator and the discriminator are optimized by minimizing the joint loss function.

[0118] In step S6, the trained ReLBGAN is used to input a fundus image of any modality, and a cross-modal fundus image of another modality is output by the corresponding generator.

[0119] Embodiment two: as shown in the embodiment, the application provides a cross-modal fundus image generation system based on a generative adversarial network, which is used to implement the cross-modal fundus image generation method based on the generative adversarial network in the above embodiment one, and comprises: Figure 6 a network construction module 100, configured to construct a fundus lesion perception and edge consistency generative adversarial network, the fundus lesion perception and edge consistency generative adversarial network taking a cycle consistency generative adversarial network as a baseline network, comprising two generators and two discriminators, and being configured to realize bidirectional cross-modal mapping of two different modal fundus images;

[0120] an attention embedding module 200, configured to embed a lesion perception hybrid attention module at a bottleneck layer of the generator, the lesion perception hybrid attention module being configured to strengthen the extraction ability of the generator to subtle texture and morphological features of a lesion region in a fundus image;

[0121] an edge extraction module design module 300, configured to design an edge information extraction module, the edge information extraction module being configured to extract key edge features of a source domain image / target domain image and a corresponding generated image, respectively;

[0122] a loss function definition module 400, configured to define a joint loss function, the joint loss function comprising a basic loss of the cycle consistency generative adversarial network and an edge consistency loss, wherein the basic loss of the cycle consistency generative adversarial network is composed of an adversarial loss, a cycle consistency loss and an identity loss, and the edge consistency loss is obtained by calculating the similarity of the edge features extracted by the edge information extraction module from a source domain / target domain image and a corresponding generated image;

[0123]

[0124] ​The network training module 500 is configured to train the fundus lesion perception and edge consistency generative adversarial network by using the multi-modal fundus data set, and optimize the parameters of the generator and the discriminator by minimizing the joint loss function.

[0125] The image generation module 600 is configured to input the fundus image of any modality into the trained fundus lesion perception and edge consistency generative adversarial network, and output the cross-modal fundus image of another modality through the corresponding generator.

[0126] The cross-modal fundus image generation system based on the generative adversarial network of the embodiment is used to implement the cross-modal fundus image generation method based on the generative adversarial network, and therefore the specific embodiments of the cross-modal fundus image generation system based on the generative adversarial network can be seen from the foregoing embodiment part of the cross-modal fundus image generation method based on the generative adversarial network, for example, the network construction module 100, the attention embedding module 200, the edge extraction module design module 300, the loss function definition module 400, the network training module 500 and the image generation module 600, which are respectively used to implement steps S1, S2, S3, S4, S5 and S6 in the foregoing cross-modal fundus image generation method based on the generative adversarial network, and therefore the specific embodiments can refer to the descriptions of the respective parts of the embodiment. In order to avoid redundancy, the details are not described herein.

[0127] In order to further illustrate the advantages of the method of the present application, the following will be described in combination with specific simulation verification. The experiment includes data set introduction, evaluation index, and experimental results. The following will be described respectively:

[0128] 1. Data set

[0129] In order to comprehensively evaluate the generalization, robustness of the generation model and the quality of the generated image, the present application uses a total of four fundus image data sets. In the training and testing of the ReLBGAN model, two multi-modal fundus image data sets are used, which are data set 1 containing OCTA and fundus color images and data set 2 containing Enface OCT and fundus color images. In the downstream experiment, data set 3 containing diabetic retinopathy Enface OCT image and public data set APTOS2019 of fundus color image are used.

[0130] Data set 1: composed of 104 pairs of fundus images of two modalities of OCTA and fundus color images, containing 102 pairs of normal images and 104 pairs of diabetic retinopathy images, and the image resolution is 512x512. In the generation experiment, the data set is divided as follows: 84 pairs of normal images and 84 pairs of diabetic retinopathy images are included in the training set, and 18 pairs of normal images and 20 pairs of diabetic retinopathy images are included in the test set.

[0131] Dataset 2: 104 pairs of fundus images from Enface OCT and fundus color photography, including 102 pairs of normal images and 104 pairs of diabetic retinopathy images, with an image resolution of 512x512. In the generation experiment, the data set is divided as follows: 84 pairs of normal images and 84 pairs of diabetic retinopathy images are included in the training set, and 18 pairs of normal images and 20 pairs of diabetic retinopathy images are included in the test set.

[0132] Dataset 3: Contains 100 Enface OCT images of diabetic retinopathy DR, mainly used to test the generalization of the ReLBGAN model and provide data augmentation for downstream experiments.

[0133] Dataset APTOS 2019: A diabetic retinopathy grading dataset released by Kaggle, containing a total of 3662 fundus color photography images, divided into 0-4 five levels. In the diabetic retinopathy grading experiment of the present application, the dataset is divided into a training set of 2922 images, a validation set of 359 images, and a test set of 361 images.

[0134] 2. Evaluation index

[0135] The present application uses Learned Perceptual Image Patch Similarity (LPIPS similarity) and Fréchet Inception Distance (FID distance) as objective indexes to evaluate the quality of the generated images. The smaller the FID distance value, the closer the feature distribution of the generated image to the real image, and the better the image quality and diversity; the lower the LPIPS similarity score, the higher the image perceptual similarity.

[0136] In the downstream diabetic retinopathy grading experiment, accuracy, F1 score, Kappa coefficient and recall rate are used as objective evaluation indexes.

[0137] 3. Results

[0138] (1) Comparative experiment

[0139] In order to verify the effectiveness of the ReLBGAN model proposed in the present application, it is compared with the widely used unsupervised generation model, and the comparison model mainly includes CUT, styleGAN-V2, CycleGAN-turbo which is a fusion diffusion model and CycleGAN, and the baseline network CycleGAN. A total of 2 sets of comparative experiments are set, which are: (a) diabetic retinopathy Enface OCT and fundus color photography cross-modal generation; (b) diabetic retinopathy OCTA and fundus color photography cross-modal generation.

[0140] (a) Diabetic retinopathy Enface OCT and fundus color photography cross-modal generation: Table 1 andFigure 7 The objective indicators and visual results of the cross-modal generation comparison experiment of the sugar network Enface OCT and fundus color photographs are respectively shown. The experimental results show that in the experiment of generating fundus color photographs by the sugar network Enface OCT, the color generated by the fundus color photograph and the consistency of the optic disc, blood vessels and lesions generated by the ReLBGAN method of the application are better than those of other comparison methods; the generated Enface OCT structure is reasonable, the lesion is clear, the content consistency is better than other models, and the LPIPS similarity and FID distance indicators are better than other comparison methods.

[0141] Table 1 shows the cross-modal generation comparison experiment of the sugar network Enface OCT and fundus color photographs

[0142]

[0143] (b) Cross-modal generation of sugar network OCTA and fundus color photographs: Table 2 and Figure 8 The objective indicators and visual results of the cross-modal generation comparison experiment of the sugar network OCTA and fundus color photographs are respectively shown. The experimental results show that in the experiment of generating fundus color photographs by the sugar network OCTA, the overall structure of the generated color photograph is more reasonable than other models; the generated OCTA is not conspicuous compared with the content generated by other methods, which to a certain extent alleviates the problem of poor content consistency caused by unsupervised training strategy, generates high-quality sugar network OCTA images, and the images do not have the problem of mode collapse, and has stronger robustness compared with other comparison models.

[0144] Table 2 shows the cross-modal generation comparison experiment of the sugar network OCTA and fundus color photographs

[0145]

[0146] (2) Ablation experiment

[0147] The CycleGAN is used as a baseline network in the application, in order to verify the effectiveness of the edge consistency loss function and the lesion perception hybrid attention module (LAHAM) on the performance improvement of the model, the cross-modal generation ablation experiment of the sugar network Enface OCT and fundus color photographs is set on the data set 2.

[0148] Table 3 shows the cross-modal generation ablation experiment results of the sugar network Enface OCT and fundus color photographs. The results show that when the baseline network is added with the edge consistency loss function And LAHAM, it can be found that the generated fundus color photographs and Enface OCT are more similar to the real images, especially the lesion structure and color in the generated fundus color photographs are basically consistent with the real images. In the generated Enface OCT images, although the lesion structure and texture details are basically consistent, there are still cases of incorrect generation of lesion gray scale, and the objective indicators are all better.

[0149] Table 3 Ablation experiment of Enface OCT and fundus color photograph cross-modal generation of sugar network

[0150]

[0151] (3) Downstream sugar network grading experiment

[0152] The present experiment aims to generate Enface OCT images from fundus color photographs by using the ReLBGAN model, as new modal data to expand the APTOS2019 dataset, so as to improve the sugar network grading performance. The RETFound model is used for grading. First, the ReLBGAN model trained on the dataset 2 is used to generate Enface OCT images from the fundus color photographs in the APTOS 2019 training set (2922). The generated Enface OCT is expanded according to the number of 20%, 40%, 60%, 80%, and 100% of the number of fundus color photographs in the original training set. The sugar network grading performance of the RETFound model under different expansion ratios is compared to verify the effectiveness of the cross-modal data expansion method. Table 4 shows the results of the sugar network grading experiment of the Enface OCT generated image cross-modal data expansion under different ratios. With the increase of the Enface OCT expansion ratio, the ACC and F1 indicators of the sugar network grading are continuously improved, and when the expansion ratio reaches 100%, all indicators reach the optimal, indicating the effectiveness of the cross-modal data expansion method, and also indicating that the ReLBGAN model of the present application can generate Enface OCT images of high quality.

[0153] Table 4 Results of the sugar network grading experiment of the fundus color photograph generated Enface OCT expansion

[0154]

[0155] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied in the medium.

[0156] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps in one or more flowcharts and / or blocks

[0157] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps in one or more flowcharts and / or blocks Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1 one or more flowcharts and / or blocks

[0158] Obviously, the above-described embodiments are only examples for clarity of description and are not limiting on the implementation. Based on the above description, other different forms of changes or variations can be made by those of ordinary skill in the art. Here, it is not necessary and impossible to enumerate all the embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A method for generating cross-modal fundus images based on generative adversarial networks, characterized in that, include: A generative adversarial network for fundus lesion perception and edge consistency is constructed. The generative adversarial network for fundus lesion perception and edge consistency is based on a cyclic consistency generative adversarial network and includes two generators and two discriminators to achieve bidirectional cross-modal mapping of fundus images of two different modalities. A lesion-aware hybrid attention module is embedded in the bottleneck layer of the generator. This module enhances the generator's ability to extract subtle textures and morphological features of lesion regions in fundus images. The lesion-aware hybrid attention module includes channel attention, spatial attention, and local importance attention. Its specific operation is as follows: For the input feature map Perform channel attention operations to obtain the output feature map of channel attention. : The output feature map of the channel attention Input spatial attention branch to obtain spatial attention output feature map ; The output feature map of the channel attention Input the local importance attention branch to obtain the output feature map of the local importance attention. ; The output feature map of the spatial attention Output feature map of the local importance attention Feature fusion is performed to obtain the final output feature map of the lesion perception hybrid attention module. ; Design an edge information extraction module, which is used to extract key edge features of the source domain image / target domain image and the corresponding generated image respectively; Define a joint loss function, which includes the base loss and edge consistency loss of the recurrent consistency generative adversarial network. The base loss of the recurrent consistency generative adversarial network consists of adversarial loss, recurrent consistency loss and identity loss. The edge consistency loss is obtained by calculating the similarity between the edge features extracted by the edge information extraction module of the source / target domain image and the corresponding generated image. The generative adversarial network for fundus lesion perception and edge consistency was trained using a multimodal fundus dataset, and the parameters of the generator and discriminator were optimized by minimizing the joint loss function. Using a trained fundus lesion perception and edge consistency generative adversarial network, a fundus image of any modality is input and a cross-modal fundus image of another modality is output through the corresponding generator.

2. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The generator includes an encoder, a bottleneck layer, and a decoder; The encoder includes three sequentially connected downsampling layers. Each downsampling layer consists of a CIL layer. Each CIL layer consists of a 3×3 convolution, instance normalization, and Leaky ReLU activation function. The input image is filled with boundary reflections and then input into the encoder. The encoder is used to extract image features layer by layer and reduce the size of the feature map. The bottleneck layer includes nine stacked residual blocks, each residual block consisting of two cascaded reflection fillers, two CIR layers and residual connections, and the lesion perception hybrid attention module is embedded between the fourth and fifth residual blocks. The decoder includes two sequentially connected upsampling layers, a reflection fill operation, and an output layer. The upsampling layer consists of a 3×3 transposed convolution, instance normalization, and a ReLU activation function. The output layer consists of a 3×3 convolution and a Tanh activation function, used to restore high-dimensional features to the original input image size.

3. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The input feature map Perform channel attention operations to obtain the output feature map of channel attention. Specifically, it includes: For the input feature map Perform global average pooling and global max pooling in the spatial dimension to obtain two different channel description vectors; Two different channel description vectors are input into a weight-shared multilayer perceptron for nonlinear transformation, and the transformed results are added together. The summation result is used to generate a channel attention weight vector through a Sigmoid activation function; The channel attention weight vector is compared with the input feature map. Channel-by-channel multiplication yields the output feature map of channel attention. .

4. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The output feature map of the channel attention Input spatial attention branch to obtain spatial attention output feature map Specifically, it includes: Output feature map of channel attention Global average pooling and global max pooling are performed on the channel dimension respectively to obtain two single-channel feature maps; Two single-channel feature maps are concatenated along the channel dimension to obtain a two-channel feature map; Spatial information is fused by performing a 7×7 convolution on the dual-channel feature map, and then a spatial attention weight mask is generated by using the Sigmoid activation function. The spatial attention weight mask and the output feature map of channel attention are compared. By multiplying point by point, we obtain the output feature map of spatial attention. .

5. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The output feature map of the channel attention Input the local importance attention branch to obtain the output feature map of the local importance attention. Specifically, it includes: Output feature map of channel attention Perform 1×1 convolutions to reduce the number of channels; The SoftPooling operation is performed on the dimensionality-reduced feature map to downsample it and preserve local feature information; The downsampling results are subjected to two 3×3 convolutions in sequence. The first convolution is used to extract high-dimensional features, and the second convolution is used to adjust the number of channels. A weight map is generated from the feature map after adjusting the number of channels using the Sigmoid activation function; The weight map is upsampled using bilinear interpolation to make its resolution similar to the output feature map of the channel attention. Consistent; The upsampled weight map is compared with the output feature map of channel attention. Element-wise multiplication yields the output feature map of the local importance attention. .

6. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The edge information extraction module includes two identical branches, one above the other, and the working process of each branch is as follows: Roberts edge detection is performed on the input image to calculate the gray-level difference between adjacent pixels in the image to initially extract edge information; Perform wavelet transform on the edge detection results to decompose and extract the edge feature components in the horizontal and vertical directions, and filter out low-frequency noise; The nonlocal mean denoising enhancement operation is performed on the feature components of the wavelet transform output to suppress noise while preserving the edge structure, thus obtaining the final edge features.

7. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The formula for calculating the edge consistency loss is as follows: ； in, For edge consistency loss, , These represent the source domain image and the target domain image, respectively. express After generator The generated image, express After generator The generated image, EIEM(·) represents the edge information obtained after being extracted by the edge information extraction module. Describing the L1 norm, and Both represent mathematical expectation operations.

8. The cross-modal fundus image generation method based on generative adversarial networks according to claim 1, characterized in that, The formula for calculating the joint loss function is as follows: ； in, For the joint loss function, Generate the basic loss for the adversarial network to achieve cycle consistency. For edge consistency loss, represents the weighting coefficient for edge consistency loss.

9. A cross-modal fundus image generation system based on generative adversarial networks, characterized in that, The system is used to implement the cross-modal fundus image generation method based on generative adversarial networks as described in any one of claims 1 to 8, comprising: The network construction module is used to construct a generative adversarial network for fundus lesion perception and edge consistency. The generative adversarial network for fundus lesion perception and edge consistency uses a cyclic consistency generative adversarial network as the baseline network and includes two generators and two discriminators to achieve bidirectional cross-modal mapping of fundus images of two different modalities. An attention embedding module is used to embed a lesion-aware hybrid attention module into the bottleneck layer of the generator. The lesion-aware hybrid attention module is used to enhance the generator's ability to extract subtle textures and morphological features of lesion areas in fundus images. An edge extraction module design module is used to design an edge information extraction module, which is used to extract key edge features of the source domain image / target domain image and the corresponding generated image, respectively. The loss function definition module is used to define the joint loss function, which includes the base loss and edge consistency loss of the recurrent consistency generative adversarial network. The base loss of the recurrent consistency generative adversarial network consists of adversarial loss, recurrent consistency loss and identity loss. The edge consistency loss is obtained by calculating the similarity between the edge features extracted by the edge information extraction module of the source domain / target domain image and the corresponding generated image. The network training module is used to train the fundus lesion perception and edge consistency generative adversarial network using a multimodal fundus dataset, and optimizes the parameters of the generator and discriminator by minimizing the joint loss function; The image generation module is used to utilize a trained fundus lesion perception and edge consistency generative adversarial network to take a fundus image of any modality as input and output a cross-modal fundus image of another modality through the corresponding generator.

Images