Cross-modal universal eye disease screening and diagnosis method and device, electronic equipment and storage medium

Abstract

The invention discloses a cross-modal universal eye disease screening and diagnosis method and device, electronic equipment and a storage medium, and is used for solving the problems that existing ophthalmology artificial intelligence models are mostly trained based on single-modal data, and lack of multi-modal integration ability, so that the generalization ability of the models in different scenes is poor, and the universality of the models is poor. The technical problem that complex and changeable eye disease screening requirements in the real world cannot be met is solved. The method comprises the following steps: acquiring a multi-modal real-world ophthalmic image; preprocessing the real-world ophthalmic image to obtain a preprocessed image; inputting the preprocessed image into a pre-constructed medical image generation model to generate a medical image; using a pre-trained basic model to extract general features of the medical image; training a cross-modal universal eye disease screening model by adopting the universal features; and inputting the to-be-screened image into the cross-modal universal eye disease screening model, and outputting a screening prediction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of eye disease screening, and in particular to a cross-modal universal eye disease screening method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Eye disease is one of the main causes of visual impairment and blindness worldwide, and early screening and diagnosis is crucial for preventing and treating visual impairment. With changes in lifestyle, the incidence of eye disease is on the rise, placing a heavy burden on society. In the grassroots, remote areas and high-risk groups, the demand for eye disease screening is particularly urgent. These areas often lack professional ophthalmic equipment and specialist doctors, making it difficult to achieve large-scale community screening, resulting in many eye disease patients missing the opportunity for diagnosis and treatment, causing irreversible visual impairment.

[0003] In recent years, the application of artificial intelligence technology in the field of ophthalmology has gradually emerged, providing new methods and technical support for eye disease screening and diagnosis. However, due to the difficulty of obtaining real-world screening data, high-quality data and annotations are scarce, and existing ophthalmic artificial intelligence models are mostly derived from laboratory data, lacking data support from the real world, resulting in poor application effect of the model in the real world, making it difficult to land. In addition, the existing ophthalmic artificial intelligence model is mostly trained based on single modal data, lacking the ability to integrate multiple modalities, resulting in poor generalization ability of the model in different scenarios, and unable to cope with the complex and variable eye disease screening needs in the real world. SUMMARY

[0004] The present application provides a cross-modal universal eye disease screening method, device, electronic equipment and storage medium, which solves the technical problem that the existing ophthalmic artificial intelligence model is mostly trained based on single modal data, lacking the ability to integrate multiple modalities, resulting in poor generalization ability of the model in different scenarios, and unable to cope with the complex and variable eye disease screening needs in the real world.

[0005] The present application provides a cross-modal universal eye disease screening method, comprising:

[0006] Obtaining multi-modal real-world ophthalmic images;

[0007] Preprocessing the real-world ophthalmic images to obtain preprocessed images;

[0008] Inputting the preprocessed images into a pre-constructed medical image generation model to generate medical images;

[0009] Extracting universal features of the medical images using a pre-trained base model, the base model being based on a visual transformer framework and using a mask image modeling method, and being pre-trained using real-world unlabeled ophthalmic images;

[0010] train a cross-modal universal eye disease screening model using the general features;

[0011] input the to-be-screened image into the cross-modal universal eye disease screening model, and output a screening prediction result.

[0012] Optionally, the step of pre-processing the real-world ophthalmic image to obtain a pre-processed image comprises:

[0013] adjusting the real-world ophthalmic image to obtain an ophthalmic image of a fixed resolution;

[0014] normalizing the ophthalmic image of the fixed resolution to obtain a normalized image;

[0015] performing data enhancement on the normalized image to obtain an enhanced image, wherein the data enhancement method comprises at least one of random rotation, flipping, cropping, translation, scaling, brightness adjustment, contrast adjustment, saturation adjustment, hue adjustment;

[0016] removing noise in the enhanced image to obtain a denoised image;

[0017] performing data cleaning on the denoised image to obtain a pre-processed image.

[0018] Optionally, the step of inputting the pre-processed image into a pre-constructed medical image generation model to generate a medical image comprises:

[0019] inputting the pre-processed image into a pre-trained real-world quality control model to obtain an image classification;

[0020] inputting the pre-processed image with the image classification satisfying a preset classification into a pre-constructed medical image generation model to generate a medical image.

[0021] Optionally, the training process of the medical image generation model comprises:

[0022] collecting real-world ophthalmic training images;

[0023] adjusting the real-world ophthalmic training images to fixed-resolution images;

[0024] training a diffusion model using the fixed-resolution images and preset prior information to obtain a medical image generation model.

[0025] The application further provides a cross-modal universal eye disease screening device, comprising:

[0026] a real-world ophthalmic image acquisition module configured to acquire multi-modal real-world ophthalmic images;

[0027] a preprocessing module configured to preprocess the real-world ophthalmic image to obtain a preprocessed image;

[0028] a medical image generation module configured to input the preprocessed image into a pre-constructed medical image generation model to generate a medical image;

[0029] a general feature extraction module configured to extract general features of the medical image by using a pre-trained base model, the base model being pre-trained based on a visual transformer framework and a mask image modeling method and using real-world unlabeled ophthalmic images;

[0030] a cross-modal general eye disease screening model training module configured to train a cross-modal general eye disease screening model by using the general features;

[0031] a screening diagnosis prediction result output module configured to input a to-be-screened image into the cross-modal general eye disease screening model to output a screening diagnosis prediction result.

[0032] Optionally, the preprocessing module comprises:

[0033] an image adjustment sub-module configured to adjust the real-world ophthalmic image according to a preset resolution to obtain a resolution-adjusted ophthalmic image;

[0034] a normalization sub-module configured to normalize the resolution-adjusted ophthalmic image to obtain a normalized image;

[0035] a data enhancement sub-module configured to perform data enhancement on the normalized image to obtain an enhanced image, wherein the data enhancement method comprises at least one of random rotation, flipping, cropping, translation, scaling, brightness adjustment, contrast adjustment, saturation adjustment, hue adjustment;

[0036] a denoising sub-module configured to remove noise in the enhanced image to obtain a denoised image;

[0037] a data cleaning sub-module configured to perform data cleaning on the denoised image to obtain the preprocessed image.

[0038] Optionally, the medical image generation module comprises:

[0039] an image classification sub-module configured to input the preprocessed image into a pre-trained real-world quality control model to obtain image classification;

[0040] a medical image generation sub-module configured to input the preprocessed image with image classification satisfying a preset classification into a pre-constructed medical image generation model to generate a medical image.

[0041] Optionally, the training process of the medical image generation model comprises:

[0042] a real-world ophthalmic training image acquisition submodule configured to acquire a real-world ophthalmic training image;

[0043] a resolution adjustment submodule configured to adjust the real-world ophthalmic training image into a fixed resolution image;

[0044] a medical image generation model training submodule configured to train a diffusion model using the fixed resolution image and preset prior information to obtain a medical image generation model.

[0045] The application further provides an electronic device, which comprises a processor and a memory:

[0046] The memory is configured to store program code and transmit the program code to the processor.

[0047] The processor is configured to execute the cross-modal universal eye disease screening method according to the instructions in the program code.

[0048] The application further provides a computer readable storage medium configured to store program code, wherein the program code is configured to execute the cross-modal universal eye disease screening method.

[0049] As can be seen from the above technical solutions, the application has the following advantages: the application provides a cross-modal universal eye disease screening method, and specifically discloses the following: a multi-modal real-world ophthalmic image is acquired; a real-world ophthalmic image is preprocessed to obtain a preprocessed image; the preprocessed image is input into a pre-constructed medical image generation model to generate a medical image; a pre-trained basic model is used to extract universal features of the medical image, the basic model is based on a visual transformer framework and uses a mask image modeling method and is pre-trained by using real-world unlabeled ophthalmic images; a cross-modal universal eye disease screening model is trained using the universal features; an image to be screened is input into the cross-modal universal eye disease screening model, and a screening prediction result is output.

[0050] The application uses unlabeled multi-modal real-world ophthalmic image data to pre-train a model, learns universal representations from unlabeled ophthalmic images, fully excavates the potential value of a large amount of unlabeled data, solves the problem of annotation scarcity, flexibly extends any modal, and thus generates a basic model that can adapt to various downstream tasks, provides a system open source for other clinical tasks, provides a solution with higher precision, lower cost and stronger interpretability for real-world medical scenarios, and trains a cross-modal universal eye disease screening model using universal features, realizes cross-modal generation and completion, deeply integrates clinical knowledge and causal reasoning, and provides a solution with precision, inclusiveness and credibility for real-world medical scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 A step flow chart of a cross-modal universal eye disease screening method provided by the embodiment of the present application.

[0053] Figure 2 A data processing flowchart for a medical image generation model;

[0054] Figure 3 A data processing flowchart for a basic model;

[0055] Figure 4 A cross-modal universal eye disease screening model training flowchart;

[0056] Figure 5 A structure block diagram of a cross-modal universal eye disease screening device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0057] The embodiment of the present application provides a cross-modal universal eye disease screening method, device, electronic equipment and storage medium, which is used to solve the technical problem that most of the existing ophthalmic artificial intelligence models are trained based on single modal data, lack of integration ability of multi-modal, and the generalization ability of the model in different scenes is poor, which cannot meet the complex and changeable eye disease screening demand in the real world.

[0058] In order to make the purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the following described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] Please refer to Figure 1 , Figure 1 A step flow chart of a cross-modal universal eye disease screening method provided by the embodiment of the present application.

[0060] The cross-modal universal eye disease screening method provided by the present application can specifically include the following steps:

[0061] Step 101, acquiring a multi-modal real-world ophthalmic image;

[0062] In the embodiment of the application, the multi-modal real-world ophthalmic image can include, but is not limited to, an anterior segment picture, a slit lamp picture, an OCT (optical coherence tomography) fundus photo, a fundus image, and the like eye picture.

[0063] Step 102, pre-processing the real-world ophthalmic image to obtain a pre-processed image;

[0064] In the embodiment of the application, when constructing a base model based on a real-world ophthalmic image, data preprocessing is a key link to ensure the performance of the model.

[0065] In one example, the step of pre-processing the real-world ophthalmic image to obtain a pre-processed image can include the following sub-steps:

[0066] S21, adjusting the real-world ophthalmic image according to a preset resolution to obtain a resolution-adjusted ophthalmic image;

[0067] S22, normalizing the resolution-adjusted ophthalmic image to obtain a normalized image;

[0068] S23, data augmenting the normalized image to obtain an augmented image, wherein the data augmentation method includes at least one of random rotation, flipping, cropping, translation, scaling, brightness adjustment, contrast adjustment, saturation adjustment, and hue adjustment;

[0069] S24, removing noise in the augmented image to obtain a de-noised image;

[0070] S25, data cleaning the de-noised image to obtain a pre-processed image.

[0071] In a specific implementation, the picture can be scaled to a resolution of 224*224 or 299*299 to adapt to the model requirements. Then, the pixel value of the resolution-adjusted ophthalmic image after uniform sizing is divided by 255, and scaled to [0, 1] to obtain a normalized image, so as to accelerate the model convergence. Then, the normalized image is data augmented to expand the sample size and improve the model generalization ability. The data augmentation method can be at least one of the following methods: random rotation, flipping (horizontal / vertical), cropping, translation, scaling, and brightness adjustment, contrast adjustment, saturation adjustment, and hue adjustment.

[0072] After completing data enhancement, for the problems of noise, artifacts and low brightness in the enhanced image, quality improvement can be performed using denoising, contrast enhancement and the like. Among them, the denoising and contrast enhancement can adopt the manner commonly used in the art, and the embodiments of the present application do not make specific limitations thereto.

[0073] The denoised image in which denoising is completed can have damaged or unreadable pictures. To avoid the influence of these pictures on model training, the damaged or unreadable pictures can be removed, and the annotation is checked and corrected.

[0074] Step 103, inputting the preprocessed image into a pre-constructed medical image generation model to generate a medical image;

[0075] After completing the preprocessing of the image, the preprocessed image can be input into a pre-constructed medical image generation model to generate a medical image.

[0076] In one example, step 103 can include the following sub-steps:

[0077] S31, inputting the preprocessed image into a pre-trained real world quality control model to obtain image classification;

[0078] Precise quality screening in real world ophthalmic images not only has important clinical value, but also is crucial for the construction of subsequent basic models and the development of general ophthalmic screening models.

[0079] Since real world data usually has characteristics such as large scale, high complexity and imperfection, the image quality can be divided into three categories: poor, general and good. The image of "poor quality" usually cannot clearly observe the key eye structure or lesion due to problems such as shooting angle, lighting condition, focus offset, etc., which lacks clinical value and cannot provide effective feature learning for the model, and should be directly excluded. The image of "good quality" can clearly present the key structure, lesion and its position required for diagnosis, and has high analysis value. The image of "general quality" is between the two, although there is certain noise or defect, it still has clinical value and occupies a large proportion in real data. Therefore, including such images in training not only can improve the robustness of the model, but also can help it learn the complex distribution characteristics of real world data.

[0080] In the data labeling phase, all images were labeled by two professional ophthalmologists in a double-blind manner based on image clarity and information integrity. For samples with discrepancies, a senior ophthalmologist made the final decision to ensure the accuracy and consistency of quality grading. Subsequently, a quality control model was built based on the InceptionV3 convolutional neural network (CNN), and the quality categories of the images were automatically classified. After the model training was completed, all classification results were manually reviewed to further verify the accuracy of the model and ensure its reliability and interpretability in clinical applications. Through this quality control process, we can effectively screen high-quality data while preserving the diversity of real-world data, thereby providing a solid foundation for subsequent model construction.

[0081] The training process of the quality control model is as follows:

[0082] (1) Data preparation, including quality label setting, data set division, data preprocessing, etc.

[0083] (2) Details of model construction, such as whether to use pre-trained weights, activation function, loss function, optimizer, etc.

[0084] (3) Details of model training, including learning rate, number of epochs, etc.

[0085] (4) Model evaluation: evaluation metrics on the test set, such as AUC, F1-score, etc.

[0086] An image quality control model was built based on the InceptionV3 convolutional neural network (CNN). The data set was divided into training set, validation set and test set according to the ratio of 6:2:2. The model was initialized with the pre-trained weights of ImageNet, the output layer used Softmax activation function to realize multi-class classification, the loss function was multi-class cross-entropy (categorical cross-entropy), the optimizer was Adam, and the initial learning rate was set to 1e-4.

[0087] The model was trained for a total of 100 epochs. When the loss of the validation set did not decrease for 10 consecutive epochs, the learning rate would automatically decay to half of the current value to improve the convergence effect of the model. The evaluation metrics included accuracy and F1-score. The former was used to measure overall performance, while the latter focused more on the classification ability of minority class samples, and could comprehensively reflect the performance of the model on various samples.

[0088] S32, input the preprocessed image that meets the preset classification of image classification into the pre-constructed medical image generation model to generate a medical image.

[0089] The preprocessed images meeting the "good quality" and "general quality" can be extracted from the preprocessed images through the quality control model, and input into the pre-constructed medical image generation model, so that the corresponding expanded images can be generated, thereby avoiding the influence of unbalanced data on the subsequent cross-modal universal eye disease screening model training. For example, if the positive samples are few and the negative samples are many, the number of positive samples can be expanded through the medical image generation model.

[0090] In one example, the training process of the medical image generation model is as follows: collecting real-world ophthalmic training images; adjusting the real-world ophthalmic training images to fixed resolution images; training a diffusion model using the fixed resolution images and preset prior information to obtain the medical image generation model.

[0091] In a specific implementation, the medical image generation model can be constructed based on a conditional diffusion model (Conditional Diffusion Model) to synthesize high-quality ophthalmic image data, make up for the deficiency of real-world data, and enhance the general applicability of the subsequently constructed cross-modal universal eye disease screening model.

[0092] The diffusion model is a type of generative model, and its core idea is to gradually add noise to the image during the training phase to make it close to a random distribution, and then gradually restore the clear image through the reverse denoising process, thereby realizing the generation of high-quality images.

[0093] In the embodiments of the present application, high-quality real-world ophthalmic training images are first collected, standardized to a fixed resolution of 224*224*3, and used as training data for the diffusion model. During the training process, the model learns to recover images from noise step by step, so that it can generate high-fidelity images consistent with the distribution of real images. In addition, a conditional control mechanism is introduced to enable the model to generate synthetic images with specific pathological characteristics under the condition of given prior information (such as disease category or lesion location), thereby enhancing its application value in medical image research.

[0094] In a specific implementation, a conditional diffusion model (Conditional Diffusion Model) is used to generate medical images, and the core is to combine prior condition information to guide the generation path of the image during the diffusion inversion process, thereby obtaining synthetic images with specific pathological characteristics. During the reverse generation process, the model learns the path of recovering images from noise step by step through a neural network. The key is that in the denoising prediction at each time step, the prior condition information is fused, that is, the model not only receives the noise image and time step information, but also receives the condition vector to participate in the prediction together. In this way, the model learns to restore the image under the premise of meeting the conditions, that is, to generate only images with specific pathological characteristics (such as hemorrhage, exudation, lesion location, etc.).

[0095] After the training of the medical image generation model is completed, the generated medical image can be quality evaluated by using structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and the like.

[0096] SSIM (Structural Similarity Index) is a perceptual index for measuring the similarity of two images in terms of brightness, contrast and structural information, and is suitable for evaluating the visual quality of images, especially in image compression, generation and the like. The SSIM of two images x (original image) and y (generated image) is defined as follows:

[0097]

[0098] wherein, , is the mean value of images x and y (representing brightness);

[0099] , is the variance of images x and y (representing contrast);

[0100] is the covariance of images x and y (representing structural similarity);

[0101] , is a small constant set to avoid zero denominator.

[0102] PSNR (Peak Signal-to-Noise Ratio) is used to measure the absolute error level of image reconstruction, focusing on pixel-level differences, and is a commonly used index for measuring image distortion.

[0103] The calculation process is as follows:

[0104] First, calculate the MSE (Mean Square Error) between images:

[0105]

[0106] Then calculate the PSNR:

[0107]

[0108] wherein: is the maximum possible value of image pixels (e.g. 255 for an 8-bit image);

[0109] The unit of PSNR is dB (decibel).

[0110] The evaluation results of the quality evaluation are shown in Table 1 as follows:

[0111]

[0112] Further, professional ophthalmologists can also be invited for further manual review to ensure the clinical usability of the generated medical images. Through the medical image generation model, not only can the data set be expanded and the robustness of subsequent cross-modal general eye disease screening model construction be improved, but also high-quality synthetic data can be provided for medical image analysis in data scarce scenarios.

[0113] The process of generating a medical image by the medical image generation model is as shown in Figure 2 .

[0114] First, the medical images including OCT images and fundus images and other modalities are input into the image encoder to extract the corresponding latent image feature representation. Then, Gaussian noise is added to the feature to simulate the diffusion process, thereby obtaining the degraded image feature. At the same time, the text description corresponding to the image is input into the text encoder for encoding, and the description includes image modality information and corresponding pathological or anatomical feature description. The text vector obtained after encoding is used as a conditional control signal.

[0115] In the denoising stage, the image feature with added noise and the text condition vector are fused and input into the denoising network through the cross-attention mechanism. The network learns to recover the medical image consistent with the original image structure from the condition-guided noise, thereby realizing accurate modeling of specific semantics.

[0116] In the inference stage, the user inputs the text description of the target modality and pathological features. The system first encodes the description to obtain the corresponding condition vector. The system then initializes a set of random noise image features conforming to the Gaussian distribution, and inputs them into the denoising network together with the text condition. Through multiple denoising steps, the latent image representation is iteratively generated, and finally restored by the image decoder to a high-fidelity medical image consistent with the expected semantics. This method realizes natural language-controlled medical image synthesis, has good controllability and clinical relevance, and is suitable for multiple scenarios such as medical image expansion, pathological image simulation, and medical AI training.

[0117] In the visual transformer framework, a mask image modeling method is used to generate

[0118] In the embodiments of the present application, based on the Vision Transformer (ViT) framework, a Masked Image Modeling (MIM) method is adopted to pre-train a large number of unlabelled ophthalmic images in the real world, so as to construct an efficient and general-purpose basic model. Specifically, all input unlabelled ophthalmic images are first adjusted to a standard resolution of 224*224*3, and then further divided into fixed-size patches of 16*16, a total of 196 patches. Subsequently, a random mask strategy is adopted to mask 40% of the fixed-size patches, and zero vectors are used to fill in to avoid the model directly obtaining the information of the masked area. For the image patches that are not masked, the encoder is first input for feature extraction, and the main body of the encoder adopts the ViT structure. The core mechanism is to use the self-attention mechanism of the Transformer to model the global and local features, so that the basic model can learn the long-distance dependence and enhance the feature expression ability.

[0119] In addition, a lightweight decoder can also be introduced after the ViT, which is responsible for reconstructing the pixel values or feature representations of the masked blocks from the output of the encoder, thereby forming a self-supervised learning task. In the training process, the iterative training is carried out by minimizing the global reconstruction loss, and the formula of the loss function is as follows:

[0120]

[0121] wherein the loss is calculated on the masked patches , and are the Gaussian distribution or the Dirichlet distribution of the regression or classification loss, respectively.

[0122] The global reconstruction loss enables the basic model to learn the global structure and local detail features of the image while restoring the content of the masked area, thereby improving its adaptability and generalization ability in the subsequent ophthalmic image analysis process. Compared with the traditional CNN structure, this method can fully exploit the potential value of a large amount of unlabelled data under unsupervised conditions, and provide more robust feature representation for computer-aided prediction of ophthalmic diseases.

[0123] The data processing flow of the basic model is shown in Figure 3 .

[0124] The self-supervised pre-training module adopts a multi-scale masked image modeling method (Local MIM) based on local occlusion reconstruction. This method first divides the input image into fixed-size image patches, and according to the set mask strategy, only part of the visible image patches are retained to form the visible feature input. The above visible image patch input is input into a multi-stage encoder structure, which can be a network based on a visual Transformer architecture (such as ViT or Swin Transformer), for extracting multi-scale visual representations at different levels.

[0125] In each encoding stage (Stage 1 to Stage 4), the extracted local features are input into the corresponding decoder to predict the reconstruction result of the occluded area in combination with the masked mask token. To improve the richness of the training signal and the diversity of the semantic level, the decoder output is compared with different scale supervision signals, such as normalized image pixels, edge structure features (such as HOG features), etc. Each stage is provided with a reconstruction loss function to guide the model to perform step-by-step reconstruction of the original image at the local scale, thereby achieving fine-grained feature modeling.

[0126] The LocalMIM architecture significantly improves the model's transferability and generalization performance in downstream visual tasks by introducing local perception and reconstruction tasks at multiple semantic levels, and is particularly suitable for feature pre-training in high-precision image scenarios such as medical images and remote sensing images.

[0127] Step 105, a cross-modal general eye disease screening model is trained using general features;

[0128] In a specific implementation, the multi-modal general features and potential lesion information are fused by a deep learning network. In this process, the model not only learns the lesion representation under a single modality, but also captures the correlation between modalities, so as to infer the possible diagnosis results of the missing image in the case of insufficient or missing slit lamp or OCT data. To further improve the robustness and generalization ability of the model, a cross-modal supervision training strategy is adopted, and a large amount of real-world multi-modal ophthalmic data is trained and verified, so that the model has the ability to cope with complex clinical scenarios; in the inference stage, input any one or several known modalities, and the model can output the prediction results of other modalities or disease risks. The general screening model not only effectively saves the cost of clinical examination and improves the screening efficiency, but also provides high-quality remote auxiliary diagnosis support for remote areas or scenes with limited medical resources, and promotes the development of eye health management in a more comprehensive and intelligent direction.

[0129] The training process of the cross-modal general eye disease screening model is shown in FIG. 4.

[0130] During the model training process, the input multi-modal image data, including OCT images, fundus images and slit lamp images, are first input into the corresponding modal feature encoders. Each encoder independently extracts image features under its modal, outputting a high-dimensional feature vector, denoted as FOCT, FFUNDUS, FSLIT. Since the features of different modalities differ in dimension and semantic space, these feature vectors cannot be directly fused.

[0131] Next, the above modal features are input into the shared feature projection layer, which maps them to a unified latent semantic space through nonlinear transformation, obtaining shared feature representations ZOCT, ZFUNDUS, ZSLIT of uniform dimension. This step completes the alignment of features, making the features of different modalities comparable and complementary, facilitating subsequent fusion.

[0132] The aligned shared features of each modality are then fused (e.g., spliced, weighted average or attention fusion) to obtain the fused feature Zfused. This fused feature is input into the decoder, which undergoes a series of nonlinear mappings to finally output the target task result y^, such as disease risk score or disease classification probability.

[0133] During the entire training process, the model introduces a modal missing simulation mechanism: in each training batch, one or more modal inputs are randomly masked, and only the remaining modal is used for training, simulating the situation of partial modal missing in real clinical practice and enhancing the robustness of the model.

[0134] At the same time, the model also combines a cross-modal supervision mechanism: when a modality is missing, the model not only learns to predict the disease label, but also uses existing modalities to infer the latent feature representation of the missing modality through reconstruction loss or contrastive learning, thereby improving its ability to model the implicit semantics between different modalities. The entire training process is optimized through the weighted combination of cross-entropy loss, contrastive loss or reconstruction loss, ultimately improving the generalization performance and diagnostic accuracy of the model in complex clinical environments.

[0135] Step 106, input the image to be screened into the cross-modal general eye disease screening model, and output the screening prediction result.

[0136] After the training of the cross-modal general eye disease screening model is completed, the image to be screened can be input into the cross-modal general eye disease screening model to obtain the corresponding screening result.

[0137] The application utilizes unannotated multi-modal real-world ophthalmic image data for model pre-training, learns general-purpose representations from unannotated ophthalmic images, fully excavates the potential value of a large amount of unannotated data, solves the problem of annotation scarcity, flexibly extends any modality, thereby generating a basic model that can adapt to various downstream tasks, provides a system open source for other clinical tasks, provides a higher-precision, lower-cost and stronger-explainability solution for real-world medical scenarios, and simultaneously trains a cross-modal general-purpose eye disease screening model through general-purpose feature training, realizes cross-modal generation and completion, deeply integrates clinical knowledge and causal reasoning, and provides a solution with precision, inclusiveness and credibility for real-world medical scenarios.

[0138] Please refer to Figure 5 , Figure 5 The structure block diagram of the cross-modal general-purpose eye disease screening device provided by the embodiment of the application is shown in FIG. 1.

[0139] The embodiment of the application provides a cross-modal general-purpose eye disease screening device, which comprises:

[0140] A real-world ophthalmic image acquisition module 501 is configured to acquire multi-modal real-world ophthalmic images.

[0141] A preprocessing module 502 is configured to preprocess the real-world ophthalmic images to obtain preprocessed images.

[0142] A medical image generation module 503 is configured to input the preprocessed images into a pre-constructed medical image generation model to generate medical images.

[0143] A general-purpose feature extraction module 504 is configured to extract general-purpose features of the medical images by using a pre-trained basic model, wherein the basic model is based on a visual transformer framework, adopts a mask image modeling method, and is generated by pre-training based on real-world unannotated ophthalmic images.

[0144] A cross-modal general-purpose eye disease screening model training module 505 is configured to train a cross-modal general-purpose eye disease screening model by using the general-purpose features.

[0145] A screening prediction result output module 506 is configured to input a to-be-screened image into the cross-modal general-purpose eye disease screening model and output a screening prediction result.

[0146] In the embodiment of the application, the preprocessing module 502 comprises:

[0147] An image adjustment sub-module is configured to adjust the real-world ophthalmic images according to a preset resolution to obtain resolution-adjusted ophthalmic images.

[0148] A normalization sub-module is configured to normalize the resolution-adjusted ophthalmic images to obtain normalized images.

[0149] a data enhancement submodule configured to perform data enhancement on the normalized image to obtain an enhanced image, wherein the data enhancement method comprises at least one of random rotation, flipping, cropping, translation, scaling, brightness adjustment, contrast adjustment, saturation adjustment, hue adjustment, and the like;

[0150] a denoising submodule configured to remove noise in the enhanced image to obtain a denoised image;

[0151] a data cleaning submodule configured to perform data cleaning on the denoised image to obtain a preprocessed image.

[0152] In the embodiment of the present application, the medical image generation module 503 comprises:

[0153] an image classification submodule configured to input the preprocessed image into a pre-trained real-world quality control model to obtain image classification;

[0154] a medical image generation submodule configured to input the preprocessed image with the image classification satisfying a preset classification into a pre-constructed medical image generation model to generate a medical image.

[0155] In the embodiment of the present application, the training process of the medical image generation model comprises:

[0156] a real-world ophthalmic training image acquisition submodule configured to acquire a real-world ophthalmic training image;

[0157] a resolution adjustment submodule configured to adjust the real-world ophthalmic training image into a fixed resolution image;

[0158] a medical image generation module training submodule configured to train a diffusion model using the fixed resolution image and preset prior information to obtain the medical image generation model.

[0159] The embodiment of the present application also provides an electronic device, which comprises a processor and a memory:

[0160] The memory is configured to store program code and transmit the program code to the processor;

[0161] The processor is configured to execute the cross-modal universal eye disease screening method according to the instructions in the program code.

[0162] The present application also provides a computer readable storage medium, which is configured to store program code, and the program code is configured to execute the cross-modal universal eye disease screening method.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0164] The various embodiments described in this specification are presented as examples. Each example is provided by way of explanation of the other examples. Thus, each of the various embodiments can be implemented alone or in combination with any of the other embodiments. Each embodiment is presented in the context of the general inventive concept, which will be apparent to those skilled in the art, from the description and drawings.

[0165] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0166] Embodiments of the present application are described herein with reference to the flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0167] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices to cause a series of operational steps to be performed on the computer or other programmable terminal devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal devices provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0169] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments described and illustrated by the applicant, without departing from the spirit and scope of the application, as defined in the following claims. Accordingly, the appended claims are intended to cover all such variations and modifications as falling within the scope of the application.

[0170] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0171] Finally, it should be noted that in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or operations have any such actual relationship or order. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or terminal device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or terminal device including the element.

[0172] The above-described and above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cross-modal universal method for screening and diagnosing eye diseases, characterized in that, include: Acquire multimodal real-world ophthalmological images; The real-world ophthalmological images are preprocessed to obtain preprocessed images; The preprocessed image is input into a pre-built medical image generation model to generate a medical image; A pre-trained base model is used to extract general features from the medical images. The base model is based on a visual transformer framework and uses a masked image modeling method. It is generated by pre-training on unlabeled ophthalmic images from the real world. The general features are used to train a cross-modal general eye disease screening model; The image to be screened is input into the cross-modal universal eye disease screening model, and the screening prediction results are output.

2. The method according to claim 1, characterized in that, The step of preprocessing the real-world ophthalmological images to obtain preprocessed images includes: The real-world ophthalmological image is adjusted according to a preset resolution to obtain a resolution-adjusted ophthalmological image; The resolution-adjusted ophthalmic image is normalized to obtain a normalized image; The normalized image is augmented to obtain an enhanced image, wherein the data augmentation method includes at least one of random rotation, flipping, cropping, translation, scaling, brightness adjustment, contrast adjustment, saturation adjustment, and hue adjustment; The noise in the enhanced image is removed to obtain a denoised image; The denoised image is then cleaned to obtain a preprocessed image.

3. The method according to claim 1, characterized in that, The step of inputting the preprocessed image into a pre-built medical image generation model to generate a medical image includes: The preprocessed image is input into a pre-trained real-world quality control model to obtain image classification; Preprocessed images that meet the preset classification criteria are input into a pre-built medical image generation model to generate medical images.

4. The method according to claim 1, characterized in that, The training process of the medical image generation model includes: Collect real-world ophthalmology training images; The real-world ophthalmology training images were adjusted to fixed resolution images; The diffusion model is trained using the fixed-resolution image and preset prior information to obtain a medical image generation model.

5. A cross-modal universal eye disease screening and diagnosis device, characterized in that, include: The real-world ophthalmology image acquisition module is used to acquire multimodal real-world ophthalmology images; The preprocessing module is used to preprocess the real-world ophthalmological images to obtain preprocessed images; The medical image generation module is used to input the preprocessed image into a pre-built medical image generation model to generate a medical image; A general feature extraction module is used to extract general features of the medical image using a pre-trained base model. The base model is based on a visual transformer framework and uses a mask image modeling method, which is generated by pre-training on unlabeled ophthalmic images from the real world. A cross-modal universal eye disease screening model training module is used to train a cross-modal universal eye disease screening model using the aforementioned universal features; The screening prediction result output module is used to input the image to be screened into the cross-modal general eye disease screening model and output the screening prediction result.

6. The apparatus according to claim 5, characterized in that, The preprocessing module includes: The image adjustment submodule is used to adjust the real-world ophthalmic image according to a preset resolution to obtain a resolution-adjusted ophthalmic image; The normalization submodule is used to normalize the resolution-adjusted ophthalmic image to obtain a normalized image; The data augmentation submodule is used to perform data augmentation on the normalized image to obtain an enhanced image. The data augmentation method includes at least one of random rotation, flipping, cropping, translation, scaling, brightness adjustment, contrast adjustment, saturation adjustment, and hue adjustment. The noise reduction submodule is used to remove noise from the enhanced image to obtain a denoised image. The data cleaning submodule is used to clean the denoised image to obtain a preprocessed image.

7. The apparatus according to claim 5, characterized in that, The medical image generation module includes: The image classification submodule is used to input the preprocessed image into a pre-trained real-world quality control model to obtain image classification. The medical image generation submodule is used to input preprocessed images that meet the preset classification into a pre-built medical image generation model to generate medical images.

8. The apparatus according to claim 5, characterized in that, The training process of the medical image generation model includes: The real-world ophthalmology training image acquisition submodule is used to acquire real-world ophthalmology training images; The resolution adjustment submodule is used to adjust the real-world ophthalmology training images to fixed resolution images; The medical image generation module training submodule is used to train the diffusion model using the fixed-resolution image and preset prior information to obtain the medical image generation model.

9. An electronic device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the cross-modal universal eye disease screening method according to any one of the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the cross-modal universal eye disease screening method according to any one of claims 1-4.

Images