Vision transformer system and method for assisting in ophthalmic disease diagnosis, and device and storage medium

Abstract

A vision transformer system and method for assisting in ophthalmic disease diagnosis, and a device and a storage medium, which solve the problems in the prior art in the field of ophthalmic disease diagnosis assistance, such as being singular in functionality, high in data dependency, great in demands for computational resources and weak in interpretability. The system comprises an input processing module, a vision transformer backbone, a mixture-of-experts layer, a decoder head and an output generation module. The vision transformer system for assisting in ophthalmic disease diagnosis can process various ophthalmic imaging modalities and execute multiple tasks. Moreover, the coordination and cooperation of the input processing module, vision transformer backbone, mixture-of-experts layer, decoder head and output generation module in the system can reduce the computation requirements while maintaining the accuracy. Experts are dynamically selected on the basis of the complexity of an input image, thereby ensuring that optimal feature processing can be provided for different inputs, and both accuracy and efficiency are achieved; and a decoder structure can be customized for characteristics of different tasks, thereby ensuring the output precision for each task.

Description

Visual conversion systems, methods, devices, and storage media for auxiliary diagnosis of eye diseases. Technical Field

[0001] This invention relates to an artificial intelligence system for medical image analysis, specifically to a visual conversion system, method, device, and storage medium for auxiliary diagnosis of eye diseases. Background Technology

[0002] Visual impairment and eye diseases are significant public health issues worldwide. Statistics show that millions globally face the risk of vision loss or even blindness due to various eye diseases, severely reducing patients' quality of life and placing a heavy burden on healthcare systems. Early detection and accurate diagnosis are crucial for halting disease progression and preserving visual function. However, the complexity of eye diseases (such as multiple coexisting types and overlapping clinical manifestations) and the uneven distribution of medical resources (especially the shortage of ophthalmologists) make traditional manual diagnostic methods insufficient to meet the demands for efficient and accurate clinical care. Innovations in medical imaging technology have provided vital support for eye disease diagnosis. Currently, mainstream ophthalmic imaging modalities include fundus photography, optical coherence tomography (OCT, providing high-resolution images of the retinal layer structure), and fluorescein angiography (FFA, showing abnormal leakage or occlusion of retinal vessels). These technologies can visually reveal subtle lesions within the eye tissues.

[0003] The rise of artificial intelligence (AI) and deep learning technologies has provided new pathways for the automation of ophthalmic disease diagnosis. Deep learning models (such as convolutional neural networks (CNNs) have shown potential in single-modal medical image analysis; for example, through training, they can automatically identify and classify diabetic retinopathy features in fundus images. However, existing AI systems for ophthalmic disease diagnosis still have significant limitations: functional limitation, with most systems targeting only a single imaging modality or a single task, making it difficult to meet the actual needs of multimodal data fusion analysis or joint diagnosis of multiple diseases in clinical practice; and strong data dependence, with high costs and time consumption in acquiring high-quality labeled data, limiting the model's generalization ability in rare ophthalmic diseases or multicenter scenarios.

[0004] In recent years, the emergence of the transformer architecture has driven rapid technological development. Visual transformers (ViTs) have demonstrated outstanding performance in image analysis tasks, becoming a powerful tool for processing image data. However, applying ViTs to the specific scenario of multimodal, multi-task ophthalmic disease diagnosis has encountered numerous technical bottlenecks. For example, a prominent challenge in developing an AI system for ophthalmic disease diagnosis is acquiring large amounts of labeled training data, which is often costly and time-consuming in the medical field. Simultaneously, the computational resources required to run complex AI models are enormous, posing a significant obstacle in resource-constrained medical environments.

[0005] In summary, existing technologies in the field of assisted diagnosis of eye diseases suffer from limitations such as limited functionality, high data dependence, high computational resource requirements, and weak interpretability, failing to meet clinical needs for efficient, accurate, and universally applicable diagnostic tools. Therefore, there is an urgent need to develop a visual transformation system and method that can integrate multimodal eye disease image data, support multi-task analysis, reduce data and computational barriers, and possess reliable interpretability, in order to improve the accessibility and accuracy of assisted diagnosis of eye diseases. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a visual conversion system, method, device, and storage medium for auxiliary diagnosis of eye diseases, which solves the problems of limited functionality, high data dependence, large computational resource requirements, and weak interpretability in the field of auxiliary diagnosis of eye diseases.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] On one hand, the present invention provides a visual conversion system for auxiliary diagnosis of eye diseases, the system comprising:

[0011] The input processing module is used to receive and preprocess input ophthalmic images;

[0012] The visual transformer backbone is used to process prepared input ophthalmic images and generate a set of image labels;

[0013] A hybrid expert layer is used to receive image labels generated by the visual transformer backbone and selectively activate expert neural networks;

[0014] At least one decoder head, configured to perform a specific ophthalmic diagnostic task using the output of a selectively activated expert neural network in a hybrid expert layer;

[0015] The output generation module processes the results from the decoder header of a specific task and generates the final diagnostic output.

[0016] The input processing module includes:

[0017] Image block extractor: The image block extractor is used to segment an input ophthalmology image into multiple blocks;

[0018] The block embedding module is used to convert 16x16 image blocks into 768-dimensional embeddings.

[0019] The visual transformer backbone includes a multi-head self-attention network and a feedforward network.

[0020] The hybrid expert layer includes:

[0021] An expert pool, which contains multiple expert neural networks;

[0022] The router's expert pool contains 16 expert neural networks.

[0023] The decoder head includes a disease classification head, a prognosis head, and a biomarker detection head.

[0024] At least one router is configured to activate the first K subsets of expert neural networks, where K is less than the total number of expert neural networks in the expert pool.

[0025] On the other hand, the present invention also provides a visual conversion method for auxiliary diagnosis of eye diseases, the method comprising:

[0026] S1, Receive input ophthalmology image. It should be noted that after receiving the input ophthalmology image, the input processing module performs initial preprocessing on the received image.

[0027] S2, using the visual transformer backbone to generate a set of image labels from the input ophthalmological image, wherein when the visual transformer backbone processes the input image to generate a set of image labels, the image labels represent key features of the ophthalmological image;

[0028] S3, where at least one router selectively activates a subset of the expert neural network in the hybrid expert layer based on image labels;

[0029] S4, which performs a specific eye disease diagnosis task by using the output of a selectively activated expert neural network from at least one task-specific decoder head.

[0030] The method for receiving input ophthalmic images includes:

[0031] S101, Load the input ophthalmology image;

[0032] S102, Perform block embedding. The image block extractor segments the input image into blocks, and the block embedding module converts the segmented blocks into high-dimensional embeddings.

[0033] S103, the visual transformer backbone processes the embedding blocks, with multi-head self-attention and feedforward network components working together to convert the embedding blocks into a set of image labels;

[0034] S104, By routing image tags through a hybrid expert layer, the router selectively activates a subset of expert neural networks from the expert pool based on the features of the image tags.

[0035] In another aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the visual conversion method for auxiliary diagnosis of eye diseases.

[0036] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual conversion method for auxiliary diagnosis of eye diseases.

[0037] (III) Beneficial Effects

[0038] This invention provides a visual conversion system, method, device, and storage medium for auxiliary diagnosis of eye diseases. Compared with the prior art, it has the following advantages:

[0039] 1. In this embodiment of the invention, the visual transformation system for auxiliary diagnosis of eye diseases can process various ophthalmic imaging modalities and perform multiple tasks. The input processing module, visual transformer backbone, hybrid expert layer, decoder head, and output generation module work together to reduce computational requirements while maintaining accuracy. Experts are dynamically selected according to the complexity of the input image to ensure that different inputs can obtain optimal feature processing, balancing accuracy and efficiency. Furthermore, the decoder structure can be customized for the characteristics of different tasks to ensure the output accuracy of each task.

[0040] 2. In this embodiment of the invention, the visual conversion system for auxiliary diagnosis of eye diseases also provides an input processing module. The input processing module consists of an image block extractor and a block embedding module. The image block extractor makes the image processing unit smaller, which helps to process the image more finely and accurately in the subsequent process. It can capture more subtle details and features in the image, laying the foundation for accurate diagnosis of eye diseases. Compared with processing the entire image directly, it can reduce the complexity of image processing, improve processing efficiency, and enable the system to preprocess the image faster, reduce the patient's waiting time, and improve diagnostic efficiency. Through the embedding conversion process of the block embedding module, the original pixel information of the image block is transformed into a higher-dimensional embedding representation, which can more fully explore and express the potential features in the image block, providing a richer information basis for extracting relevant features from the input ophthalmological image, and helping to improve the accuracy of eye disease diagnosis.

[0041] 3. In the embodiments of the present invention, the visual conversion system for auxiliary diagnosis of eye diseases has the advantages of multimodal data compatibility, efficient multi-task processing, low computing resource requirements, and strong clinical scalability. It ultimately solves the pain points of "single data, fragmented tasks, limited resources, and weak interpretability" in traditional eye disease diagnosis, and provides highly accessible and accurate AI-assisted diagnostic tools for resource-scarce areas.

[0042] 4. In this embodiment of the invention, the system supports input of multiple ophthalmic imaging modes such as fundus images, OCT, and FFA, overcoming the limitation of traditional diagnostic systems that can only process single-modal data. Through unified feature extraction of multimodal data via a visual transformer backbone, cross-modal information fusion analysis is achieved, providing doctors with more comprehensive lesion assessment criteria and avoiding missed diagnoses or misjudgments that may result from single-modal data.

[0043] 5. In this embodiment of the invention, the system adopts a sparse hybrid expert layer and a top K-gated network design, activating only the subset of experts most relevant to the current input or task. This mechanism significantly reduces redundant computation (e.g., simple cases require only a small number of experts to process), greatly reducing energy consumption and processing time while maintaining diagnostic accuracy. It provides two variants: multi-gate MoE-ViT (independent router for each task + shared expert pool) and task-conditional MoE-ViT (shared router + task embedding driver). No additional routers are needed for new tasks; functional expansion can be achieved simply by expanding the expert network or adjusting the task embedding. This design significantly reduces system upgrade and maintenance costs, enabling rapid adaptation to newly emerging clinical ophthalmic disease diagnostic needs (such as rare disease or multimodal joint diagnostic tasks), and improving the system's long-term usability and adaptability.

[0044] Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 shows the architecture of a visual conversion system used for auxiliary diagnosis of eye diseases.

[0047] Figure 2 shows the architecture diagram of the input processing module.

[0048] Figure 3 shows the architecture diagram of a multi-gate vision transformer.

[0049] Figure 4 shows a flowchart of the implementation of a visual conversion method for auxiliary diagnosis of eye diseases.

[0050] Figure 5 shows a schematic diagram of the implementation process of the method for receiving input ophthalmic images.

[0051] Figure 6 illustrates the key steps of a visual conversion system for auxiliary diagnosis of eye diseases.

[0052] Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This application provides a visual conversion system, method, device, and storage medium for auxiliary diagnosis of eye diseases. It solves the problems of existing technologies in the field of auxiliary diagnosis of eye diseases, such as single function, high data dependence, large computing resource requirements, and weak interpretability. It enables dynamic selection of experts based on the complexity of the input image, ensuring that different inputs can obtain optimal feature processing, balancing accuracy and efficiency. Furthermore, it can customize the decoder structure for the characteristics of different tasks to ensure the output accuracy of each task.

[0055] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0056] This application achieves efficient, accurate, and flexible conversion from input to diagnostic results through a full-process design of "multimodal preprocessing, efficient feature extraction, dynamic expert scheduling, and multi-task parallel output," which solves the pain points of "single data, fragmented tasks, and limited resources" in traditional eye disease diagnosis.

[0057] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0058] Example 1:

[0059] In recent years, the emergence of the transformer architecture has driven rapid technological development. Visual transformers (ViTs) have demonstrated outstanding performance in image analysis tasks, becoming a powerful tool for processing image data. However, applying ViTs to the specific scenario of multimodal, multi-task ophthalmic disease diagnosis has encountered numerous technical bottlenecks. For example, a prominent challenge in developing an AI system for ophthalmic disease diagnosis is acquiring large amounts of labeled training data, which is often costly and time-consuming in the medical field. Simultaneously, the computational resources required to run complex AI models are enormous, posing a significant obstacle in resource-constrained medical environments.

[0060] In a first aspect, the present invention proposes a visual conversion system for auxiliary diagnosis of eye diseases. Figure 1 shows an architecture diagram of the visual conversion system for auxiliary diagnosis of eye diseases, which includes:

[0061] Input processing module 100 is configured to receive and preprocess input ophthalmic images. In some cases, the input ophthalmic images may be fundus images, optical coherence tomography (OCT) images, or fluorescein angiography (FFA) images. Input processing module 100 may prepare image data for further processing by subsequent components of a visual conversion system used for assisted diagnosis of eye diseases.

[0062] The visual transformer backbone 200 processes the prepared input ophthalmic images and generates a set of image labels. Furthermore, the visual transformer backbone 200 can apply self-attention mechanisms and other transformations to extract relevant features from the input images.

[0063] A hybrid expert layer 300 comprises multiple expert neural networks and at least one router 320. The hybrid expert layer 300 is used to receive image labels generated by the visual transformer backbone 200. In some cases, the router within the hybrid expert layer 300 can selectively activate a subset of the expert neural networks based on the received image labels. This selective activation may achieve efficient processing of image data by utilizing only the expert neural networks most relevant to a given input.

[0064] At least one decoder head 400, comprising multiple task-specific components, each task-specific decoder head 400 configured to perform a specific ophthalmic diagnostic task using the output of a selectively activated expert neural network in a hybrid expert layer 300. These task-specific tasks may include, but are not limited to, disease classification, prognosis determination, or biomarker detection.

[0065] Output generation module 500 processes the results from decoder head 400 for a specific task and generates final diagnostic output. Output generation module 500 may format and present the diagnostic results in a manner suitable for interpretation or further analysis by medical professionals.

[0066] It should be noted that the components of the visual transformation system for auxiliary diagnosis of eye diseases can work together to process input ophthalmic images through various analysis stages, and efficiently perform multiple ophthalmic diagnostic tasks by leveraging the advantages of visual transformers and hybrid expert architecture. The visual transformation system for auxiliary diagnosis of eye diseases can be simply described as the EyeMOE system architecture.

[0067] In this embodiment of the invention, the visual transformation system for auxiliary diagnosis of eye diseases can process various ophthalmic imaging modalities and perform multiple tasks. The input processing module 100, the visual transformer backbone 200, the hybrid expert layer 300, the decoder head 400, and the output generation module 500 work together to reduce computational requirements while maintaining accuracy. Experts are dynamically selected based on the complexity of the input image to ensure that different inputs can obtain optimal feature processing, balancing accuracy and efficiency. Furthermore, the decoder structure can be customized for the characteristics of different tasks to ensure the output accuracy of each task.

[0068] In this embodiment of the invention, an input processing module 100 is provided. Figure 2 shows an architectural diagram of the input processing module 100. The input processing module 100 includes:

[0069] Image block extractor 110 is used to segment an input ophthalmology image into multiple blocks, and image block extractor 110 can have a block dimension of 16x16 pixels.

[0070] The block embedding module 120 processes the input ophthalmic image after the image block extractor 110 segments it into blocks. The block embedding module 120 converts 16x16 image blocks into 768-dimensional embeddings. This conversion process may transform the spatial information of the image blocks into a format suitable for further processing by subsequent components of the system.

[0071] The embedded blocks generated by the block embedding module 120 can be passed to the visual transformer backbone 200 for further processing. The visual transformer backbone 200 then applies various transformations to the embedded blocks to extract relevant features from the input ophthalmic image. Furthermore, by preprocessing the input ophthalmic image through block extraction and embedding, the input processing module 100 can prepare image data for efficient analysis by subsequent components of the system, including the hybrid expert layer 300 and the task-specific decoder head 400.

[0072] In this embodiment of the invention, the visual transformer backbone 200 may be configured to process embedding blocks generated by the block embedding module 120. In some cases, the visual transformer backbone 200 may be a data-efficient visual transformer (DeiT). The visual transformer backbone 200 may be designed to efficiently extract relevant features from the input ophthalmic image while minimizing computational requirements.

[0073] In this embodiment of the invention, an input processing module 100 is provided, which consists of an image block extractor 110 and a block embedding module 120. The image block extractor 110 makes the image processing unit smaller, which helps to process the image more finely and accurately in the subsequent process. It can capture more subtle details and features in the image, laying the foundation for accurate diagnosis of eye diseases. Compared with processing the entire image directly, it can reduce the complexity of image processing, improve processing efficiency, enable the system to preprocess the image faster, reduce the patient's waiting time, and improve diagnostic efficiency. Through the embedding transformation process of the block embedding module 120, the original pixel information of the image block is transformed into a higher-dimensional embedding representation, which can more fully explore and express the potential features in the image block, providing a richer information basis for extracting relevant features from the input ophthalmological image, and helping to improve the accuracy of eye disease diagnosis.

[0074] The visual transformer backbone 200 includes multiple components that work together to process embedded blocks. The visual transformer backbone 200 includes a multi-head self-attention network 210 and a feedforward network 220. These components may be arranged in a series of encoder iterations to progressively transform the input data.

[0075] It should be noted that the visual transformer backbone 200 can use 12 encoder iterations. Each encoder iteration may involve processing data through multi-head self-attention 210 and feedforward network 220. This iterative process may allow the visual transformer backbone 200 to capture complex relationships and patterns in the input ophthalmic image.

[0076] Multi-head self-attention 210 can be configured to analyze the relationships between different parts of the input data. By applying multiple attention mechanisms in parallel, multi-head self-attention 210 can capture various types of dependencies and patterns within embedded blocks.

[0077] Following multi-head self-attention 210, the feedforward network 220 can further process the attention-processed data. The feedforward network 220 can apply non-linear transformations to the data, allowing the visual transformer backbone 200 to model complex functions and extract high-level features from the input.

[0078] By repeatedly applying multi-head self-attention 210 and feedforward network 220 in encoder iterations, the visual transformer backbone 200 is able to generate a set of image labels. These image labels may represent high-level encodings of the input ophthalmic images, capturing relevant features and patterns that can be used for subsequent diagnostic tasks.

[0079] The image labels generated by the visual transformer backbone 200 can then be passed to the hybrid expert layer 300 for further processing. By efficiently converting the input ophthalmic image into a set of information-rich image labels, the visual transformer backbone 200 plays a key role in preparing data for subsequent stages of the system, including task-specific diagnostic processes performed by the task-specific decoder head 400.

[0080] In this embodiment of the invention, a visual transformer backbone 200 is provided. Through the collaborative work of multiple components, the visual transformer backbone 200 can process embedded blocks and generate image tags. These image tags represent high-level encodings of the input ophthalmic image, capturing relevant features and patterns that can be used for subsequent diagnostic tasks. These image tags are then passed to a hybrid expert layer 300 for further processing. After multi-head self-attention 210, a feedforward network 220 can further process the attention-processed data and apply nonlinear transformations to the data. This allows the visual transformer backbone 200 to model complex functions and extract high-level features from the input. This helps to transform the captured raw features into more representative and discriminative high-level features, providing stronger information support for subsequent diagnostic tasks.

[0081] In this embodiment of the invention, a multi-head self-attention mechanism 210 is added to the hybrid expert layer 300 to construct a non-pipelined multimodal + multi-task system, rather than a traditional pipelined architecture. Its core advantage lies in overcoming the limitations of traditional architectures in modality fusion, task collaboration, and interpretability through a "dynamic collaboration + global perception" design. Simultaneously, it significantly broadens the application boundaries of the visual transformer backbone 200 in medical AI. The sparse activation mechanism of MOE (activating only relevant experts) combined with the focusing characteristics of attention (focusing only on key tokens) avoids the problem of repetitive computation at each stage in a pipelined architecture. For example, when processing multimodal input, a pipeline might need to extract full-image features for each modality separately, while the MOE + attention function processes multimodality simultaneously through a unified attention mechanism, allocating computational resources only in key regions (such as lesion sites), significantly reducing inference latency and meeting the real-time requirements of clinical practice.

[0082] In this embodiment of the invention, a hybrid expert layer 300 is also provided, the hybrid expert layer 300 comprising:

[0083] Expert pool 310, which contains multiple expert neural networks, each of which is used to process features or patterns of a type in image tags;

[0084] Router 320, expert pool 310 contains 16 expert neural networks. This configuration may allow for diverse specialized processing capabilities within the hybrid expert layer 300;

[0085] In Mixture of Experts (MOE) systems, unbalanced expert loads and sudden failures (such as performance drops and parameter crashes due to overload) are core issues affecting system robustness. Traditional MOE routers typically select experts dynamically based solely on input features (e.g., Top-k routing), but lack proactive monitoring of expert load status and performance levels. This can easily lead to some experts "crashing" due to overload (e.g., latency spikes and accuracy drops), while low-load experts are not fully utilized, ultimately reducing overall system efficiency and reliability. Router 320 can be configured to selectively activate a subset of expert neural networks from the expert pool 310 based on input image labels. This selective activation can achieve sparse activation of the expert neural network, allowing for efficient use of computing resources. In this embodiment, to achieve expert failure handling and load balancing, predefined expert status indicators (load indicators, performance indicators), threshold triggering, and dynamic response mechanisms are also included. Combined with dynamic routing probability adjustment strategies and hierarchical fault response mechanisms, the system's stability and reliability are significantly improved. This design not only avoids system failures caused by expert overload crashes but also optimizes computing resource utilization through load balancing.

[0086] Router 320 is capable of activating the top K subsets of expert neural networks, with at least one router 320 configured to activate the top K subsets of expert neural networks, where K is less than the total number of expert neural networks in expert pool 310. This approach may allow hybrid expert layers 300 to focus on the most relevant expert neural network for a given input, potentially improving both efficiency and accuracy simultaneously.

[0087] In this embodiment of the invention, the selection count K can be set to different values ​​according to the specific requirements of the system. In some cases, the selection count K can be 4. For 16 experts and 4 selection counts, the router 320 may only activate 25% of the available expert neural networks for each input, thereby reducing computational overhead while maintaining diagnostic accuracy. Specifically, during the training of the expert neural networks, specific modal experts are defined. Through the specific definition of modal experts, such as fundus focusing and OCT focusing, each expert neural network can specifically perform in-depth analysis of image features of a particular modality to achieve accurate diagnosis. For example, experts focusing on fundus imaging can meticulously capture subtle vascular changes and retinal color abnormalities in fundus images, providing reliable evidence for the diagnosis of fundus diseases. Furthermore, the expert neural network incorporates a self-organizing network (SON) mechanism, allowing experts to autonomously optimize their processing capabilities for specific features through network training, further enhancing the system's long-term usability and generalization. The SON mechanism monitors the feature distribution of the input image labels (such as lesion features of newly added rare eye diseases) and automatically adjusts the expert's network parameters (such as convolutional kernel weights and attention head orientation) to more accurately capture newly emerging feature patterns (such as specific lesions of rare hereditary eye diseases), without requiring manual redesign of the expert structure. Through performance evaluation during training (such as classification accuracy and task loss value), the SON mechanism can identify and eliminate inefficient experts (such as redundant experts contributing very little to the current dataset) while strengthening the parameters of high-contribution experts (such as increasing their routing weights), achieving "self-simplification" of the expert pool and reducing computational redundancy.

[0088] It should be noted that the expert neural network also includes a caching network. The visual transformer backbone and 300 hybrid expert layers are pre-trained on an unlabeled, large-scale ophthalmology dataset (self-supervised) to reduce reliance on labeled data. Furthermore, for each network routing path, we define specific modality experts, thereby reducing latency.

[0089] During training, the expert neural network defines the Load-Importance Balance Loss (LIB Loss) with a weight of 0.01. LIB Loss (weight = 0.01) serves as a key training mechanism, mathematically constraining the distribution of expert load and task matching, directly addressing core issues such as expert crashes and uneven load distribution. It's important to note that Load-Importance Balance Loss (LIB Loss) is a core component of VisionMOE. By constraining the distribution of expert load and task importance, it achieves fair expert utilization (avoiding some experts being idle or overloaded) and stable multi-task training (preventing low-performance experts from outputting incorrect results). This embodiment adopts this mechanism and optimizes parameters for ophthalmology scenarios, overcoming the problems of uneven expert load, performance crashes caused by expert overload, and insufficient stability in multi-task training in existing technologies.

[0090] The process of routing image tags via the hybrid expert layer 300 may involve several steps. First, router 320 may analyze the incoming image tags to determine which expert neural networks are best suited to process the current input. Based on this analysis, router 320 may then selectively activate a subset of expert neural networks from the expert pool 310.

[0091] The activated expert neural networks can then process image labels, applying their specialized knowledge to extract relevant features or patterns. The outputs from these activated expert neural networks may be combined or aggregated in some way and then passed to the task-specific decoder head 400 for further processing.

[0092] By implementing sparse activation and selective routing, the hybrid expert layer 300 ensures the system can efficiently process various ophthalmic images while adapting to the specific features of each input. This approach may enable the system to perform multiple ophthalmic diagnostic tasks with improved efficiency and accuracy.

[0093] It should be noted that, in this embodiment of the invention, the decoder head 400 for a specific task includes a disease classification head 410, a prognosis head 420, and a biomarker detection head 430. Each head can be designed to perform a specific type of eye disease diagnostic task.

[0094] In this embodiment of the invention, the disease classification head 410 is capable of classifying various eye diseases based on processed image data. The disease classification head 410 is used to analyze the output from the hybrid expert layer 300 to identify indicative patterns or features of specific eye diseases. In some cases, the disease classification head 410 can output a probability distribution of multiple possible eye diseases or conditions.

[0095] The prognostic head 420 is used to provide disease prognosis or progression prediction. It analyzes processed image data to estimate the likelihood of disease progression or predict potential patient outcomes. In some cases, the prognostic head 420 can also generate time-based predictions or risk assessments for various eye diseases.

[0096] Biomarker detection head 430 is configured to detect systemic biomarkers from ophthalmic images. Biomarker detection head 430 is used to analyze output from the hybrid expert layer 300 to identify indicators of systemic health conditions that may be visible in the eye. In some cases, biomarker detection head 430 is able to detect biomarkers associated with diabetes, hypertension, or other systemic diseases that may affect the eye.

[0097] In this embodiment, the decoder head 400 for each specific task may generate diagnostic output based on the output of the selectively activated expert neural network in the hybrid expert layer 300. The diagnostic output may include disease classification, prognostic prediction, or detected biomarkers, depending on the specific task performed by each decoder head.

[0098] In some cases, the decoder heads 400 for specific tasks may run in parallel, allowing the system to perform multiple diagnostic tasks on a single input ophthalmic image simultaneously. This parallel processing capability may enable comprehensive ophthalmic disease diagnosis from a single image analysis.

[0099] The output from the task-specific decoder head 400 may be passed to the output generation module 500 for further processing and presentation. By utilizing specialized decoder heads 400 for different diagnostic tasks, the system may provide comprehensive analysis of ophthalmic images, potentially improving the accuracy and efficiency of eye disease diagnosis.

[0100] The system may include multiple variants to accommodate different processing requirements and task configurations. Two such variants are the multi-gate variant and the task-conditional variant, each offering unique advantages in terms of flexibility and efficiency.

[0101] In this embodiment of the invention, the system supports input from multiple ophthalmic imaging modes, such as fundus images, OCT, and FFA, overcoming the limitation of traditional diagnostic systems that can only process single-modal data. Through unified feature extraction of multimodal data using a visual transformer backbone, cross-modal information fusion analysis is achieved, providing doctors with more comprehensive lesion assessment criteria and avoiding missed diagnoses or misjudgments that may result from single-modal data.

[0102] The system employs a sparse hybrid expert layer and a top-K-gated network design, activating only the subset of experts most relevant to the current input or task. This mechanism significantly reduces redundant computation (e.g., simple cases require only a small number of experts), greatly reducing energy consumption and processing time while maintaining diagnostic accuracy. It offers two variants: multi-gate MoE-ViT (independent routers for each task + shared expert pool) and task-conditional MoE-ViT (shared routers + task embedding drivers). Functionality expansion is achieved simply by extending the expert network or adjusting the task embedding, without needing to add additional routers for new tasks. This design significantly reduces system upgrade and maintenance costs, enabling rapid adaptation to new clinical ophthalmic diagnostic needs (such as rare disease or multimodal joint diagnostic tasks), thus improving the system's long-term usability and adaptability.

[0103] In this embodiment of the invention, the visual transformation system for auxiliary diagnosis of eye diseases also includes a multi-gate visual transformer 600. Figure 3 shows the architecture diagram of the multi-gate visual transformer 600, which represents an implementation of a multi-gate variant.

[0104] The multi-door vision transformer 600 may include a shared vision tag 602, which serves as the input to multiple parallel routers.

[0105] In the multi-gate variant, the system employs multiple parallel routers, each dedicated to a specific task. The multi-gate vision transformer 600 includes a first task router 604, a second task router 606, and a third task router 608. Each task router is configured to independently process the shared visual tag 602.

[0106] The multi-gate variant comprises a shared expert pool 610 containing multiple expert modules. In some cases, the shared expert pool 610 includes an initial expert module 612 and an end expert module 616, as well as other expert modules, such as expert module 614. Each parallel task router may select a top K subset of experts from the shared expert pool 610 based on input tags and specific task requirements.

[0107] After expert selection and processing, the output may be directed to a task decoder 622 for a specific task. The multi-gate vision transformer 600 includes a first task decoder 618 associated with a first task router 604, a second task decoder 620 associated with a second task router 606, and a third task decoder 622 associated with a third task router 608. Each task decoder is capable of performing a specific ophthalmic diagnostic task using the output from the selected expert module.

[0108] Multi-gate variants offer advantages when different tasks require different routing strategies. By employing a separate parallel router for each task, the system can potentially optimize the expert selection process for each specific diagnostic task, potentially improving accuracy and efficiency.

[0109] In addition to the multi-gate variant, the system may also implement a task-conditional variant. In the task-conditional variant, the system may utilize a single shared router that receives the connection tag and task embedding.

[0110] Task-conditional variants may employ a lightweight 64-dimensional task embedding. This task embedding can be concatenated with visual tags generated by the visual transformer backbone 200. The combined tags and task embedding can then be processed by a shared router to determine which expert modules to activate.

[0111] By using task embeddings, task condition variations allow for scalable expansion of system capabilities without the need to add additional routers. Shared routers can learn to interpret task embeddings and select the most appropriate expert module for each specific task.

[0112] Task conditional variants offer advantages in terms of efficiency and flexibility. Using a single shared router may reduce computational overhead compared to multi-gate variants, while task embedding methods may allow for the easy addition of new diagnostic tasks without major architectural changes.

[0113] The system's multi-gate and task-conditional variants can leverage a hybrid expert layer 300 to dynamically route computation to the expert module most relevant to each input and task. This approach could enable efficient processing of ophthalmic images across multiple diagnostic tasks, potentially improving the system's overall performance and adaptability.

[0114] Example 2:

[0115] Secondly, the present invention also provides a visual conversion method for auxiliary diagnosis of eye diseases. Figure 4 shows a flowchart of the implementation of the visual conversion method for auxiliary diagnosis of eye diseases, the method comprising:

[0116] S1, Receive input ophthalmological image. It should be noted that after receiving the input ophthalmological image, the input processing module 310 can perform initial preprocessing on the received image.

[0117] S2, using the visual transformer backbone 200 to generate a set of image labels from the input ophthalmic image, wherein when the visual transformer backbone 200 processes the input image to generate a set of image labels, the image labels represent key features of the ophthalmic image;

[0118] S3, at least one router 320 selectively activates a subset of expert neural networks in the hybrid expert layer 300 based on image labels. It should be noted that the router 320 in the hybrid expert layer 300 is able to analyze image labels and determine which expert neural networks are best suited to process the current input.

[0119] S4, a specific eye disease diagnosis task is performed by at least one task-specific decoder head 400 using the output of a selectively activated expert neural network.

[0120] This invention provides a visual transformation method for auxiliary diagnosis of eye diseases. The method supports input from various ophthalmic imaging modalities, such as fundus images, OCT, and FFA, through an input preprocessing module. Standardized preprocessing eliminates data differences between different devices and formats, ensuring that the subsequent visual transformer backbone can stably extract features. This overcomes the limitation of traditional systems that only support a single modality. The visual transformer backbone, through a multi-head self-attention mechanism and multi-round encoder iteration, can capture both the global correlation of ophthalmic images and focus on local details. The generated image tags representing key features contain multi-dimensional semantic information, providing high-quality input for subsequent dynamic computation of the hybrid expert layer. Multiple task-specific decoder heads in the transformation method steps can be executed in parallel, simultaneously outputting multi-dimensional diagnostic results, rather than the serial output of traditional single-task models. This achieves efficient, accurate, and flexible transformation from input to diagnostic results, addressing the pain points of "single data, fragmented tasks, and limited resources" in traditional eye disease diagnosis.

[0121] It should be noted that the embodiments of the present invention also provide a method for receiving input ophthalmic images. Figure 5 shows a schematic diagram of the implementation process of the method for receiving input ophthalmic images. The method for receiving input ophthalmic images includes:

[0122] S101, Load the input ophthalmology image;

[0123] S102, perform block embedding, the image block extractor 110 segments the input image into blocks, and the block embedding module 120 converts the segmented blocks into high-dimensional embeddings;

[0124] S103, the visual transformer backbone 200 processes the embedded blocks, and the multi-head self-attention 210 and feedforward network 220 components work together to convert the embedded blocks into a set of image labels;

[0125] S104, by routing image tags through the hybrid expert layer 300, the router 320 selectively activates a subset of the expert neural network from the expert pool 310 based on the features of the image tags.

[0126] In this embodiment of the invention, Figure 6 illustrates key steps in the visual conversion system flow for assisted diagnosis of eye diseases. After receiving an input ophthalmic image in step 1, the system generates image tags in step 2 and sends these tags for further processing in step 3. Step 4 in Figure 6 illustrates the selective activation of the expert neural network, corresponding to the operation of the hybrid expert layer 300. In step 5, the selected expert neural network is used to process the image tags. The final stages of the method operation are represented in steps 7 and 8 of Figure 6. In step 7, the task-specific decoder head 400 is able to perform a specific eye disease diagnosis task using the output from the selectively activated expert neural network. This step may involve the disease classification head 410, prognosis head 420, and biomarker detection head 430 working in parallel to analyze the processed data. Finally, in step 8, diagnostic results for multiple eye disease tasks are output. The output generation module 500 compiles and formats these results for presentation to medical professionals or for further analysis.

[0127] The systems and methods described in this invention can be implemented using computer hardware and software components, including a processor that executes instructions stored on a non-transitory computer-readable storage medium. When executed, these instructions may cause the processor to perform various steps of a multi-tasking ophthalmic diagnostic method.

[0128] This approach could allow for more efficient and accurate diagnosis of multiple eye diseases from a single input image, potentially improving patient care and resource utilization in ophthalmological practice. The described system may be particularly useful in environments with limited specialist care resources, as it may enable comprehensive analysis of ophthalmic images across multiple diagnostic tasks.

[0129] It is understood that the visual conversion method for auxiliary diagnosis of eye diseases provided in the embodiments of the present invention corresponds to the visual conversion system for auxiliary diagnosis of eye diseases described above. The explanation, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the visual conversion system for auxiliary diagnosis of eye diseases, and will not be repeated here.

[0130] Example 3:

[0131] Thirdly, the present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, implements the steps of the visual conversion method for auxiliary diagnosis of eye diseases described in any of the above embodiments and preferred embodiments, the method mainly including:

[0132] S1, Receive input ophthalmological image. It should be noted that after receiving the input ophthalmological image, the input processing module 310 can perform initial preprocessing on the received image.

[0133] S2, using the visual transformer backbone 200 to generate a set of image labels from the input ophthalmic image, wherein when the visual transformer backbone 200 processes the input image to generate a set of image labels, the image labels represent key features of the ophthalmic image;

[0134] S3, at least one router 320 selectively activates a subset of expert neural networks in the hybrid expert layer 300 based on image labels. It should be noted that the router 320 in the hybrid expert layer 300 is able to analyze image labels and determine which expert neural networks are best suited to process the current input.

[0135] S4, a specific eye disease diagnosis task is performed by at least one task-specific decoder head 400 using the output of a selectively activated expert neural network.

[0136] It is understood that the computer device provided in the embodiments of the present invention corresponds to the visual conversion system and method for auxiliary diagnosis of eye diseases described above. The explanation, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the visual conversion system and method for auxiliary diagnosis of eye diseases, and will not be repeated here.

[0137] Example 4:

[0138] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the steps of the visual conversion method for auxiliary diagnosis of eye diseases described in any of the above embodiments and preferred embodiments, the method mainly comprising:

[0139] S1, Receive input ophthalmological image. It should be noted that after receiving the input ophthalmological image, the input processing module 310 can perform initial preprocessing on the received image.

[0140] S2, using the visual transformer backbone 200 to generate a set of image labels from the input ophthalmic image, wherein when the visual transformer backbone 200 processes the input image to generate a set of image labels, the image labels represent key features of the ophthalmic image;

[0141] S3, at least one router 320 selectively activates a subset of expert neural networks in the hybrid expert layer 300 based on image labels. It should be noted that the router 320 in the hybrid expert layer 300 is able to analyze image labels and determine which expert neural networks are best suited to process the current input.

[0142] S4, a specific eye disease diagnosis task is performed by at least one task-specific decoder head 400 using the output of a selectively activated expert neural network.

[0143] It is understood that the computer device provided in the embodiments of the present invention corresponds to the visual conversion system, method and device for auxiliary diagnosis of eye diseases described above. The explanation, examples and beneficial effects of the relevant content can be referred to the corresponding content in the visual conversion system, method and device for auxiliary diagnosis of eye diseases, and will not be repeated here.

[0144] In summary, compared with existing technologies, it has the following beneficial effects:

[0145] 1. In this embodiment of the invention, the visual transformation system for auxiliary diagnosis of eye diseases can process various ophthalmic imaging modalities and perform multiple tasks. The input processing module 100, the visual transformer backbone 200, the hybrid expert layer 300, the decoder head 400, and the output generation module 500 work together to reduce computational requirements while maintaining accuracy. Experts are dynamically selected based on the complexity of the input image to ensure that different inputs can obtain optimal feature processing, balancing accuracy and efficiency. Furthermore, the decoder structure can be customized for the characteristics of different tasks to ensure the output accuracy of each task.

[0146] 2. In this embodiment of the invention, an input processing module 100 is provided, which consists of an image block extractor 110 and a block embedding module 120. The image block extractor 110 makes the image processing unit smaller, which helps to process the image more finely and accurately in the future. It can capture more subtle details and features in the image, laying the foundation for accurate diagnosis of eye diseases. Compared with processing the entire image directly, it can reduce the complexity of image processing, improve processing efficiency, enable the system to preprocess the image faster, reduce the patient's waiting time, and improve diagnostic efficiency. Through the embedding transformation process of the block embedding module 120, the original pixel information of the image block is transformed into a higher-dimensional embedding representation, which can more fully explore and express the potential features in the image block, providing a richer information basis for extracting relevant features from the input ophthalmological image in the future, which helps to improve the accuracy of eye disease diagnosis.

[0147] 3. In this embodiment of the invention, a visual transformer backbone 200 is provided. Through the collaborative work of multiple components, the visual transformer backbone 200 can process embedded blocks and generate image tags. These image tags represent high-level encodings of the input ophthalmic image, capturing relevant features and patterns that can be used for subsequent diagnostic tasks. These image tags are then passed to the hybrid expert layer 300 for further processing. After multi-head self-attention 210, the feedforward network 220 can further process the attention-processed data and apply nonlinear transformations to the data. This allows the visual transformer backbone 200 to model complex functions and extract high-level features from the input. This helps to transform the captured raw features into more representative and discriminative high-level features, providing stronger information support for subsequent diagnostic tasks.

[0148] 4. In this embodiment of the invention, a visual transformation method for auxiliary diagnosis of eye diseases is provided. The input preprocessing module supports input from multiple ophthalmic imaging modalities such as fundus images, OCT, and FFA. Standardized preprocessing eliminates data differences between different devices and formats, ensuring that the subsequent visual transformer backbone can stably extract features. This overcomes the limitation of traditional systems that only support a single modality. The visual transformer backbone, through a multi-head self-attention mechanism and multi-round encoder iteration, can capture both the global correlation of ophthalmic images and focus on local details. The generated image markers representing key features contain multi-dimensional semantic information, providing high-quality input for the subsequent dynamic computation of the hybrid expert layer. Multiple task-specific decoder heads in the transformation method steps can be executed in parallel, simultaneously outputting multi-dimensional diagnostic results, rather than the serial output of traditional single-task models. This achieves efficient, accurate, and flexible transformation from input to diagnostic results, addressing the pain points of "single data, fragmented tasks, and limited resources" in traditional eye disease diagnosis.

[0149] 5. During training, the Load-Importance Balance Loss (LIB Loss) is defined with a weight of 0.01. LIB Loss (weight = 0.01) serves as a key training mechanism, mathematically constraining the distribution of expert load and task matching, directly addressing core issues such as expert crashes and uneven load distribution. It's important to note that Load-Importance Balance Loss (LIB Loss) is a core component of VisionMOE. By constraining the distribution of expert load and task importance, it achieves fair expert utilization (avoiding some experts being idle or overloaded) and stable multi-task training (preventing low-performance experts from outputting incorrect results). This embodiment uses this mechanism and optimizes parameters for ophthalmology scenarios, overcoming the problems of uneven expert load, performance crashes caused by expert overload, and insufficient stability in multi-task training in existing technologies.

[0150] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0151] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual conversion system for auxiliary diagnosis of eye diseases, characterized in that, The system includes: The input processing module is used to receive and preprocess input ophthalmic images; The visual transformer backbone is used to process prepared input ophthalmological images and generate a set of image labels; Hybrid expert layers are used to receive image labels generated by the visual transformer backbone and selectively activate expert neural networks; At least one decoder head, configured to perform a specific ophthalmic diagnostic task using the output of a selectively activated expert neural network in a hybrid expert layer; The output generation module processes the results from the decoder header of a specific task and generates the final diagnostic output.

2. The visual conversion system for auxiliary diagnosis of eye diseases as described in claim 1, characterized in that, The input processing module includes: Image block extractor: The image block extractor is used to segment an input ophthalmology image into multiple blocks; The block embedding module is used to convert 16x16 image blocks into 768-dimensional embeddings.

3. The visual conversion system for auxiliary diagnosis of eye diseases as described in claim 1, characterized in that, The visual transformer backbone includes a multi-head self-attention network and a feedforward network.

4. The visual conversion system for auxiliary diagnosis of eye diseases as described in claim 3, characterized in that, The hybrid expert layer includes: An expert pool, which contains multiple expert neural networks; The router's expert pool contains 16 expert neural networks.

5. The visual conversion system for auxiliary diagnosis of eye diseases as described in claim 3, characterized in that, The decoder head includes a disease classification head, a prognosis head, and a biomarker detection head.

6. The visual conversion system for auxiliary diagnosis of eye diseases as described in claim 1, characterized in that, At least one router is configured to activate the first K subsets of expert neural networks, where K is less than the total number of expert neural networks in the expert pool.

7. A visual conversion method for auxiliary diagnosis of eye diseases, characterized in that, The method includes: S1, Receive input ophthalmology image. It should be noted that after receiving the input ophthalmology image, the input processing module performs initial preprocessing on the received image. S2, using the visual transformer backbone to generate a set of image labels from the input ophthalmological image, wherein when the visual transformer backbone processes the input image to generate a set of image labels, the image labels represent key features of the ophthalmological image; S3, where at least one router selectively activates a subset of the expert neural network in the hybrid expert layer based on image labels; S4, which performs a specific eye disease diagnosis task by using the output of a selectively activated expert neural network from at least one task-specific decoder head.

8. The visual conversion method for auxiliary diagnosis of eye diseases as described in claim 7, characterized in that, The method for receiving input ophthalmic images includes: S101, Load the input ophthalmology image; S102, Perform block embedding. The image block extractor segments the input image into blocks, and the block embedding module converts the segmented blocks into high-dimensional embeddings. S103, the visual transformer backbone processes the embedding blocks, with multi-head self-attention and feedforward network components working together to convert the embedding blocks into a set of image labels; S104, By routing image tags through a hybrid expert layer, the router selectively activates a subset of expert neural networks from the expert pool based on the features of the image tags.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the visual conversion method for auxiliary diagnosis of eye diseases as described in claim 7 or 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the visual conversion method for auxiliary diagnosis of eye diseases as described in claim 7 or 8.

Images