A retinal image classification method and system based on multi-modal incremental learning

Abstract

The present application relates to a kind of retinal image classification method and system based on multi-modal incremental learning, method includes: S1, obtains multi-modal retinal image, and multi-modal retinal image is preprocessed and encoded processing, obtains global semantic feature vector;S2, global semantic feature vector is mapped to the embedding space of large language model, and obtains visual embedding feature;S3, according to text prompt word, visual embedding feature and global semantic feature vector, constructs multi-modal prompt sequence;S4, current low rank matrix parameter LoRA is introduced into large language model, and the original model main body parameter is frozen, according to multi-modal prompt sequence, current low rank matrix parameter LoRA is trained, repeats S4 until completing target stage training, obtains the large language model after training;S5, the retinal image data to be classified is obtained, and input is classified into the large language model after training, and the prediction result of output subretinal hydrops class and vision class.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical image analysis technology, and in particular to a retinal image classification method and system based on multimodal incremental learning. Background Technology

[0002] The absorption of retinal fluid and the trend of visual acuity changes are key indicators for assessing the progression and treatment effectiveness of retinal-related diseases (such as age-related macular degeneration and diabetic macular edema (DME)). This process is highly dependent on the comprehensive analysis of medical images. In actual clinical practice, doctors typically utilize multiple imaging modalities, including optical coherence tomography (OCT), fundus photography, and non-vasculature layer blood flow imaging (NVL), and combine these with pre- and post-treatment image changes to assess the dynamic changes in intraretinal fluid, the regression of choroidal neovascularization, and the trend of visual function recovery.

[0003] However, this analysis process relies on human experience, which is not only time-consuming and labor-intensive, but also highly subjective, greatly influenced by factors such as the physician's individual judgment and experience level. This is especially true when assessing minute structural changes or fluid residue, where it suffers from low accuracy and poor repeatability. Furthermore, manual analysis struggles to simultaneously process large amounts of multimodal and multi-time-point imaging data, limiting its efficiency and stability in complex clinical settings.

[0004] In recent years, deep learning technology has made some progress in the field of automatic medical image analysis. However, in the task of retinal prognosis analysis for multimodal and time-series models, existing methods still face the following technical challenges: (1) Imbalanced distribution of modal data: In practical applications, there are significant differences in the number of multimodal medical images collected, especially OCT images, which are significantly more numerous than Fundus or NVL modalities. This leads to the model being prone to overfitting to the dominant modality during training, inhibiting the effective modeling and utilization of other auxiliary modalities. (2) Significant differences in visual representation between modalities: Different imaging modalities have natural differences in image texture, structural hierarchy, spatial resolution, etc., making it difficult for the model to learn high-quality cross-modal shared representations in a unified space. This results in poor multimodal fusion and limits the model's predictive ability. (3) Lack of flexible scalability of existing models: Most existing multimodal analysis frameworks adopt a fixed architecture, which is difficult to adapt to the real needs of gradually accessing modal information in clinical scenarios. Once a new modality is added or the task objective is adjusted, the model structure often needs to be significantly modified and retrained, which seriously restricts the deployability and application capabilities of the model.

[0005] Therefore, there is an urgent need to propose a retinal image classification method and system based on multimodal incremental learning, so as to achieve accurate analysis of multimodal and multitemporal retinal image data. Summary of the Invention

[0006] The purpose of this invention is to provide a retinal image classification method and system based on multimodal incremental learning, which aims to solve the problems of modal data imbalance, strong visual heterogeneity and poor model expansion capability that are common in current multimodal image analysis.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A retinal image classification method based on multimodal incremental learning includes:

[0009] S1. Acquire multimodal retinal images, perform preprocessing and encoding on the multimodal retinal images, and obtain global semantic feature vectors;

[0010] S2. Map the global semantic feature vector to the embedding space of the large language model to obtain visual embedding features;

[0011] S3. Construct a multimodal prompt sequence based on the text prompt words, the visual embedding features, and the global semantic feature vector;

[0012] S4. Introduce the current low-rank matrix parameter LoRA into the large language model and freeze the original main parameters of the large language model. Train the current low-rank matrix parameter LoRA according to the multimodal cue sequence. Repeat S4 until the target stage training is completed and obtain the trained large language model.

[0013] S5. Obtain the retinal image data to be classified, input it into the trained large language model for classification, and output the prediction results of subretinal fluid category and visual acuity category.

[0014] Optionally, preprocessing the multimodal retinal images includes: performing size normalization and random data augmentation on the multimodal retinal images.

[0015] Optionally, encoding the multimodal retinal image includes: inputting the preprocessed multimodal retinal image into a visual encoder to extract a token for classification and obtain the global semantic feature vector.

[0016] Optionally, obtaining the text prompts includes: constructing Chinese prompts with semantic identifiers based on the current training stage and the modality type of the retinal image.

[0017] Optionally, training the current low-rank matrix parameters LoRA based on the multimodal cue sequence includes:

[0018] The first low-rank matrix parameter LoRA1 is introduced into the large language model, the original model parameters of the large language model are frozen, and the first low-rank matrix parameter LoRA1 is trained according to the multimodal cue sequence corresponding to the first modality retinal image to obtain the large language model after the first stage of training. The multimodal cue sequence corresponding to the first modality retinal image includes the text cue words, visual embedding features and global semantic feature vectors corresponding to the first modality retinal image.

[0019] The second low-rank matrix parameter LoRA2 is introduced into the large language model after the first stage of training and the original model parameters are frozen. The second low-rank matrix parameter LoRA2 is trained according to the multimodal cue sequence corresponding to the second modality retinal image to obtain the large language model after the second stage of training. The multimodal cue sequence corresponding to the second modality retinal image includes the text cue words, visual embedding features, global semantic feature vector, and visual embedding features corresponding to the first modality retinal image.

[0020] The third low-rank matrix parameter LoRA3 is introduced into the large language model after the second stage of training, and the original model parameters are frozen. The third low-rank matrix parameter LoRA3 is trained according to the multimodal cue sequence corresponding to the third modality retinal image to obtain the large language model after the third stage of training. The multimodal cue sequence corresponding to the third modality retinal image includes the text cue words, visual embedding features, global semantic feature vector, visual embedding features corresponding to the first modality retinal image, and visual embedding features corresponding to the second modality retinal image.

[0021] Optionally, the output of each training stage includes: obtaining the global classification label vector in the last hidden state of the large model at each stage, and inputting the global classification label vector into the classification head for prediction;

[0022] p s =arg max(softmax(Cls1·θ1)),p v =sigmoid(Cls2·θ2),

[0023] Where θ1 and θ2 represent the learnable weight parameters of the two classification heads, respectively, and p s For the multi-class classification results of subretinal fluid absorption status, p v The binary classification prediction results for visual function recovery status are shown. Cls1 represents the token associated with subretinal fluid accumulation, and Cls2 represents the token associated with the patient's visual function recovery status.

[0024] Optionally, the loss functions for training at each stage include: multi-class cross-entropy loss for the liquid absorption state and binary cross-entropy loss for the visual function recovery state.

[0025] This invention also provides a retinal image classification system based on multimodal incremental learning, comprising:

[0026] A medical image preprocessing module is used to acquire multimodal retinal images and preprocess the multimodal retinal images;

[0027] The visual encoding module is used to encode the preprocessed multimodal retinal images to obtain global semantic feature vectors.

[0028] The phase-aware prompt construction module is used to construct text prompt words based on the current training phase and the modality type of the retinal image;

[0029] A multimodal cue sequence construction module is used to construct a multimodal cue sequence based on text cue words, the visual embedding features, and the global semantic feature vector;

[0030] The modal incremental adaptation module is used to introduce the current low-rank matrix parameter LoRA into the large language model and freeze the original model body parameters of the large language model.

[0031] The temporal pathology modeling module is used to train the current low-rank matrix parameter LoRA based on the multimodal cue sequence until the target stage training is completed, and to obtain the trained large language model.

[0032] The beneficial effects of this invention are as follows:

[0033] Supports a flexible and efficient modal incremental learning mechanism: This invention can progressively introduce and integrate multiple ophthalmic imaging modalities (such as OCT, fundus color photography, non-vascular layer blood flow imaging, etc.). Through the dynamic expansion and weight update of the low-rank knowledge matrix, it achieves seamless collaborative learning among multiple modalities, significantly reducing the model performance degradation caused by modality loss, modality heterogeneity, or uneven sample distribution, and improving the robustness and clinical adaptability of the model.

[0034] This method possesses the capability for stage-based joint modeling based on pre- and post-treatment images: It can fully capture the structural and functional changes in patients' conditions at different treatment stages, meeting the clinical needs for dynamic disease monitoring and efficacy evaluation. The multi-stage modeling mechanism can uncover potential change patterns and related features across time points, providing data support for the accurate identification of disease evolution processes and a scientific basis for individualized treatment decisions, while also improving long-term follow-up and prognostic prediction capabilities.

[0035] This method enables collaborative analysis and intelligent decision support of multimodal and multi-stage medical information: By integrating multimodal image data with task-specific information, it outputs accurate multidimensional diagnostic analysis results, effectively improving clinicians' efficiency in identifying and making decisions regarding age-related macular degeneration. Furthermore, this method possesses significant decision support capabilities, not only helping to optimize treatment strategies and improve patient satisfaction, but also playing a crucial role in promoting the development of intelligent ophthalmological diagnosis and treatment, optimizing the allocation of medical resources, and improving diagnostic and treatment efficiency, providing key technical support for the digital and intelligent upgrading of ophthalmological medical services. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a framework diagram of a retinal image classification method based on multimodal incremental learning according to an embodiment of the present invention;

[0038] Figure 2 These are the prediction results for two samples at different stages in an embodiment of the present invention;

[0039] Figure 3 This is a flowchart of a retinal image classification method based on multimodal incremental learning, according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example 1:

[0043] like Figure 3 As shown, this embodiment provides a retinal image classification method based on multimodal incremental learning, including:

[0044] S1. Acquire multimodal retinal images, perform preprocessing and encoding on the multimodal retinal images, and obtain global semantic feature vectors;

[0045] S2. Map the global semantic feature vector to the embedding space of the large language model to obtain visual embedding features;

[0046] S3. Construct a multimodal prompt sequence based on text prompt words, visual embedding features, and global semantic feature vectors;

[0047] S4. Introduce the current low-rank matrix parameter LoRA into the large language model and freeze the original main parameters of the large language model. Train the current low-rank matrix parameter LoRA according to the multimodal cue sequence. Repeat S4 until the target stage training is completed and the trained large language model is obtained.

[0048] S5. Obtain the retinal image data to be classified, input it into the trained large language model for classification, and output the prediction results of subretinal fluid category and visual acuity category.

[0049] Specifically, this method dynamically introduces and integrates information from new modalities while maintaining the original model structure and parameters, avoiding redundant training and improving model scalability and transferability. Introducing the low-rank matrix parameter LoRA into the large language model enables the model to learn the structural and pathological change characteristics of that modality, enhancing its ability to model the progression of retinal diseases. Through comprehensive analysis of the aforementioned cue sequences using the large language model, the representation of task-specific tokens is extracted for downstream prediction tasks and passed to the corresponding classification head to complete the identification and prediction of retinal structural changes and visual acuity changes.

[0050] Further preprocessing of multimodal retinal images includes: size normalization and random data augmentation.

[0051] Specifically, pre-treatment and post-treatment images are paired and input, and then uniformly cropped to a size of 224×224 pixels. Data augmentation is performed on the images with a probability of 0.5, and then global image features are extracted using a visual encoder.

[0052] Furthermore, the encoding process for the multimodal retinal images includes: inputting the preprocessed multimodal retinal images into a visual encoder to extract tokens for classification and obtain global semantic feature vectors.

[0053] Furthermore, obtaining text prompts includes: constructing semantically labeled Chinese prompts based on the current training stage and the modality type of the retinal image.

[0054] Furthermore, training the LoRA parameter of the current low-rank matrix based on the multimodal cue sequence includes:

[0055] The first low-rank matrix parameter LoRA1 is introduced into the large language model, the original parameters of the large language model are frozen, and the first low-rank matrix parameter LoRA1 is trained according to the multimodal cue sequence corresponding to the first modality retinal image to obtain the large language model after the first stage of training. The multimodal cue sequence corresponding to the first modality retinal image includes the text cue words, visual embedding features and global semantic feature vectors corresponding to the first modality retinal image.

[0056] The second low-rank matrix parameter LoRA2 is introduced into the large language model after the first stage of training and the original model parameters are frozen. The second low-rank matrix parameter LoRA2 is trained according to the multimodal cue sequence corresponding to the second modality retinal image to obtain the large language model after the second stage of training. The multimodal cue sequence corresponding to the second modality retinal image includes the text cue words, visual embedding features, global semantic feature vector, and visual embedding features corresponding to the first modality retinal image.

[0057] The third low-rank matrix parameter LoRA3 is introduced into the large language model after the second stage of training, and the original model parameters are frozen. The third low-rank matrix parameter LoRA3 is trained according to the multimodal cue sequence corresponding to the third modality retinal image to obtain the large language model after the third stage of training. The multimodal cue sequence corresponding to the third modality retinal image includes the text cue words, visual embedding features, global semantic feature vector, visual embedding features corresponding to the first modality retinal image, and visual embedding features corresponding to the second modality retinal image.

[0058] Specifically, this method employs a multi-stage modal incremental learning strategy, gradually introducing new modal information to improve the model's diagnostic performance. The specific process is as follows: In the first stage, 60 training epochs are performed on samples containing only the OCT modality to learn basic retinal structure perception capabilities; in the second stage, the LoRA parameters obtained in the first stage are frozen, a new low-rank adaptation module is introduced, and the model is trained for 50 epochs on samples containing both the OCT and Fundus modalities to fully utilize the complementary information between different modalities; in the third stage, the model is further trained for 40 epochs on samples containing all three modalities—OCT, Fundus, and NVL—to improve the model's performance and robustness in retinal effusion change detection and vision recovery prediction tasks.

[0059] Furthermore, the output of each training stage includes: obtaining the global classification label vector in the last hidden state of the large model at each stage, and inputting the global classification label vector into the classification head for prediction;

[0060] p s =arg max(softmax(Cls1·θ1)),p v=sigmoid(Cls2·θ2),

[0061] Where θ1 and θ2 represent the learnable weight parameters of the two classification heads, respectively, and p s For the multi-class classification results of subretinal fluid absorption status, p v The binary classification prediction results for visual function recovery status are shown. Cls1 represents the token associated with subretinal fluid accumulation, and Cls2 represents the token associated with the patient's visual function recovery status.

[0062] Furthermore, the loss functions for each training stage include: multi-class cross-entropy loss for the liquid absorption state and binary cross-entropy loss for the visual function recovery state.

[0063] The method of this embodiment will be further explained below in conjunction with optical coherence tomography (OCT), fundus imaging (Fundus), and nonvascular flow imaging (NVL):

[0064] A retinal image classification method based on multimodal incremental learning is proposed, primarily applied to the prognostic prediction of age-related macular degeneration. The method includes the following steps:

[0065] First, the model receives multimodal retinal image data before and after treatment, including but not limited to optical coherence tomography (OCT), fundus photography (Fundus), and nonvascular flow imaging (NVL). After uniform resizing and synchronous data augmentation, each modal image is input into the visual coding module to extract its global semantic feature vector.

[0066] Subsequently, the model introduces different modalities into stages, generating corresponding Chinese prompt words through a stage-aware prompting module, and constructing a multimodal prompt sequence by combining image features and task-specific markers. This sequence is injected into the large language model, where a modality incremental adaptation module is used to introduce adaptation parameters for the new modality in the form of a low-rank matrix, achieving non-destructive knowledge transfer and incremental learning, and avoiding forgetting of already learned modalities.

[0067] This embodiment employs a phased training strategy: the first phase uses only the OCT modality to model the basic structure of the model; the second phase introduces fundus color imaging to enhance the model's understanding of texture and vascular distribution features; the third phase further introduces non-vascular layer blood flow imaging to achieve modal complementarity and fusion, improving the ability to perceive pathological features and predict accuracy. In each phase, the method freezes the previously trained low-rank parameters and only updates the adaptation modules corresponding to the new modality to maintain model stability and learning efficiency.

[0068] This method has good scalability and versatility, enabling efficient integration between different medical modalities and significantly improving the joint predictive ability of retinal fluid absorption status and visual function recovery status, providing scientific decision support for clinicians and promoting the development of intelligent ophthalmic diagnosis and treatment systems.

[0069] The following combination Figure 1 The method of this embodiment will be described as follows:

[0070] In step 1, image preprocessing is performed using the Python programming language. Specifically, the `resize` function from the OpenCV library (cv2) in Python is used to normalize the input images, uniformly adjusting them to a pixel size of 224×224. To enhance the model's generalization ability, a random data augmentation strategy is further introduced: First, a floating-point number is randomly generated in the interval [0,1]. When this value is greater than 0.5, an augmentation method (such as random rotation, pixel shift, scaling, etc.) is randomly selected from a preset set of data augmentation methods and applied synchronously to paired images before and after treatment to ensure temporal consistency.

[0071] In step 1.1, to extract high-quality visual features from the image, this method employs a visual encoding model pre-trained on large-scale medical image-text alignment data—BiomedCLIP (a variant based on Vision Transformer). Since the BiomedCLIP model possesses excellent semantic alignment capabilities between medical images and text, its parameters are fixed (weights frozen) in this method, serving as the core of the visual encoder module. The output of the visual encoder cannot be input into a large language model; therefore, an adapter is added to transform and adapt the features to the large model. During the encoding process, the input medical image is first segmented into several visual tokens, and feature modeling is performed using a Transformer structure. The model specifically extracts a special token used for classification in the ViT model as a global visual feature representation of the image, used for subsequent fusion modeling with prompt words. The formula for this part is as follows:

[0072] v pre,1 =BiomedCLIP(I pre,1 ),v post,1 =BiomedCLIP(I post,1 ),

[0073] Among them, I pre,1 and I post,1 These are pre-treatment and post-treatment images from the OCT modality, v pre,1 ,v post,1 ∈R768 is the global feature extracted by BiomedCLIP.

[0074] In step 2, the visual features before and after treatment are projected onto the embedding space of the large language model through a mapper, as follows:

[0075]

[0076] Among them, W v ∈R 4096×768 It is a mapping matrix, b v This is a bias. Then, based on this stage, the corresponding text prompt words are selected, and the text prompt words, visual embeddings, and task-specific tokens (i.e., the global semantic feature vectors Cls′1 and Cls′2) are concatenated to obtain a multimodal prompt sequence. Inject it into a large language for modeling.

[0077] In step 3, to achieve efficient adaptation of the large language model to medical modalities, this method employs a Parameter Efficient Fine-Tuning (PEFT) strategy combined with the Low-Rank Adaptation (LoRA) mechanism to achieve non-destructive incremental modality learning. Specifically, the model uses the LLaMA3-8B large language model as its backbone, keeping the main parameters of the original model frozen, and incrementally inserting independent LoRA low-rank matrices according to the stages of modality introduction. In each stage, only the LoRA parameters corresponding to the current stage are trained, while the remaining parameters remain unchanged, thereby effectively avoiding catastrophic forgetting while achieving modality expansion. This strategy has a good modular structure and scalability, which can significantly improve the transfer efficiency and diagnostic accuracy of the model in ophthalmic multimodal tasks, meeting the actual needs of multimodal fusion and continuous knowledge learning in clinical environments. This step inserts LoRA1 to achieve this part of knowledge learning.

[0078] In step 4, the large language model performs comprehensive semantic analysis on the injected multimodal cue sequences and outputs hidden layer features. In this embodiment, two global classification label vectors, denoted as Cls1 and Cls2, are extracted from the last hidden state of the large language model to carry high-level semantic features related to retinal pathological states. Cls1 represents a token related to subretinal effusion, used to predict its changing trend (task categories: "no absorption," "partial absorption," and "complete absorption"). Cls2 represents a token related to the patient's visual function recovery, used for binary classification prediction ("improvement" and "no improvement"). Subsequently, these two token features carrying pathological information are input into two independent classification heads to perform prediction tasks, such as... Figure 2 As shown. This process can be formally represented as follows:

[0079] ps =arg max(softmax(Cls1·θ1)),p v =sigmoid(Cls2·θ2),

[0080] Where θ1 and θ2 represent the learnable weight parameters of the two classification heads, respectively, and p s For the multi-class classification results of subretinal fluid absorption status, p v The result is a binary classification prediction of the visual function recovery state.

[0081] According to the implementation steps, in order to implement the incremental learning strategy of the model, the model has been trained on the primary modality (i.e., OCT) to learn key pathological features of the retina. In the second stage, to enhance the model's understanding of fundus images, a modality-aware cue word construction module is used to obtain the text cue word T2 for the second stage. Then, a multimodal cue word sequence is constructed, while the knowledge learned by LoRA1 is frozen, and a low-rank matrix LoRA2 is added to learn fundus image knowledge.

[0082]

[0083] in, and The Fundus modality is embedded in the model; this sequence is input into a large language model, and the contextual dependencies between multimodalities are analyzed to achieve higher accuracy in prognostic judgment. Building upon the OCT and Fundus knowledge learning, the third stage further introduces the non-vascular flow imaging (NVL) modality. Using the same strategy, a new adaptation module is constructed for NVL, and training continues based on the model parameters from the first two stages. In this stage, the model maintains its inheritance of OCT and Fundus knowledge while learning supplementary information from NVL features in retinal pathology analysis, further enhancing its ability to handle complex multimodal inputs. The multimodal cue sequence is further expanded to achieve trimodal synergistic fusion. The multimodal cue word sequence for this stage is as follows:

[0084]

[0085] In each training phase, the model employs two classification loss functions to jointly optimize model performance, and dynamically adjusts the weights of the loss functions according to task requirements to improve the final prediction results. The specific definitions of the loss functions are as follows:

[0086]

[0087] Where L1 represents tri-class classification loss (e.g., predicting the degree of liquid absorption), L2 represents binary classification loss (e.g., judging visual recovery status), and y i,c and pi,c Let be the true label and the predicted probability of the i-th sample, respectively, and N be the batch size. The total optimization loss is:

[0088]

[0089] Here, a and b are adjustable loss weights. In the experiment, it was found that a=1 and b=2 made the model achieve the best diagnostic performance.

[0090] Based on the above design, the proposed model was evaluated against several mainstream comparative models, including traditional natural image classification models (such as GoogLeNet and ResNet) and typical modality fusion methods (such as MM-MIL and MM-CNN). The performance metrics of each model in this task are shown in Table 1, where bold text indicates the best result in that column.

[0091] Table 1

[0092]

[0093] In summary, the method proposed in this invention demonstrates significant advantages in multimodal retinal image processing and prognostic prediction tasks. Comparative experiments with various existing methods show that the method of this invention achieves optimal performance across all evaluation metrics, effectively improving the model's joint predictive ability in both retinal effusion absorption and visual function recovery states. The three-stage progressive training strategy not only ensures the full integration of information from different modalities but also enables continuous accumulation and updating of model knowledge, exhibiting good stability, scalability, and clinical adaptability. This method can provide reliable technical support for the construction of intelligent auxiliary diagnostic systems in ophthalmology, possessing high practical value and potential for widespread application.

[0094] Example 2:

[0095] A retinal image classification system based on multimodal incremental learning includes:

[0096] The medical image preprocessing module is used to acquire multimodal retinal images and preprocess them.

[0097] Specifically, the medical image preprocessing module is used to receive retinal image data before and after treatment, and perform preprocessing to obtain standard format data that meets the model input requirements. The image data comes from multiple imaging modalities, including but not limited to optical coherence tomography (OCT), fundus images (Fundus), and nonvascular flow imaging (NVL).

[0098] In this embodiment, all modal images are uniformly adjusted to a size of 224×224 pixels, and the same data augmentation operation is applied to images of the same modality with a probability of 0.5. The augmentation methods include random rotation, pixel translation, and scaling.

[0099] The visual encoding module is used to encode the preprocessed multimodal retinal images to obtain global semantic feature vectors.

[0100] Specifically, the visual encoding module is used to encode the temporal images of each modality into global semantic feature vectors.

[0101] In this embodiment, the features extracted by the visual encoding module include semantic features of pre-treatment and post-treatment images, forming two-dimensional feature pairs to enhance the ability to model structural pathological changes.

[0102] The phase-aware prompt construction module is used to construct text prompt words based on the current training phase and the modality type of the retinal image;

[0103] Specifically, the stage-aware prompt construction module constructs Chinese prompt words with semantic identifiers based on the training stage and modality type of the input image.

[0104] A multimodal cue sequence construction module is used to construct multimodal cue sequences based on text cue words, visual embedding features, and global semantic feature vectors;

[0105] Specifically, the multimodal cue sequence construction module concatenates Chinese cue words, visual features, and task-related special tokens into a cue sequence to support subsequent diagnostic tasks;

[0106] In this embodiment, the multimodal cue sequence construction module concatenates the stage cue words, visual modal features, and task tokens into a complete sequence, which is then input into the large language model to complete the diagnostic task.

[0107] The modal incremental adaptation module is used to introduce the current low-rank matrix parameters LoRA into the large language model and freeze the original main parameters of the large language model.

[0108] Specifically, the modal incremental adaptation module injects new modalities into training by introducing low-rank matrix parameters, keeping the original model parameters frozen, thus achieving non-destructive incremental learning and effectively avoiding catastrophic forgetting.

[0109] In this embodiment, the modal incremental adaptation module adjusts the weight parameters in the large language model by injecting a low-rank matrix and adopts a phased freezing strategy to maintain the original knowledge and prevent catastrophic forgetting, thereby achieving efficient modal transfer.

[0110] The modal incremental adaptation module supports a phased training process, including: an initial single-modal training phase, a dual-modal fusion training phase, and a multi-modal incremental training phase. Each phase introduces specific cue sequences and low-rank matrices to gradually achieve effective integration of new modal knowledge. The specific training methods include:

[0111] (1) Pair the retinal images before and after treatment, unify the image size and apply synchronous data enhancement operation;

[0112] (2) Use the visual coding module to extract the image features of each modality and form a feature set before and after treatment;

[0113] (3) Generate corresponding Chinese prompts through the phase-aware prompt construction module to identify the modality type and training phase;

[0114] (4) The new modalities are injected into the large language model in the form of a low-rank matrix through the modal increment adaptation module, and the training is completed while the original parameters are frozen.

[0115] (5) The extracted visual features are concatenated with the prompt sequence and then input into the large language model, and the prognosis is predicted by the task token;

[0116] (6) The prediction of liquid absorption state and visual function recovery state is optimized by using the joint cross-entropy loss function, and the weight coefficients of the two losses are dynamically adjusted.

[0117] The temporal pathology modeling module is used to train the current low-rank matrix parameters LoRA based on the multimodal cue sequence until the target stage training is completed, and to obtain the trained large language model.

[0118] Specifically, the temporal pathology modeling module inputs the constructed multimodal cue sequences into the large language model to complete the joint prediction task of retinal fluid absorption status (multi-classification) and visual function recovery status (binary classification).

[0119] The final prediction loss function consists of two parts: the multi-class cross-entropy loss for the liquid absorption state and the binary cross-entropy loss for the visual function recovery state. The weight coefficients of each part can be set according to the task requirements for joint optimization.

[0120] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A retinal image classification method based on multimodal incremental learning, characterized in that, include: S1. Acquire multimodal retinal images, perform preprocessing and encoding on the multimodal retinal images, and obtain global semantic feature vectors; S2. Map the global semantic feature vector to the embedding space of the large language model to obtain visual embedding features; S3. Construct a multimodal prompt sequence based on the text prompt words, the visual embedding features, and the global semantic feature vector; S4. Introduce the current low-rank matrix parameter LoRA into the large language model and freeze the original main parameters of the large language model. Train the current low-rank matrix parameter LoRA according to the multimodal cue sequence. Repeat S4 until the target stage training is completed and obtain the trained large language model. Training the LoRA parameter of the current low-rank matrix based on the multimodal cue sequence includes: The first low-rank matrix parameter LoRA1 is introduced into the large language model, the original model parameters of the large language model are frozen, and the first low-rank matrix parameter LoRA1 is trained according to the multimodal cue sequence corresponding to the first modality retinal image to obtain the large language model after the first stage of training. The multimodal cue sequence corresponding to the first modality retinal image includes the text cue words, visual embedding features and global semantic feature vectors corresponding to the first modality retinal image. The second low-rank matrix parameter LoRA2 is introduced into the large language model after the first stage of training and the original model parameters are frozen. The second low-rank matrix parameter LoRA2 is trained according to the multimodal cue sequence corresponding to the second modality retinal image to obtain the large language model after the second stage of training. The multimodal cue sequence corresponding to the second modality retinal image includes the text cue words, visual embedding features, global semantic feature vector, and visual embedding features corresponding to the first modality retinal image. The third low-rank matrix parameter LoRA3 is introduced into the large language model after the second stage of training, and the original model parameters are frozen. The third low-rank matrix parameter LoRA3 is trained according to the multimodal cue sequence corresponding to the third modality retinal image to obtain the large language model after the third stage of training. The multimodal cue sequence corresponding to the third modality retinal image includes text cue words, visual embedding features, global semantic feature vector, visual embedding features corresponding to the first modality retinal image, and visual embedding features corresponding to the second modality retinal image. S5. Obtain the retinal image data to be classified, input it into the trained large language model for classification, and output the prediction results of subretinal fluid category and visual acuity category.

2. The retinal image classification method based on multimodal incremental learning according to claim 1, characterized in that, Preprocessing of the multimodal retinal images includes: size normalization and random data augmentation of the multimodal retinal images.

3. The retinal image classification method based on multimodal incremental learning according to claim 2, characterized in that, Encoding the multimodal retinal image includes: inputting the preprocessed multimodal retinal image into a visual encoder to extract a token for classification and obtaining the global semantic feature vector.

4. The retinal image classification method based on multimodal incremental learning according to claim 1, characterized in that, Obtaining the text prompts includes: constructing Chinese prompts with semantic identifiers based on the current training stage and the modality type of the retinal image.

5. The retinal image classification method based on multimodal incremental learning according to claim 1, characterized in that, The output of each training stage includes: obtaining the global classification label vector in the last hidden state of the large model at each stage, and inputting the global classification label vector into the classification head for prediction; ， Where θ1 and θ2 represent the learnable weight parameters of the two classification heads, respectively, and p s For the multi-class classification results of subretinal fluid absorption status, p v The binary classification prediction results for visual function recovery status are shown. Cls1 represents the token associated with subretinal fluid accumulation, and Cls2 represents the token associated with the patient's visual function recovery status.

6. The retinal image classification method based on multimodal incremental learning according to claim 1, characterized in that, The loss functions for each training stage include: multi-class cross-entropy loss for the liquid absorption state and binary cross-entropy loss for the visual function recovery state.

7. A retinal image classification system based on multimodal incremental learning implemented by the method according to any one of claims 1-6, characterized in that, include: A medical image preprocessing module is used to acquire multimodal retinal images and preprocess the multimodal retinal images; The visual encoding module is used to encode the preprocessed multimodal retinal images to obtain global semantic feature vectors. The phase-aware prompt construction module is used to construct text prompt words based on the current training phase and the modality type of the retinal image; A multimodal cue sequence construction module is used to construct a multimodal cue sequence based on text cue words, the visual embedding features, and the global semantic feature vector; The modal incremental adaptation module is used to introduce the current low-rank matrix parameter LoRA into the large language model and freeze the original model body parameters of the large language model. The temporal pathology modeling module is used to train the current low-rank matrix parameter LoRA based on the multimodal cue sequence until the target stage training is completed, and to obtain the trained large language model.

Images