Eye fundus image classification method and system based on SLIP pre-training and frequency domain information fusion, terminal and storage medium

Abstract

The invention discloses a fundus image classification method and system based on SLIP pre-training and frequency domain information fusion, a terminal and a storage medium, and the method comprises the steps: constructing a fundus image classification model which comprises an SLIP pre-training module and a hidden space feature extraction module; collecting pre-training data, and inputting the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model; and obtaining a to-be-processed fundus image, inputting the to-be-processed fundus image into the target classification model for multi-modal data alignment and hidden space feature extraction, and outputting a final feature extraction result through a classifier to obtain a classification result of the to-be-processed fundus image. According to the method, the SLIP framework is used for image-text contrast learning and image self-supervised learning, frequency domain and spatial domain feature extraction is adopted for the image data and the text data in the hidden space to capture high-level semantic information in the image data and the text data, and the eye fundus image classification precision is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, terminal, and computer-readable storage medium for classifying fundus images based on SLIP pre-training and frequency domain information fusion.

[0002] Background Technology In the field of image analysis targeting specific subjects, the inherent characteristics of the analyzed object and the imaging conditions together constitute the core challenges. Taking infant eye images as an example, the difficulty of analysis stems first from the physiological and behavioral characteristics of the subjects. The imaging process is easily affected by factors such as the subject's inability to actively cooperate and physiological movements, resulting in original images often accompanied by quality defects such as motion blur, inaccurate focus, and uneven exposure, causing key details to be obscured by noise and artifacts. Secondly, to comprehensively assess the condition, clinical practice often requires the collection of data from multiple imaging modalities, such as color fundus photographs and angiography sequences. These images from different modalities differ significantly in information density, feature distribution, and noise patterns, but they are highly correlated at the semantic level, jointly describing the same anatomical structure and pathophysiological changes. How to effectively and collaboratively utilize this multi-source, heterogeneous image data is key to improving analytical accuracy.

[0003] Existing technologies have significant limitations when handling complex multimodal image analysis tasks. Mainstream methods typically rely on single-encoder architectures pre-trained on large natural image datasets for feature extraction. These general-purpose visual encoders are insufficient in representing complex degradation patterns and subtle feature variations present in domain-specific images, resulting in low-level features extracted from low-quality or feature-complex images lacking robustness and discriminative power. Furthermore, most methods employ simple post-fusion or contrastive learning strategies based on a single projection layer for multimodal alignment and fusion. This approach ignores the inherent differences in data distribution and information characteristics between different imaging modalities, forcing them to be mapped into a unified feature space, which easily leads to information compression or loss, especially of subtle features sensitive to specific modalities. This coarse fusion method makes it difficult for the model to fully mine and utilize the complementary information between multimodal data, thus limiting the performance ceiling and generalization ability of the final image analysis task (such as classification and segmentation).

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a fundus image classification method, system, terminal, and storage medium based on SLIP pre-training and frequency domain information fusion. This invention aims to address the problem in existing technologies that, when processing fundus images, ignore the inherent differences between modalities and commonly use a single encoder or simple contrastive learning for modal alignment, resulting in insufficient representation of some modal information and thus low accuracy in fundus image classification.

[0006] To achieve the above objectives, this invention provides a fundus image classification method based on SLIP pre-training and frequency domain information fusion. The fundus image classification method based on SLIP pre-training and frequency domain information fusion includes the following steps: A fundus image classification model is constructed, which includes a SLIP pre-training module and a latent space feature extraction module. Collect pre-training data and input the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model; A fundus image to be processed is acquired, and the fundus image to be processed is input into the target classification model for multimodal data alignment and latent space feature extraction. The final feature extraction result is output through a classifier to obtain the classification result of the fundus image to be processed.

[0007] Optionally, the fundus image classification method based on SLIP pre-training and frequency domain information fusion, wherein the SLIP pre-training module includes a shared image encoder and a text encoder; The shared image encoder includes a multi-head self-attention layer and a feedforward neural network layer. The feedforward neural network layer introduces a modality-specific LoRA adapter to enable images of different modalities to obtain independent low-rank incremental updates in the shared backbone.

[0008] Optionally, the fundus image classification method based on SLIP pre-training and frequency domain information fusion, wherein the step of collecting pre-training data and inputting the pre-training data into the SLIP pre-training module for pre-training to obtain the target classification model specifically includes: A set of fundus images is collected for pre-training, and fundus color photography and fluorescein fundus angiography corresponding to each fundus image are obtained from the fundus image set; Each of the fundus color images is subjected to random enhancement processing to obtain multiple first CFP random views and multiple second CFP random views. Each of the fluorescein fundus angiography images is subjected to random enhancement processing to obtain multiple first FFA random views and multiple second FFA random views. All the first CFP random views and the second CFP random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-CFP adapter of the feedforward neural network layer for transformation to obtain multiple first semantic features; All the first FFA random views and the second FFA random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-FFA adapter of the feedforward neural network layer for transformation to obtain multiple second semantic features; The image self-supervised loss is calculated based on all the first semantic features and the second semantic features, and the parameters of the shared image encoder are updated based on the self-supervised loss to complete the training of the shared image encoder. A clinical dataset for pre-training is collected, the clinical dataset is input into the text encoder for encoding, and the encoding result is input into the text projection head for mapping to obtain multiple third semantic features; The image-text contrast loss is calculated based on all the first semantic features, the second semantic features, and the third semantic features. The parameters of the text encoder are then updated based on the image-text contrast loss to complete the training of the text encoder and obtain the target classification model.

[0009] Optionally, in the fundus image classification method based on SLIP pre-training and frequency domain information fusion, the first semantic feature includes a first CFP semantic feature. Second CFP semantic features The second semantic feature includes the first FFT semantic feature. Second FFT semantic features ; The step of calculating the image self-supervised loss based on all the first semantic features and the second semantic features specifically includes: Based on all the first CFP semantic features The second CFP semantic features The first FFT semantic feature and the second FFT semantic features Calculate the image self-supervised loss: ; in, This represents the image self-supervised loss. This indicates the batch size for training the shared image encoder. Indicates the current image number. , Indicates the modal type, when When, i represents the CFP modality type, when When i represents the FFT mode type, Represents an exponential function. Indicates hyperparameters, Indicates first Modal semantic features Indicates the second Modal semantic features.

[0010] Optionally, the fundus image classification method based on SLIP pre-training and frequency domain information fusion, wherein calculating the image-text contrast loss based on all the first semantic features, the second semantic features, and the third semantic features specifically includes: Based on cross-modal alignment constraints, the image-text contrast loss is calculated according to all the first semantic features, the second semantic features, and the third semantic features: ; in, This indicates the batch size used for training the text encoder. This represents the transpose operation, where a and c are ordinal numbers representing semantic features. , This represents the a-th semantic feature. This represents the c-th semantic feature.

[0011] Optionally, the fundus image classification method based on SLIP pre-training and frequency domain information fusion includes a latent space feature extraction module comprising: a frequency domain feature decomposition and fusion unit, a grouped Mamba sequence modeling unit, and a cross-attention unit.

[0012] Optionally, the fundus image classification method based on SLIP pre-training and frequency domain information fusion, wherein the step of inputting the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction specifically includes: The fundus image to be processed is input into the target classification model for multimodal data alignment to obtain the first target semantic features, the second target semantic features, and the third target semantic features; The first target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and CFP amplitude map and CFP phase map are obtained. The second target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and FFA amplitude map and FFA phase map are obtained. The CFP amplitude map and the FFA amplitude map are fused to obtain the fused amplitude, and the CFP phase map and the FFA phase map are fused to obtain the fused phase. The fused amplitude and the fused phase are convolved separately, and the convolution results are fused to obtain fused features. The fused features are then converted back to the spatial domain using inverse Fourier transform to obtain image features. The third target semantic feature is input into the RMSNorm layer of the grouped Mamba sequence modeling unit for regularization. After adding position encoding to the regularization result, it is mapped through a linear layer to obtain the reconstructed feature. The reconstructed features are grouped, and the reconstructed features of each group are input into a parallel S6 state space unit for long-range dependency capture to obtain the structured feature representation corresponding to each group. The structured feature representation corresponding to each group is activated by SiLU, and the activation results of each group are interacted between groups through Group Shuffle to obtain interactive features. Normalization, nonlinear transformation and dimensionality enhancement operations are performed on the interaction features to obtain grouped features. The grouped features are then residually connected with the third target semantic features to obtain text features. The image features and text features are input into the cross-attention unit for cross-modal fusion to obtain the final feature extraction result.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a fundus image classification system based on SLIP pre-training and frequency domain information fusion, wherein the fundus image classification system based on SLIP pre-training and frequency domain information fusion includes: The model building module is used to build a fundus image classification model, which includes a SLIP pre-training module and a latent space feature extraction module. The SLIP pre-training module is used to collect pre-training data, and input the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model. The target image processing module is used to acquire the fundus image to be processed, input the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction, and output the final feature extraction result through the classifier to obtain the classification result of the fundus image to be processed.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a fundus image classification program based on SLIP pre-training and frequency domain information fusion stored in the memory and executable on the processor, wherein when the fundus image classification program based on SLIP pre-training and frequency domain information fusion is executed by the processor, it implements the steps of the fundus image classification method based on SLIP pre-training and frequency domain information fusion as described above.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a fundus image classification program based on SLIP pre-training and frequency domain information fusion, and the fundus image classification program based on SLIP pre-training and frequency domain information fusion, when executed by a processor, implements the steps of the fundus image classification method based on SLIP pre-training and frequency domain information fusion as described above.

[0016] In this invention, a fundus image classification model is constructed, comprising a SLIP pre-training module and a latent space feature extraction module. Pre-training data is collected and input into the SLIP pre-training module for pre-training to obtain a target classification model. A fundus image to be processed is acquired and input into the target classification model for multimodal data alignment and latent space feature extraction. The final feature extraction result is then output through a classifier to obtain the classification result of the fundus image to be processed. This invention utilizes the SLIP framework for image-text contrastive learning and image self-supervised learning, pre-training image and text encoders to align different modalities in a shared latent space. Subsequently, frequency domain and spatial domain features are extracted from the image and text data in the latent space to capture high-level semantic information. Finally, a cross-attention module is used to fuse cross-modal feature information and input it into the classifier for classification, significantly improving the accuracy of fundus image classification. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of the fundus image classification method based on SLIP pre-training and frequency domain information fusion of the present invention; Figure 2 This is an overall architecture diagram of the fundus image classification model in the fundus image classification method based on SLIP pre-training and frequency domain information fusion of the present invention; Figure 3 This is an architecture diagram of the cross-attention unit in the fundus image classification method based on SLIP pre-training and frequency domain information fusion in this invention; Figure 4 This is a structural diagram of a preferred embodiment of the fundus image classification system based on SLIP pre-training and frequency domain information fusion of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0018] This application provides a fundus image classification method, system, and terminal based on SLIP pre-training and frequency domain information fusion. To make the purpose, technical solution, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0019] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0020] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0021] The preferred embodiment of the present invention describes a fundus image classification method based on SLIP pre-training and frequency domain information fusion, such as... Figure 1 and Figure 2 As shown, the fundus image classification method based on SLIP pre-training and frequency domain information fusion includes the following steps: Step S10: Construct a fundus image classification model, which includes a SLIP pre-training module and a latent space feature extraction module.

[0022] Specifically, such as Figure 2As shown, the fundus image classification model constructed in this invention mainly consists of the following parts: pre-training based on the SLIP framework (Self-supervision meets Language-Image Pre-training) and LoRA adapter to achieve alignment of different modal data; feature decomposition and fusion of different image modal data in the frequency domain to achieve decoupled representation of high-frequency details of lesions and low-frequency structures of the fundus; and feature extraction in the latent space of text data using grouped Mamba sequence modeling units to achieve a balance between volume and accuracy.

[0023] The fundus image classification model mainly consists of a SLIP pre-training module and a latent space feature extraction module. The SLIP pre-training module is used to align data from different modalities. The latent space feature extraction module extracts image and text modal features from the frequency and spatial domains in the latent space, respectively. It then groups the feature maps and uses a state-space model to extract features separately, achieving efficient feature extraction with relatively low computational cost. Finally, the high-level semantic features are fused and input into the classifier to achieve accurate classification of fundus images.

[0024] Step S20: Collect pre-training data and input the pre-training data into the SLIP pre-training module for pre-training to obtain the target classification model.

[0025] In this embodiment, the SLIP pre-training module includes a shared image encoder and a text encoder; wherein, the shared image encoder includes a multi-head self-attention layer and a feedforward neural network layer, and the feedforward neural network layer introduces a modality-specific LoRA adapter to enable images of different modalities to obtain independent low-rank incremental updates in the shared backbone.

[0026] Specifically, a set of fundus images is collected for pre-training, and fundus color photography and fluorescein fundus angiography corresponding to each fundus image are obtained from the fundus image set; Each of the fundus color images is subjected to random enhancement processing to obtain multiple first CFP random views and multiple second CFP random views. Each of the fluorescein fundus angiography images is subjected to random enhancement processing to obtain multiple first FFA random views and multiple second FFA random views. All the first CFP random views and the second CFP random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-CFP adapter of the feedforward neural network layer for transformation to obtain multiple first semantic features; All the first FFA random views and the second FFA random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-FFA adapter of the feedforward neural network layer for transformation to obtain multiple second semantic features; The image self-supervised loss is calculated based on all the first semantic features and the second semantic features, and the parameters of the shared image encoder are updated based on the self-supervised loss to complete the training of the shared image encoder. A clinical dataset for pre-training is collected, the clinical dataset is input into the text encoder for encoding, and the encoding result is input into the text projection head for mapping to obtain multiple third semantic features; The image-text contrast loss is calculated based on all the first semantic features, the second semantic features, and the third semantic features. The parameters of the text encoder are then updated based on the image-text contrast loss to complete the training of the text encoder and obtain the target classification model.

[0027] Understandably, to fully utilize multimodal ophthalmic images and clinical text information, this application constructs a SLIP pre-training module based on a LoRA adapter to achieve efficient parameter fine-tuning of the shared image encoder and cross-modal representation learning. Its structure is as follows: Figure 2 As shown, the entire system consists of two parts: an image-image self-supervised branch (shared image encoder) and an image-text contrast branch (text encoder). First, for two different modalities of images in the training set—fundus photography (CFP) and fluorescein fundus angiography (FFA)—two random augmented views (view1 and view2) are generated, resulting in a first and second random CFP view corresponding to the fundus photography, and a first and second random FFA view corresponding to the fluorescein fundus angiography. These are then input into the same shared image encoder. To avoid the multimodal imaging differences weakening the representational ability of the shared encoder, this application introduces a modality-specific LoRA adapter in the encoder's feedforward layer. This allows different modalities to obtain independent low-rank incremental updates within the shared backbone, thereby maintaining the structural and textural differences of each modality with extremely low parameter overhead, resulting in the first and second semantic features.

[0028] The first semantic feature includes the first CFP semantic feature. Second CFP semantic features The second semantic feature includes the first FFT semantic feature. Second FFT semantic features .

[0029] Meanwhile, to enhance the model's clinical semantic understanding capability, this application collects a clinical dataset for pre-training. This clinical dataset is input into the text encoder for encoding. The clinical dataset includes structured clinical data such as the user's gender and weight. The structured clinical data, including the user's gender and weight, is input into the text encoder, and third semantic features are obtained through the text projection head. .

[0030] Further, the step of calculating the image self-supervised loss based on all the first semantic features and the second semantic features specifically includes: Based on all the first CFP semantic features The second CFP semantic features The first FFT semantic feature and the second FFT semantic features Calculate the image self-supervised loss: ; in, This represents the image self-supervised loss. This indicates the batch size for training the shared image encoder. Indicates the current image number. , Indicates the modal type, when When, i represents the CFP modality type, when When i represents the FFT mode type, Represents an exponential function. Indicates hyperparameters, Indicates first Modal semantic features Indicates the second Modal semantic features.

[0031] It is understood that in this embodiment, the encoder output is: the first semantic feature. (Include and ) and second semantic features (Include and These are used to construct the image-image contrast learning task, with two enhanced views as positive samples and the remaining views as negative samples, forming the image self-supervised loss in SLIP. .

[0032] Furthermore, the step of calculating the image-text contrast loss based on all the first semantic features, the second semantic features, and the third semantic features specifically includes: Based on cross-modal alignment constraints, the image-text contrast loss is calculated according to all the first semantic features, the second semantic features, and the third semantic features: ; in, This indicates the batch size used for training the text encoder. This represents the transpose operation, where a and c are ordinal numbers representing semantic features. , This represents the a-th semantic feature. This represents the c-th semantic feature.

[0033] In this embodiment, the first semantic feature Second semantic features With third semantic features Construct CLIP image-text contrast loss between them By employing cross-modal alignment constraints, consistency between images from different modalities and clinical semantic information is maintained within a shared latent space.

[0034] Step S30: Obtain the fundus image to be processed, input the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction, and output the final feature extraction result through the classifier to obtain the classification result of the fundus image to be processed.

[0035] Specifically, the process involves acquiring the fundus image of the target user to be processed, and obtaining the corresponding target fundus color image and target fluorescein fundus angiography from the fundus image to be processed.

[0036] It is understandable that color fundus photography and fluorescein fundus angiography are two independent fundus imaging techniques. They are not extracted from existing fundus images, but are obtained directly through specialized equipment and clinical procedures. The core process is as follows: Color fundus photography does not require contrast agents. First, the pupils of the subject are dilated (mydriatic drops are used to enlarge the pupils) to fully expose the fundus structures. Then, a fundus camera (equipped with a high-resolution color sensor) is used to focus on the fundus through an optical system and capture color images of the retina, optic nerve, macula, and other areas. It can capture two-dimensional structural details of the fundus (such as vascular morphology, pigmentation, and lesion location) in one go. The operation is fast and non-invasive and suitable for routine screening.

[0037] Fluorescein fundus angiography requires the visualization of blood vessels using a contrast agent. The procedure consists of three steps: First, the patient is given an intravenous injection of sodium fluorescein contrast agent. As the contrast agent circulates through the blood vessels in the fundus, the fundus angiography machine (including an excitation light and filter system) is activated to continuously capture dynamic images at fixed time intervals, recording the flow trajectory of the contrast agent in the arterial, venous, and post-conjunctival phases. This results in a dynamic image sequence that clearly shows the integrity of the blood vessels (such as leakage, stenosis, and neovascularization), and is suitable for diagnosing the depth of lesions (such as retinal vasculitis and macular edema).

[0038] The target fundus color image and the target fluorescein fundus angiography are input into the target classification model for multimodal data alignment. Through the target classification model, the different image modal data are decomposed and fused in the frequency domain, realizing the decoupled representation of high-frequency details of lesions and low-frequency structures of the fundus.

[0039] Furthermore, the step of inputting the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction specifically includes: The fundus image to be processed is input into the target classification model for multimodal data alignment to obtain the first target semantic features, the second target semantic features, and the third target semantic features; The first target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and CFP amplitude map and CFP phase map are obtained. The second target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and FFA amplitude map and FFA phase map are obtained. The CFP amplitude map and the FFA amplitude map are fused to obtain the fused amplitude, and the CFP phase map and the FFA phase map are fused to obtain the fused phase. The fused amplitude and the fused phase are convolved separately, and the convolution results are fused to obtain fused features. The fused features are then converted back to the spatial domain using inverse Fourier transform to obtain image features. The third target semantic feature is input into the RMSNorm layer of the grouped Mamba sequence modeling unit for regularization. After adding position encoding to the regularization result, it is mapped through a linear layer to obtain the reconstructed feature. The reconstructed features are grouped, and the reconstructed features of each group are input into a parallel S6 state space unit for long-range dependency capture to obtain the structured feature representation corresponding to each group. The structured feature representation corresponding to each group is activated by SiLU, and the activation results of each group are interacted between groups through Group Shuffle to obtain interactive features. Normalization, nonlinear transformation and dimensionality enhancement operations are performed on the interaction features to obtain grouped features. The grouped features are then residually connected with the third target semantic features to obtain text features. The image features and text features are input into the cross-attention unit for cross-modal fusion to obtain the final feature extraction result.

[0040] Understandably, after SLIP pre-training, this embodiment can obtain refined latent space multimodal image representations: first target semantic features and second target semantic features. To further explore the complementarity of fundus images and FFA image modalities in spatial structure and lesion texture, this embodiment adopts a latent space frequency domain feature decomposition and fusion strategy based on Fourier transform. Specifically, firstly, a two-dimensional fast Fourier transform (FFT) is applied to the latent space features of the two modalities respectively, mapping the spatial features to the frequency domain to obtain their amplitude spectrum and phase spectrum. This process can be expressed as: ; ; ; in, This represents the feature map after applying a two-dimensional fast Fourier transform. Represents the amplitude spectrum. Represents the phase spectrum. These are the numerical values ​​of the original feature map. and These represent the number of sampling points in the height and width directions of the feature map, respectively. The imaginary unit, and They are respectively The real and imaginary parts, Represents frequency domain coordinates, Represents spatial domain coordinates, and These represent the image height and width, respectively.

[0041] Furthermore, the amplitude component of fundus images primarily expresses brightness and texture intensity, highlighting the contrast of lesions; while the phase component expresses the overall spatial layout and shape information of the optic disc, macula, and vascular orientation. The amplitude component of FFA images expresses fluorescence signal intensity, directly reflecting the brightness differences of lesions such as leakage points and non-perfusion areas; while the phase component contains the precise locational relationship of the lesion area.

[0042] After superimposing the amplitude and phase of the two modal images, this embodiment will perform convolution operations on the amplitude spectrum and phase spectrum separately to highlight important regions in the image and further extract complementary information from the feature maps of different modalities. First, a normalization layer is used to stabilize the feature distribution. Then, pointwise convolution PW-Conv is used to achieve linear remapping in the channel dimension, enhancing the correlation and information interaction between channels. Next, depthwise separable convolution DW-Conv is used to independently model the spatial local pattern within each channel, effectively capturing fine-grained local texture and edge features in fundus images and FFA images. This convolution combination, while maintaining extremely low parameter count and computational cost, can effectively extract more discriminative lesion structure information. Finally, the signal is converted back to the spatial domain through inverse Fourier transform, completing the feature extraction and fusion of different modal image signals. This process can be represented as: .

[0043] Furthermore, the third target semantic features are input into the grouped Mamba sequence modeling unit for text feature extraction.

[0044] In this embodiment, the grouped Mamba sequence modeling unit is a key component for text information extraction. This module is introduced to further enhance the temporal modeling capability of text data in the latent space. Mamba belongs to the recently proposed selective state-space model, capable of modeling long-distance dependencies with linear complexity, and outperforming the Transformer's self-attention mechanism in processing long sequence data.

[0045] To further reduce computational load and achieve more efficient text information extraction, this embodiment adopts a grouping structure, dividing the latent space sequence into multiple subgroups. Each subgroup is modeled using an independent state space block, thereby achieving local semantic capture within the group and lightweight long-range dependency modeling. Specifically, the input feature sequence is first regularized through an RMSnorm layer, followed by positional encoding and mapping via a linear layer. The reconstructed feature sequence is then divided into two subgroups, with each group's features fed into parallel S6 state space units to capture long-range dependencies. After SiLU activation, channel information is re-exchanged through a Group Shuffle to achieve inter-group interaction. The fused sequence then undergoes normalization and a feedforward neural network for nonlinear transformation and dimensionality enhancement, and is finally added to the initial input via a residual connection to obtain the final text features.

[0046] Furthermore, such as Figure 3 As shown, the image features and text features are input into the cross-attention unit to obtain the final feature extraction result. In the cross-attention unit, the fused image features are used... and text features For input: First, linear projections are performed on the two types of sequences to obtain Query, Key, and Value. Then, a bidirectional cross-attention mechanism is used to retrieve the Key and Value of the text using the image's Query, injecting clinical semantic information into the image representation; simultaneously, the Key and Value of the image are retrieved using the text's Query, allowing textual features to perceive the visual context of the lesion. After layer normalization, the attention output is then passed through an MLP layer to complete nonlinear remapping, ultimately obtaining a unified multimodal latent space representation that includes both retinal structure and lesion details, as well as semantic cues such as gender and weight, which helps the classifier achieve accurate classification.

[0047] Furthermore, such as Figure 4 As shown, based on the above-mentioned fundus image classification method based on SLIP pre-training and frequency domain information fusion, the present invention also provides a fundus image classification system based on SLIP pre-training and frequency domain information fusion, wherein the fundus image classification system based on SLIP pre-training and frequency domain information fusion includes: Model building module 51 is used to build a fundus image classification model, which includes a SLIP pre-training module and a latent space feature extraction module. The SLIP pre-training module 52 is used to collect pre-training data and input the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model. The target image processing module 53 is used to acquire the fundus image to be processed, input the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction, and output the final feature extraction result through the classifier to obtain the classification result of the fundus image to be processed.

[0048] Furthermore, such as Figure 5 As shown, based on the above-mentioned fundus image classification method and system based on SLIP pre-training and frequency domain information fusion, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0049] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a fundus image classification program 40 based on SLIP pre-training and frequency domain information fusion. This fundus image classification program 40 based on SLIP pre-training and frequency domain information fusion can be executed by the processor 10, thereby implementing the fundus image classification method based on SLIP pre-training and frequency domain information fusion in this application.

[0050] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the fundus image classification method based on SLIP pre-training and frequency domain information fusion.

[0051] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visualized patient interface. The components of the terminal communicate with each other via a system bus.

[0052] In one embodiment, when the processor 10 executes the fundus image classification program 40 based on SLIP pre-training and frequency domain information fusion stored in the memory 20, the following steps are performed: A fundus image classification model is constructed, which includes a SLIP pre-training module and a latent space feature extraction module. Collect pre-training data and input the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model; A fundus image to be processed is acquired, and the fundus image to be processed is input into the target classification model for multimodal data alignment and latent space feature extraction. The final feature extraction result is output through a classifier to obtain the classification result of the fundus image to be processed.

[0053] The SLIP pre-training module includes a shared image encoder and a text encoder. The shared image encoder includes a multi-head self-attention layer and a feedforward neural network layer. The feedforward neural network layer introduces a modality-specific LoRA adapter to enable images of different modalities to obtain independent low-rank incremental updates in the shared backbone.

[0054] The process of collecting pre-training data and inputting it into the SLIP pre-training module for pre-training to obtain a target classification model specifically includes: A set of fundus images is collected for pre-training, and fundus color photography and fluorescein fundus angiography corresponding to each fundus image are obtained from the fundus image set; Each of the fundus color images is subjected to random enhancement processing to obtain multiple first CFP random views and multiple second CFP random views. Each of the fluorescein fundus angiography images is subjected to random enhancement processing to obtain multiple first FFA random views and multiple second FFA random views. All the first CFP random views and the second CFP random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-CFP adapter of the feedforward neural network layer for transformation to obtain multiple first semantic features; All the first FFA random views and the second FFA random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-FFA adapter of the feedforward neural network layer for transformation to obtain multiple second semantic features; The image self-supervised loss is calculated based on all the first semantic features and the second semantic features, and the parameters of the shared image encoder are updated based on the self-supervised loss to complete the training of the shared image encoder. A clinical dataset for pre-training is collected, the clinical dataset is input into the text encoder for encoding, and the encoding result is input into the text projection head for mapping to obtain multiple third semantic features; The image-text contrast loss is calculated based on all the first semantic features, the second semantic features, and the third semantic features. The parameters of the text encoder are then updated based on the image-text contrast loss to complete the training of the text encoder and obtain the target classification model.

[0055] The first semantic feature includes the first CFP semantic feature. Second CFP semantic features The second semantic feature includes the first FFT semantic feature. Second FFT semantic features ; The step of calculating the image self-supervised loss based on all the first semantic features and the second semantic features specifically includes: Based on all the first CFP semantic features The second CFP semantic features The first FFT semantic feature and the second FFT semantic features Calculate the image self-supervised loss: ; in, This represents the image self-supervised loss. This indicates the batch size for training the shared image encoder. Indicates the current image number. , Indicates the modal type, when When, i represents the CFP modality type, when When i represents the FFT mode type, Represents an exponential function. Indicates hyperparameters, Indicates first Modal semantic features Indicates the second Modal semantic features.

[0056] The step of calculating the image-text contrast loss based on all the first semantic features, the second semantic features, and the third semantic features specifically includes: Based on cross-modal alignment constraints, the image-text contrast loss is calculated according to all the first semantic features, the second semantic features, and the third semantic features: ; in, This indicates the batch size used for training the text encoder. This represents the transpose operation, where a and c are ordinal numbers representing semantic features. , This represents the a-th semantic feature. This represents the c-th semantic feature.

[0057] The latent space feature extraction module includes: a frequency domain feature decomposition and fusion unit, a grouped Mamba sequence modeling unit, and a cross-attention unit.

[0058] Specifically, inputting the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction includes: The fundus image to be processed is input into the target classification model for multimodal data alignment to obtain the first target semantic features, the second target semantic features, and the third target semantic features; The first target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and CFP amplitude map and CFP phase map are obtained. The second target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and FFA amplitude map and FFA phase map are obtained. The CFP amplitude map and the FFA amplitude map are fused to obtain the fused amplitude, and the CFP phase map and the FFA phase map are fused to obtain the fused phase. The fused amplitude and the fused phase are convolved separately, and the convolution results are fused to obtain fused features. The fused features are then converted back to the spatial domain using inverse Fourier transform to obtain image features. The third target semantic feature is input into the RMSNorm layer of the grouped Mamba sequence modeling unit for regularization. After adding position encoding to the regularization result, it is mapped through a linear layer to obtain the reconstructed feature. The reconstructed features are grouped, and the reconstructed features of each group are input into a parallel S6 state space unit for long-range dependency capture to obtain the structured feature representation corresponding to each group. The structured feature representation corresponding to each group is activated by SiLU, and the activation results of each group are interacted between groups through Group Shuffle to obtain interactive features. Normalization, nonlinear transformation and dimensionality enhancement operations are performed on the interaction features to obtain grouped features. The grouped features are then residually connected with the third target semantic features to obtain text features. The image features and text features are input into the cross-attention unit for cross-modal fusion to obtain the final feature extraction result.

[0059] In summary, this invention proposes a fundus image classification method, system, terminal, and storage medium based on SLIP pre-training and frequency domain information fusion. The method includes: constructing a fundus image classification model, which includes a SLIP pre-training module and a latent space feature extraction module; collecting pre-training data and inputting the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model; acquiring a fundus image to be processed and inputting the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction; and outputting the final feature extraction result through a classifier to obtain the classification result of the fundus image to be processed. This invention utilizes the SLIP framework for image-text contrastive learning and image self-supervised learning, pre-training image and text encoders to align different modalities in a shared latent space. Then, frequency domain and spatial domain features are extracted from the image and text data in the latent space respectively to capture high-level semantic information. Finally, a cross-attention module is used to achieve cross-modal feature information fusion and input into the classifier for classification, significantly improving the classification accuracy of fundus images.

[0060] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0061] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0062] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A fundus image classification method based on SLIP pre-training and frequency domain information fusion, characterized in that, The fundus image classification method based on SLIP pre-training and frequency domain information fusion includes: A fundus image classification model is constructed, which includes a SLIP pre-training module and a latent space feature extraction module. Collect pre-training data and input the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model; A fundus image to be processed is acquired, and the fundus image to be processed is input into the target classification model for multimodal data alignment and latent space feature extraction. The final feature extraction result is output through a classifier to obtain the classification result of the fundus image to be processed.

2. The fundus image classification method based on SLIP pre-training and frequency domain information fusion according to claim 1, characterized in that, The SLIP pre-training module includes a shared image encoder and a text encoder; The shared image encoder includes a multi-head self-attention layer and a feedforward neural network layer. The feedforward neural network layer introduces a modality-specific LoRA adapter to enable images of different modalities to obtain independent low-rank incremental updates in the shared backbone.

3. The fundus image classification method based on SLIP pre-training and frequency domain information fusion according to claim 2, characterized in that, The process of collecting pre-training data and inputting it into the SLIP pre-training module for pre-training to obtain a target classification model specifically includes: A set of fundus images is collected for pre-training, and fundus color photography and fluorescein fundus angiography corresponding to each fundus image are obtained from the fundus image set; Each of the fundus color images is subjected to random enhancement processing to obtain multiple first CFP random views and multiple second CFP random views. Each of the fluorescein fundus angiography images is subjected to random enhancement processing to obtain multiple first FFA random views and multiple second FFA random views. All the first CFP random views and the second CFP random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-CFP adapter of the feedforward neural network layer for transformation to obtain multiple first semantic features; All the first FFA random views and the second FFA random views are input into the multi-head self-attention layer of the shared image encoder for encoding, and the encoding results are input into the LoRA-FFA adapter of the feedforward neural network layer for transformation to obtain multiple second semantic features; The image self-supervised loss is calculated based on all the first semantic features and the second semantic features, and the parameters of the shared image encoder are updated based on the self-supervised loss to complete the training of the shared image encoder. A clinical dataset for pre-training is collected, the clinical dataset is input into the text encoder for encoding, and the encoding result is input into the text projection head for mapping to obtain multiple third semantic features; The image-text contrast loss is calculated based on all the first semantic features, the second semantic features, and the third semantic features. The parameters of the text encoder are then updated based on the image-text contrast loss to complete the training of the text encoder and obtain the target classification model.

4. The fundus image classification method based on SLIP pre-training and frequency domain information fusion according to claim 3, characterized in that, The first semantic feature includes the first CFP semantic feature. Second CFP semantic features The second semantic feature includes the first FFT semantic feature. Second FFT semantic features ; The step of calculating the image self-supervised loss based on all the first semantic features and the second semantic features specifically includes: Based on all the first CFP semantic features The second CFP semantic features The first FFT semantic feature and the second FFT semantic features Calculate the image self-supervised loss: ； in, This represents the image self-supervised loss. This indicates the batch size for training the shared image encoder. Indicates the current image number. , Indicates the modal type, when When, i represents the CFP modality type, when When i represents the FFT mode type, Represents an exponential function. Indicates hyperparameters, Indicates first Modal semantic features Indicates the second Modal semantic features.

5. The fundus image classification method based on SLIP pre-training and frequency domain information fusion according to claim 4, characterized in that, The step of calculating the image-text contrast loss based on all the first semantic features, the second semantic features, and the third semantic features specifically includes: Based on cross-modal alignment constraints, the image-text contrast loss is calculated according to all the first semantic features, the second semantic features, and the third semantic features: ； in, This indicates the batch size used for training the text encoder. This represents the transpose operation, where a and c are ordinal numbers representing semantic features. , This represents the a-th semantic feature. This represents the c-th semantic feature.

6. The fundus image classification method based on SLIP pre-training and frequency domain information fusion according to claim 1, characterized in that, The latent space feature extraction module includes: a frequency domain feature decomposition and fusion unit, a grouped Mamba sequence modeling unit, and a cross-attention unit.

7. The fundus image classification method based on SLIP pre-training and frequency domain information fusion according to claim 6, characterized in that, The step of inputting the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction specifically includes: The fundus image to be processed is input into the target classification model for multimodal data alignment to obtain the first target semantic features, the second target semantic features, and the third target semantic features; The first target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and CFP amplitude map and CFP phase map are obtained. The second target semantic feature is input into the frequency domain feature decomposition and fusion unit to perform latent space frequency domain feature decomposition based on Fourier transform, and FFA amplitude map and FFA phase map are obtained. The CFP amplitude map and the FFA amplitude map are fused to obtain the fused amplitude, and the CFP phase map and the FFA phase map are fused to obtain the fused phase. The fused amplitude and the fused phase are convolved separately, and the convolution results are fused to obtain fused features. The fused features are then converted back to the spatial domain using inverse Fourier transform to obtain image features. The third target semantic feature is input into the RMSNorm layer of the grouped Mamba sequence modeling unit for regularization. After adding position encoding to the regularization result, it is mapped through a linear layer to obtain the reconstructed feature. The reconstructed features are grouped, and the reconstructed features of each group are input into a parallel S6 state space unit for long-range dependency capture to obtain the structured feature representation corresponding to each group. The structured feature representation corresponding to each group is activated by SiLU, and the activation results of each group are interacted between groups through Group Shuffle to obtain interactive features. Normalization, nonlinear transformation and dimensionality enhancement operations are performed on the interaction features to obtain grouped features. The grouped features are then residually connected with the third target semantic features to obtain text features. The image features and text features are input into the cross-attention unit for cross-modal fusion to obtain the final feature extraction result.

8. A fundus image classification system based on SLIP pre-training and frequency domain information fusion, characterized in that, The fundus image classification system based on SLIP pre-training and frequency domain information fusion includes: The model building module is used to build a fundus image classification model, which includes a SLIP pre-training module and a latent space feature extraction module. The SLIP pre-training module is used to collect pre-training data, and input the pre-training data into the SLIP pre-training module for pre-training to obtain a target classification model. The target image processing module is used to acquire the fundus image to be processed, input the fundus image to be processed into the target classification model for multimodal data alignment and latent space feature extraction, and output the final feature extraction result through the classifier to obtain the classification result of the fundus image to be processed.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a fundus image classification program based on SLIP pre-training and frequency domain information fusion stored in the memory and executable on the processor. When the fundus image classification program based on SLIP pre-training and frequency domain information fusion is executed by the processor, it implements the steps of the fundus image classification method based on SLIP pre-training and frequency domain information fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a fundus image classification program based on SLIP pre-training and frequency domain information fusion. When the fundus image classification program based on SLIP pre-training and frequency domain information fusion is executed by a processor, it implements the steps of the fundus image classification method based on SLIP pre-training and frequency domain information fusion as described in any one of claims 1-7.

Images