Construction method of macular degeneration classification system and macular degeneration classification system

Abstract

The invention provides a construction method of a macular degeneration classification system and the macular degeneration classification system, and the method comprises the steps: carrying out the same AMD degeneration label pairing of an obtained multi-modal image through employing a loose pairing technology, and expanding a multi-modal AMD image data set; performing feature extraction on the OCT-CFP training sample pair by using a feature extraction module fused with a low-rank adaptive technology to obtain a multi-modal feature vector; based on a deep canonical correlation analysis module of the neural network, correlation maximization of the multi-modal features in the shared space is realized; fusing the multi-modal features based on a feature fusion module, outputting target fusion features for classifier training, and training to obtain a target classifier with a minimum loss function; and integrating a loose pairing technology, a feature extraction module, a deep canonical correlation analysis module, a feature fusion module and a target classifier to obtain the macular degeneration classification system. According to the invention, computing resources are saved, and the denaturation classification efficiency and accuracy are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a construction method of a macular degeneration classification system and a macular degeneration classification system. Background Art

[0002] Age-related macular degeneration (AMD) is a retinal degenerative disease that is highly prevalent in the elderly population and is one of the leading causes of irreversible vision loss worldwide. Based on pathological features and disease progression, AMD can be further divided into dry AMD, wet AMD, and polypoidal choroidal vasculopathy (PCV). Given the differences in treatment methods for these subtypes, accurate differentiation between normal retina and different AMD subtypes is of great clinical significance for developing personalized treatment strategies and improving patient prognosis. In the diagnosis of AMD, color fundus photography (CFP) and optical coherence tomography (OCT) are two commonly used noninvasive fundus imaging techniques. CFP can provide high-resolution color images of the retinal surface, which helps detect surface abnormalities, but has limited ability to identify changes in deep retinal structures; while OCT can show the layered structure of the retina in detail and reveal the characteristics of deep lesions, but the judgment of surface color and texture is relatively insufficient. However, when multimodal information (CFP / OCT images and clinical records) is provided, experts make fewer diagnostic errors when recommending referrals. Due to the lack of experienced ophthalmologists, more and more research is devoted to automated AMD screening methods based on CFP images, OCT images, or a combination of both.

[0003] The problems existing in the prior art are:

[0004] ① The accuracy and robustness of most AMD automated screening methods rely on the support of a large number of high-quality annotated datasets. However, in the field of ophthalmology, such datasets are scarce and difficult to obtain due to high annotation costs and privacy issues, which poses a huge challenge to the development and optimization of automated screening systems.

[0005] ② In existing studies, using the weights pre-trained on ImageNet as the initial point of the model for transfer learning has gradually become a common strategy. Recently, a model called RETFound has attracted widespread attention. As a general retinal base model based on VisionTransformer (ViT), this model learns rich visual feature representations by pre-training on large-scale unlabeled retinal image data, and only limited labeled data is needed for efficient fine-tuning in subsequent tasks. It is worth noting that the model performs well in recognizing complex patterns and features related to eye health under single modality. However, as a single-modality model, RETFound's structure can only process single-modality data, which to some extent limits its comprehensive capture of retinal lesion characteristics. Especially when diagnosing complex lesions such as AMD, relying solely on single-modality data may lead to the loss of some key pathological information. In contrast, multimodal data fusion can combine the advantages of different modalities to provide a more comprehensive and detailed description of lesions. Therefore, in the process of fine-tuning downstream tasks, it is an important direction worth exploring to integrate multimodal information into the RETFound model to improve its recognition ability and diagnostic accuracy of complex AMD lesions.

[0006] ③ In practical applications, the training and fine-tuning of multimodal models face the dual challenges of computational complexity and modality fusion. As the number of modalities increases, the number of model parameters will increase exponentially, resulting in a significant increase in computational overhead, especially when integrating CFP and OCT images for AMD diagnosis.

[0007] ④ The heterogeneity between different modal features further increases the difficulty of effective information fusion. How to give full play to the complementary advantages of each modality while ensuring computational efficiency remains a key issue. Summary of the invention

[0008] To this end, the present invention provides a method for constructing a macular degeneration classification system and a macular degeneration classification system, aiming to solve the technical problems of the existing AMD automated screening method, such as difficulty in obtaining training data, insufficient recognition ability, low accuracy, and inability to ensure computational efficiency.

[0009] To achieve the above objectives, the present invention adopts the following technical solutions:

[0010] According to a first aspect of the present invention, the present invention provides a method for constructing a macular degeneration classification system, the method comprising:

[0011] Acquire multiple training samples with different AMD degeneration labels to form a multimodal AMD image dataset; wherein each of the training samples includes an OCT image and a CFP image; and the AMD degeneration labels include normal, dry, wet, and polypoidal choroidal vasculopathy;

[0012] Pairing the OCT images and CFP images in the multimodal AMD image dataset with the same AMD degeneration label using a loose pairing technique to obtain multiple groups of OCT-CFP training sample pairs;

[0013] Constructing a feature extraction module integrating low-rank adaptive technology, and using the feature extraction module to extract features from the OCT-CFP training sample pair to obtain an OCT high-dimensional feature vector and a CFP high-dimensional feature vector;

[0014] Constructing a deep canonical correlation analysis module based on a neural network, performing nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector respectively, and obtaining an OCT target feature vector and a CFP target feature vector with maximized correlation in a shared space;

[0015] Constructing a multi-modal feature fusion module, performing feature fusion on the OCT target feature vector and the CFP target feature vector, and outputting target fusion features;

[0016] Building a classifier based on a preset classification algorithm, taking the target fusion feature as the input of the classifier, the output subsex category as the prediction result, and the AMD degeneration label as the actual result, training and verifying the classifier to obtain a target classifier with a minimized loss function;

[0017] The loose pairing technique, the feature extraction module, the deep canonical correlation analysis module, the feature fusion module and the target classifier are integrated to obtain a macular degeneration classification system for classifying AMD degeneration.

[0018] Furthermore, before pairing the OCT images and CFP images in the multimodal AMD image dataset with the same AMD subtype label using the loose pairing technique, the method further includes:

[0019] Performing image preprocessing on the multimodal image dataset to obtain a preprocessed target image dataset; the target image dataset includes a plurality of training samples carrying different AMD degeneration labels; each training sample includes a preprocessed OCT image and a CFP image; wherein the image preprocessing includes the following steps:

[0020] S1: Deleting abnormal image data in the multimodal image dataset to obtain a first image dataset;

[0021] S2: performing image normalization processing and image weighted enhancement processing on the first image data in the first image data set to obtain a second image data set;

[0022] S3: Perform image enhancement processing on the second image data in the second image data set.

[0023] Furthermore, the image normalization processing includes at least one of normalization processing for the channel value range of image pixels, averaging processing for image pixels, standardization processing for image pixels, and uniform adjustment for image size;

[0024] and / or,

[0025] The processing formula of the image weighted enhancement is expressed as:

[0026] I weight =I org *α+I blur *β+γ

[0027] I blur =I org *kernal h×w

[0028] Among them, I weight I is the second image data; org I is the first image data; blur is the image after Gaussian blurring of the first image data, kernel h×w represents a Gaussian kernel of size h×w; α and β are weighting coefficients; γ is a constant;

[0029] and / or,

[0030] The image enhancement includes at least one of image rotation, image translation, random flipping, application of contrast-limited adaptive histogram equalization to CFP images, and application of median filtering to OCT images.

[0031] Furthermore, the loose pairing technique is used to pair the OCT images and CFP images in the multimodal AMD image dataset with the same AMD degeneration label to obtain multiple groups of OCT-CFP training sample pairs, including:

[0032] For the target image dataset I aug For each training sample i in the aug _f i ) and OCT images (I aug _o i ) as the original pair;

[0033] Initialize an empty loose pairing list;

[0034] For other training samples j (j≠i) in the target image dataset, if the AMD degeneration labels of the training samples i and j are the same, then (I aug _f i , I aug _o j ) and (I aug _f j , I aug _o i ) is added to the loose pairing list;

[0035] The original pairings and the loose pairing lists are merged to obtain a multimodal AMD image set that expands multiple groups of OCT-CFP training sample pairs.

[0036] Furthermore, the feature extraction module integrating the low-rank adaptive technology is constructed, and the feature extraction module is used to extract features from the OCT-CFP training sample pair to obtain the OCT high-dimensional feature vector and the CFP high-dimensional feature vector, including:

[0037] Construct a feature extraction module consisting of two independent ViT-large encoders; the ViT-large encoder contains a patch embedding layer, 24 Transformer blocks, and a 1024-dimensional embedding vector;

[0038] The CFP images (X cfp ∈R H×W×C ) and OCT images (X oct ∈R H ×W×C ) is input into two independent ViT-large encoders;

[0039] The patch embedding layer is used to split the image into fixed-size patches and map them into 1024-dimensional embedding vectors, which are denoted as P cfp ∈R 1×1024 and P oct ∈R 1×1024 ;

[0040] The embedding vector is processed using position encoding to generate the corresponding input feature vectors, which are denoted as Z 0 cfp ∈R 1×1024 and Z 0 oct ∈R 1×1024 ;

[0041] The input feature vector is input into the Transformer block for processing, and the output formula is expressed as:

[0042]

[0043] After 24 Transformer blocks, the high-dimensional feature vector Z to be transformed is obtained. cfp ∈R 1×1024 and Z oct ∈R 1 ×1024 .

[0044] The method further comprises:

[0045] The query matrix WQ and the value projection matrix WV in the self-attention layer of the Transformer block are low-rank decomposed using the low-rank adaptive technology to optimize the weight update, which is expressed as follows:

[0046] ΔW=BA

[0047] Among them, B∈R d×r and A∈R r×d It is the matrix generated by low-rank decomposition, and the rank r is much smaller than the dimension d;

[0048] The input feature vector is processed using the Transformer block after weight update optimization to obtain the OCT high-dimensional feature vector h oct ∈R 1×1024 and CFP high-dimensional eigenvector h cfp ∈R 1×1024 , the formula is as follows:

[0049] h=W0Z+ΔWZ=W0Z+BAZ

[0050] Among them, W0 is the initial pre-training weight; ΔW is the weight update term after low-rank decomposition; Z is the high-dimensional feature vector to be converted.

[0051] Furthermore, the deep canonical correlation analysis module based on the neural network is constructed to perform nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector respectively to obtain the OCT target feature vector and the CFP target feature vector with maximized correlation in the shared space;

[0052] A deep canonical correlation analysis module including two independent deep neural networks is constructed, and the OCT high-dimensional feature vector and the CFP high-dimensional feature vector are respectively input into two independent deep neural networks for multiple nonlinear transformations to generate output feature vectors. cfp ∈R N×d and O oct ∈R N×d, the formula is as follows:

[0053] O cfp =f cfp (h cfp ,W1),O oct =f oct (h oct ,W2)

[0054] in, are the CFP instance feature matrix and the OCT instance feature matrix respectively; N is the number of instances; d1 is the feature dimension of the CFP modality; d2 is the feature dimension of the OCT modality; W1 and W2 are the parameters of the two deep neural networks for nonlinear transformation; d is the output dimension;

[0055] The output features are centralized to construct a cross-modal covariance matrix:

[0056]

[0057] Among them, Σ 11 and Σ 22 Respectively represent the covariance matrix of the CFP mode and the covariance matrix of the OCT mode; Σ 12 is the cross-modal covariance matrix;

[0058] Maximize the output feature O using the following formula cfp and O oct The correlation in the shared space is used to obtain the OCT target feature vector and the CFP target feature vector, which are expressed as follows:

[0059]

[0060] Furthermore, the constructing of a multi-modal feature fusion module to fuse the OCT target feature vector and the CFP target feature vector and output a target fusion feature includes:

[0061] The OCT target feature vector and the CFP target feature vector are used as input features to initialize the sliding average variable y avg It is a zero matrix with the same size as the input features;

[0062] Update the sliding average variable y avg , the formula is as follows:

[0063] y avg =β·y avg +(1-β)·y t

[0064] Where β is the sliding mean attenuation coefficient; y t is the input feature at the current moment; yt is the result of element-wise multiplication of the OCT target feature vector and the CFP target feature vector at time t;

[0065] Output fusion feature y output =y avg ;

[0066] y output As the input for the next time t+1;

[0067] Select a specific time step t * As a termination, the final target fusion feature is output.

[0068] Furthermore, the preset classification algorithm includes at least one of a support vector machine, a random forest, a K nearest neighbor algorithm, and a multi-layer perceptron;

[0069] The loss function formula is expressed as:

[0070]

[0071] in, is the classification loss function; is the deep canonical correlation loss function; N is the number of training samples; C is the number of categories; y i,j is the actual result of training sample i; is the probability that the prediction result of training sample i is category j; λ is the loss weight used to balance the impact of classification loss and deep canonical correlation loss on classifier training.

[0072] According to a second aspect of the present invention, the present invention provides a macular degeneration classification system constructed according to the method for constructing a macular degeneration classification system according to any one of the first aspects of the present invention, the system comprising: a loose pairing module, a feature extraction module, a deep canonical correlation analysis module, a feature fusion module and a target classifier;

[0073] The loose pairing module is used to obtain the OCT image and the CFP image to be classified, and use the loose pairing technology to pair the OCT image and the CFP image with the same AMD degeneration label to obtain multiple groups of OCT-CFP training sample pairs;

[0074] The feature extraction module is used to extract features from the OCT-CFP training sample pair to obtain an OCT high-dimensional feature vector and a CFP high-dimensional feature vector after feature conversion based on a low-rank adaptive technology;

[0075] The deep canonical correlation analysis module is used to perform nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector to obtain the OCT target feature vector and the CFP target feature vector with maximized correlation in the shared space;

[0076] The feature fusion module is used to perform feature fusion on the OCT target feature vector and the CFP target feature vector to obtain a target fusion feature;

[0077] The target classifier is used to take the target fusion feature as input and output the AMD degeneration category of the OCT image and the CFP image to be classified;

[0078] Among them, the AMD degeneration categories include normal, dry, wet and polypoidal choroidal vasculopathy.

[0079] The present invention adopts the above technical solution and has at least the following beneficial effects:

[0080] Through the scheme of the present invention, a large pre-trained single-modal retinal basic model is fine-tuned to achieve multimodal AMD recognition. The present invention not only focuses on how to effectively expand the model to integrate multimodal data during the fine-tuning process, but also focuses on reducing computational costs and optimizing information fusion between modalities. Considering the significant increase in computational complexity during the integration of multimodal data, the present invention adopts the Low-Rank Adaptation (LoRA) method, which introduces the adaptation mechanism of the low-rank matrix to decompose the full parameter matrix into the product of two low-rank matrices, so that only part of the parameters of the model are optimized during fine-tuning. This method significantly reduces the computational burden and memory requirements while ensuring that the performance and accuracy of the model are not affected when processing multimodal data. In order to bridge the heterogeneity of different modal features, the present invention also introduces the deep canonical correlation analysis (DCCA) method, which uses two independent neural networks to transform the nonlinear features between CFP and OCT modal features under fine-grained cross-modality, and combines the classic canonical correlation analysis (CCA) for regularization to maximize the linear correlation between the two modal features. During the training process, the features of the two modalities are mapped into a coordinated high-dimensional hyperspace, enabling the model to learn highly correlated multimodal features. It should also be noted that the present invention completely retains the original network structure of the large pre-trained single-modal retinal base model, thereby ensuring that the feature processing capability under single modality is not affected.

[0081] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0083] Figure 1 A schematic diagram showing a process of constructing a macular degeneration classification system provided by an embodiment of the present invention;

[0084] Figure 2 A module processing flow chart of a method for constructing a macular degeneration classification system provided by an embodiment of the present invention is shown;

[0085] Figure 3 A schematic diagram of the image preprocessing process provided by an embodiment of the present invention is shown;

[0086] Figure 4 A schematic diagram of a network structure of a low-rank adaptive technology provided by an embodiment of the present invention is shown;

[0087] Figure 5 A schematic diagram of the network structure of a deep canonical correlation analysis module provided by an embodiment of the present invention is shown;

[0088] Figure 6 A schematic diagram of the structure of a macular degeneration classification system provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0089] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0090] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0091] The embodiment of the present invention provides a method for constructing a macular degeneration classification system, such as Figure 1 As shown, at least the following steps S101 to S107 may be included:

[0092] Step S101 , obtaining a plurality of training samples carrying different AMD degeneration labels to form a multimodal AMD image dataset.

[0093] The multimodal AMD image data set obtained in the embodiment of the present invention includes a subsample set corresponding to normal and different age-related macular degeneration (AMD) subclass disease labels, and each training sample in each subsample set includes color fundus photography (CFP) and optical coherence tomography (OCT). In other words, each training sample includes an OCT image and a CFP image. The AMD degeneration labels in the embodiment of the present invention include normal, dry, wet, and polypoidal choroidal vasculopathy.

[0094] In the research process of the present invention, in order to construct a multimodal AMD image dataset, 1094 color fundus photography (CFP) images and 1289 optical coherence tomography (OCT) images were first collected from the public medical imaging dataset MMC-AMD. These images cover normal retinas from 1093 eyes and different subtypes of AMD, including dry AMD (dryAMD), wet AMD (wetAMD) and polypoidal choroidal vasculopathy (PCV). Each image was annotated in detail by a professional ophthalmologist (i.e., AMD degeneration label) to ensure accurate classification of the lesion type. In the process of constructing the dataset, since the CFP image is a three-channel color image and the OCT image is a single-channel grayscale image, in order to ensure that the images of the two modalities can be correctly paired and aligned when input into the model, the embodiment of the present invention can use a channel replication method to expand the single-channel OCT image into three channels to ensure the consistency of the CFP and OCT images in the channel dimension. In addition, in order to reduce the impact of class imbalance on the classification results, the categories with a small number of samples can also be resampled.

[0095] like Figure 2 As shown in FIG. 1 , a module processing flow chart of a method for constructing a macular degeneration classification system is shown. After obtaining a multimodal AMD image dataset containing two modal image data (OCT image / CFP image), the multimodal image dataset can also be subjected to image preprocessing to obtain a preprocessed target image dataset. Figure 3 FIG. 1 is a flowchart of image preprocessing according to an embodiment of the present invention.

[0096] Specifically, image preprocessing may include the following steps:

[0097] Step S1: deleting abnormal image data in the multimodal image data set to obtain a first image data set;

[0098] Step S2: performing image normalization processing and image weighted enhancement processing on the first image data in the first image data set to obtain a second image data set;

[0099] Step S3: performing image enhancement processing on the second image data in the second image data set.

[0100] After performing abnormal data cleaning and other operations on the two modal image data, the embodiment of the present invention performs data preprocessing steps including image normalization, image weighted enhancement, and image enhancement to improve the image quality and meet the requirements of the model input. In this process, the original image input is I org , after image normalization and image weighted enhancement, we get image I weight Finally, the image enhancement operation is performed to obtain the image output as I aug.

[0101] Among them, the image normalization processing includes normalization processing for the channel value range of image pixels, averaging processing for image pixels, standardization processing for image pixels, and consistent adjustment for image size. Specifically, the normalization processing for the channel value range of image pixels can be understood as range scaling processing for each channel (R, G, B) to ensure that the value range of each channel is consistent; averaging processing can make the image more consistent by calculating the average value of image pixels and subtracting the average value from each pixel in the image; standardization processing can reduce the variation of image data by calculating the standard deviation of image pixels and dividing each pixel in the image by the standard deviation to make the data more stable; resizing refers to adjusting the size of the two modal images to be consistent to ensure data consistency and meet the input requirements of the model, for example, the size is adjusted to 224*224.

[0102] Image weighted enhancement refers to the enhancement of the original image I org Gaussian Blurring is performed to reduce the high-frequency details of the image and the original image I org And the resulting Gaussian blurred image I blur Weighted to generate enhanced image I weight , the processing formula can be expressed as:

[0103] I weight =I org *α+I blur *β+γ

[0104] Gaussian blurring refers to using a Gaussian kernel to blur the original image. org Perform a convolution operation to produce a blurred image I blur , the formula is as follows:

[0105] I blur =I org *kernal h×w

[0106] Among them, I weight I is the second image data; org I is the first image data; blur is the image after Gaussian blurring of the first image data, kernel h×w represents a Gaussian kernel of size h×w; α and β are weighting coefficients; γ is a constant.

[0107] Image enhancement may include image rotation, image translation, random flipping, contrast-limited adaptive histogram equalization for CFP images, and median filtering for OCT images. Specifically, the following steps may be included: rotating the image by 45° or 90°; randomly translating the image horizontally (x-axis) or vertically (y-axis), with the translation distance ranging from 0 to 10% of the image width or height; random flipping; applying contrast-limited adaptive histogram equalization (CLAHE) for CFP images to enhance low-contrast areas in the image; using a median filtering method with a 3×3 kernel for OCT images to remove salt and pepper noise in the image by replacing the pixel value with the median of the neighboring pixel values, while retaining the edge information of the image.

[0108] Step S102 , pairing the OCT images and CFP images in the multimodal AMD image dataset with the same AMD degeneration label using a loose pairing technique to obtain multiple groups of OCT-CFP training sample pairs.

[0109] To obtain more training examples, a natural strategy is to strictly select CFP and OCT images from the same eye based on the disease label. Assume that there are two eyes with wet AMD in the training set. a and eye b , that is, setting And set Where f and o represent CFP image and OCT image respectively. Strict pairing. In order to increase the number of multimodal examples for training, the embodiment of the present invention constructs input pairs based on disease category labels rather than the same eye. That is, as long as the disease category labels are the same, CFP images can be used to construct input pairs with OCT images. Based on the loose pairing method, the two eyes eye a and eye b The original five sets of available training instances are expanded to ten sets of available training instances: That is, as long as the disease category labels of the two modal images are the same, they can be paired together to form a loose image pair. Through the loose pairing technology of the embodiment of the present invention, more training instances are constructed for the category labels of patients, and the original training data is effectively expanded. The specific implementation process of the loose pairing technology includes the following steps:

[0110] Step S1: I aug As the data set input, I aug Each sample in contains CFP images and OCT images; each sample i corresponds to a true AMD subtype label;

[0111] Step S2: record the AMD subtype label of sample i as "Pi ", where P i One of the four AMD subtype labels {"normal", "dryAMD", "wetAMD", "pcv"}; establish a disease category dictionary to map the AMD subtype label of each sample to the corresponding multimodal image data;

[0112] Step S3: For the target image dataset I aug For each training sample i in the aug _f i ) and OCT images (I aug _o i ) as the original pair;

[0113] Step S4: Initialize an empty loose pairing list (LoosePairs=[]);

[0114] Step S5: Traverse the target image dataset I aug , for other training samples j (j≠i), if the AMD variant labels of training samples i and j are the same, a new multimodal image pairing is constructed, and (I aug _f i , I aug _o j ) and (I aug _f j , I aug _o i ) is added to the loose pairing list;

[0115] Step S6: Merge the original pairing list and the loose pairing list to obtain a multimodal AMD image set with expanded multiple groups of OCT-CFP training sample pairs.

[0116] Step S103, constructing a feature extraction module integrating low-rank adaptive technology, and using the feature extraction module to extract features from the OCT-CFP training sample pairs to obtain OCT high-dimensional feature vectors and CFP high-dimensional feature vectors.

[0117] The feature extraction module in the embodiment of the present invention is based on the ViT-large encoder in the RETFound model, which is trained on large-scale unlabeled retinal images through self-supervised learning and is suitable for ocular and systemic disease detection tasks with clear annotations. The model adopts a specially configured autoencoder architecture, which includes an encoder and a decoder. Among them, the encoder uses a ViT-large structure, including 24 Transformer blocks and a 1024-dimensional embedding vector. The encoder processes unmasked 16×16 pixel image blocks and projects these image blocks into 1024-dimensional feature vectors. The Transformer block generates high-level features through a multi-head self-attention mechanism and a multi-layer perceptron. The decoder part is a ViT-small structure, which reconstructs the masked image blocks through 8 Transformer blocks and a 512-dimensional embedding vector. In the embodiment of the present invention, the decoder part is discarded, and only the ViT-large encoder is retained for feature extraction. The CFP image and the OCT image are processed by two independent ViT-large encoders respectively, so as to ensure that the features of each modality are extracted independently. Specifically, let the input of the CFP image be X cfp ∈R H×W×C , the input of OCT image is X oct ∈R H×W×C Each image first passes through the patch embedding layer to split it into patches of fixed size, and then undergoes position encoding to generate the corresponding input feature vectors, which are denoted as P and cfp ∈R 1×1024 and P oct ∈R 1×1024 Each ViT-large encoder is processed through a series of Transformer blocks, each of which contains a multi-head self-attention mechanism and a feed-forward neural network, and finally obtains two high-dimensional feature vectors Z cfp ∈R 1×1024 and Z oct ∈R 1×1024 . These feature vectors represent the representation of the image after deep encoding, capturing important information in different modalities. It is worth noting that although CFP and OCT images are processed in independent encoders, both follow the same feature extraction process, thereby ensuring the consistency of cross-modal features to facilitate multimodal interaction in the subsequent feature fusion stage. In summary, the feature extraction module in the embodiment of the present invention includes two independent ViT-large encoders, each ViT-large encoder includes a patch embedding layer, 24 Transformer blocks and a 1024-dimensional embedding vector. The specific implementation process of the feature extraction module includes the following steps:

[0118] Step S1: respectively transform the CFP images (X cfp ∈R 224×224×3 ) and OCT images (X oct ∈R 224×224×3 ) is input into two independent ViT-large encoders;

[0119] Step S2: The patch embedding layer is used to split the patch into fixed-size (16×16) patches, and each patch is mapped to a 1024-dimensional embedding vector, denoted as P cfp ∈R 1×1024 and P oct ∈R 1×1024 ;

[0120] Step S3: Using the position code E pos The embedded vector is processed to generate the corresponding input feature vector, which is denoted as Z 0 cfp ∈R 1×1024 and Z 0 oct ∈R 1×1024 ;

[0121] Step S4: Input the input feature vector into the Transformer block pair for processing. Each block contains a multi-head attention mechanism and a feedforward neural network. The output formula is expressed as:

[0122]

[0123] After 24 Transformer blocks, the high-dimensional feature vector Z to be transformed is obtained. cfp ∈R 1×1024 and Z oct ∈R 1 ×1024 .

[0124] Furthermore, in the low-rank adaptation process, the high-dimensional feature vector to be converted can be further converted to reduce the computational load of the model fine-tuning stage. In order to cope with the high computational requirements of cross-modal feature fusion of CFP and OCT images during fine-tuning, the embodiment of the present invention adopts low-rank adaptation (LoRA) technology to fine-tune the model parameters. Figure 4 As shown in the figure, it is a schematic diagram of the network structure of low-rank adaptive technology. The core idea is to optimize the self-attention layer of Transformer through low-rank decomposition to achieve effective control of computational complexity.

[0125] Specifically, the input CFP image and OCT image are converted into the input feature vector Z after patch embedding and position encoding. 0 cfp ∈R 1×1024 and Z 0 oct ∈R 1×1024 These vectors are then fed into their respective ViT-large encoders, each of which consists of multiple Transformer blocks, which contain the query matrix WQ, key matrix WK, and value projection matrix WV of the multi-head self-attention mechanism. The query matrix WQ and value projection matrix WV in the self-attention layer of the Transformer block are low-rank decomposed using low-rank adaptation techniques to optimize weight updates, as shown in the following formula:

[0126] ΔW=BA

[0127] Among them, B∈R d×r and A∈R r×d The matrix generated by low-rank decomposition has a rank r that is much smaller than the dimension d, which aims to reduce the computational load in the model fine-tuning stage.

[0128] After LoRA processing, the eigenvectors h of the CFP and OCT modalities are cfp ∈R 1×1024 and h oct ∈R 1×1024 They are generated by the following formulas:

[0129] h=W0Z+ΔWZ=W0Z+BAZ

[0130] Among them, W0 is the initial pre-training weight; ΔW is the weight update term after low-rank decomposition; Z is the high-dimensional feature vector to be converted before low-rank adaptive fine-tuning.

[0131] It can be understood that by independently processing and updating the features of CFP images and OCT images in this way, it is ensured that the multimodal data has efficient and accurate feature expression before fusion. In addition, by setting different ranks of low-rank matrices, the balance between model complexity and computational efficiency can be adjusted. When fine-tuning the model, only these low-rank matrices are adjusted, and other parameters remain unchanged. This strategy can effectively reduce training time and computing resources while maintaining model performance.

[0132] Step S104, constructing a deep canonical correlation analysis module based on a neural network, performing nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector respectively, and obtaining the OCT target feature vector and the CFP target feature vector with maximized correlation in the shared space;

[0133] The embodiment of the present invention introduces deep canonical correlation analysis (DCCA) into the multimodal AMD classification task, aiming to solve the challenges of heterogeneity and nonlinear feature alignment and correlation modeling of CFP and OCT modal data. The core idea of ​​DCCA is to perform nonlinear mapping on the input features of each modality through a deep neural network, transform it into a shared latent space, and then maximize the correlation of different modal features in the space to enhance the synergy between modalities, thereby improving the effect of multimodal feature fusion. Figure 5 As shown, this is a schematic diagram of the network structure of the deep canonical correlation analysis module.

[0134] Specifically, the deep canonical correlation analysis module consists of two independent neural networks, which process the features of the CFP modality and the OCT modality respectively. and Represent the instance feature matrices from the CFP modality and the OCT modality, respectively. Where N is the number of instances, d1 and d2 are the feature dimensions of the two modalities. The input features of each modality are transformed through multiple layers of nonlinear transformations in their respective deep neural networks to generate two new feature representations O cfp ∈R N×d and O oct ∈R N×d :

[0135] O cfp =f cfp (H cfp ,W1),O oct =f oct (H oct ,W2)

[0136] Among them, W1 and W2 are the parameters of the two deep neural networks for nonlinear transformation; d is the output dimension. In order to measure the correlation between the two modal features, we use the canonical correlation analysis (CCA) method. First, the output features of the two modalities are centered, and then the cross-modal covariance matrix is ​​constructed:

[0137]

[0138] Among them, ∑ 11 and∑ 22 Represent the covariance matrix of CFP mode and the covariance matrix of OCT mode respectively; ∑ 12 is the cross-modal covariance matrix. The goal of DCCA is to jointly optimize the neural network parameters W1 and W2 to maximize the cross-modal feature O cfp and O oct The correlation in the shared space can be expressed by the following formula:

[0139]

[0140] After training the two neural networks, the transformed features O cfp and O oct The projection direction in the shared space is adjusted to maximize its correlation. According to the above construction process, DCCA brings the following advantages to multimodal AMD recognition: by transforming different modalities separately, the two modalities can be explicitly extracted. cfp and O oct The changing characteristics of the two modal transformations can be easily checked to examine the characteristics and relationship of the two modal transformations; under the specified CCA constraints, the nonlinear mapping (f cfp () and f oct ()), and make the model retain the information related to AMD.

[0141] Step S105, constructing a multi-modal feature fusion module, performing feature fusion on the OCT target feature vector and the CFP target feature vector, and outputting target fusion features.

[0142] When automatically diagnosing AMD disease, relying solely on single-modal data may lead to the loss of some key pathological information. In addition, the deep learning training process pursues the global optimal solution, but the setting of hyperparameters (such as learning rate) often causes the model to fall into the local optimal solution of certain specific points, which in turn causes the model to stop optimizing. The embodiment of the present invention integrates the features of the model output by constructing a multi-modal feature fusion module FFM, which can combine the advantages of different modalities to provide a more comprehensive and detailed description of the lesion, significantly improve the model's comprehensive analysis capabilities for key lesion information, and help solve the optimization dilemma caused by local optimal solutions. Specifically, the embodiment of the present invention uses a sliding average method to fuse the feature outputs of the two modal images, and the steps are as follows:

[0143] Step S1: taking the OCT target feature vector and the CFP target feature vector as input features;

[0144] Step S2: Set a sliding average variable to record the fused features; initialize the sliding average variable y avg is a zero matrix (y avg =0), the size (H×W×C) is consistent with the input feature;

[0145] Step S3: For each time t, update the sliding average variable y avg , the formula is as follows:

[0146] y avg =β·y avg +(1-β)·y t

[0147] Among them, β is the sliding average attenuation coefficient, which takes a value close to 1. The sliding average attenuation coefficient determines the influence of historical information on the current fusion result; y t is the input feature at the current moment; y t is the result of element-wise multiplication of the OCT target feature vector and the CFP target feature vector at time t. Preferably, the sliding average attenuation coefficient may be 0.8;

[0148] Step S4: Output fusion feature y output =y avg ;

[0149] Step S5: Set y output As the input for the next time t+1;

[0150] Step S6: Select a specific time step t * As a termination, the final target fusion feature is output.

[0151] Based on the sliding average process proposed in the embodiment of the present invention, the fusion features at each moment will take into account the information of the historical moments, thereby achieving smooth fusion of multimodal image features.

[0152] Step S106, construct a classifier based on a preset classification algorithm, use the target fusion feature as the input of the classifier, the output subsex category as the prediction result, and the AMD degeneration label as the actual result, train and verify the classifier to obtain a target classifier with minimized loss function.

[0153] The embodiment of the present invention is aimed at constructing a macular degeneration classification system. The obtained target fusion features are used as input, and after being processed by activation function, global average pooling, discarding method and fully connected layer, they enter the classifier constructed by using a preset classification algorithm to train and optimize the classifier. The preset classification algorithm can be selected from support vector machine (SVM), random forest (RandomForest), K-Nearest Neighbors (KNN), multilayer perceptron (MultilayerPerceptron, MLP), etc. The training goal of the classifier is to minimize the loss function. The embodiment of the present invention uses cross entropy loss as the classification loss. To optimize:

[0154]

[0155] Where N is the number of training samples; C is the number of categories; y i,j is the actual label of training sample i; is the probability that training sample i is predicted to be of category j.

[0156] In addition to the classification loss, the embodiment of the present invention also introduces the DCCA loss function, which aims to enhance the correlation between multimodal features. In order to adapt to the gradient descent optimization process, the objective function of DCCA is converted to a negative correlation, so that it can be optimized by minimization method. Its loss function is defined as follows:

[0157]

[0158] In the embodiment of the present invention, the overall loss of the classifier is a weighted combination of the classification loss and the DCCA loss, that is:

[0159]

[0160] Among them, λ is used to balance the classification loss and deep canonical correlation loss The loss weight that affects the classifier training. Overall loss function The relevance optimization of feature extraction and the accuracy of classification tasks are comprehensively considered to achieve the coordinated optimization of the two. Through this dual optimization strategy, the classifier can not only learn highly relevant multimodal features, but also show higher accuracy in classification tasks, thus achieving better overall performance in multimodal AMD classification tasks.

[0161] Step S107, integrating the loose pairing technology, the feature extraction module, the deep canonical correlation analysis module, the feature fusion module and the target classifier to obtain a macular degeneration classification system for classifying AMD degeneration.

[0162] The macular degeneration classification system constructed by the embodiment of the present invention obtains OCT images and CFP images, sets the two modal images to a preset size, and processes them respectively through the above-mentioned loose pairing technology, feature extraction module, deep canonical correlation analysis module, and feature fusion module, and then uses the trained target classifier to perform classification prediction on the two modal images as input. The final classification prediction result is one of the four AMD degeneration labels of normal retina (normal), dry AMD (dryAMD), wet AMD (wetAMD), and polypoidal choroidal vasculopathy (PCV).

[0163] The present invention proposes a method for constructing a macular degeneration classification system, aiming to solve the problems of insufficient accuracy and high computational complexity caused by the scarcity of high-quality annotated data and modality heterogeneity in the existing AMD automated screening. By fine-tuning a large pre-trained single-modality retinal basic model (such as RETFound), combined with the multimodal information of color fundus photography (CFP) and optical coherence tomography (OCT) images, the present invention realizes the effective fusion of multimodal features. The RETFound model already has a powerful feature extraction capability in a single modality, and the present invention extends it to multimodal applications, so that it can not only make full use of the surface texture information of CFP images, but also effectively capture the deep structural features in OCT images, thereby significantly improving the recognition ability and diagnostic accuracy of complex AMD lesions. In order to cope with the computational burden brought by multimodal fusion, the present invention introduces Low-Rank Adaptation (LoRA) technology. LoRA decomposes the full parameter matrix in the model into two low-rank matrices, so that only a small number of parameters need to be adjusted during the fine-tuning process, which greatly reduces the computational and memory requirements. At the same time, in order to solve the heterogeneity problem between CFP and OCT modalities, the deep canonical correlation analysis (DCCA) method is adopted. Two independent neural networks are used to perform nonlinear transformations on the features of different modalities, and their linear correlation is maximized through regularization. This method can map the features of different modalities into a unified high-dimensional space, thereby laying the foundation for the deep fusion of cross-modal information. After that, the feature outputs of the two modal images are fused by the sliding average method, and the features between the modalities are fused by element-by-element multiplication. In this way, multimodal information can be fully utilized to avoid the loss and redundancy of feature information. At the same time, the sliding average method can smooth historical information and improve the stability and consistency of the fusion results. In addition, the sliding average multimodal image fusion method is real-time and can be fused while extracting features, without waiting for all features to be extracted before processing. This can save time and computing resources and improve the efficiency of the overall processing. The superiority of the present invention is mainly reflected in the following five aspects:

[0164] ① By fine-tuning the large pre-trained unimodal retinal base model, it is effectively expanded into a model for multimodal AMD recognition; while achieving multimodal recognition, the powerful processing capabilities of the large unimodal model in a single modality are retained.

[0165] ② In order to cope with the challenge of increased computational complexity in the process of multimodal data integration, the present invention introduces the LoRA method, which decomposes the model parameters into a low-rank matrix, so that only a small number of parameters need to be optimized in the fine-tuning stage, significantly reducing the demand for computing resources;

[0166] ③ In order to solve the heterogeneity problem between different modal features, the present invention adopts the DCCA method to perform nonlinear transformation on CFP and OCT modal features through independent neural networks, thereby achieving highly correlated multimodal feature learning;

[0167] ④ The sliding average method is used to fuse the features of the two modal images, making full use of multimodal information, avoiding feature loss and redundancy, and improving the stability and consistency of the fusion results. At the same time, it saves time and computing resources, improves the overall processing efficiency, and is of great value to the two-modal image data processing and model training.

[0168] ⑤The present invention can be used for AMD classification during primary screening.

[0169] Furthermore, the present invention also provides a macular degeneration classification system, through Figure 1 The macular degeneration classification system is constructed by the method shown in Figure 6 As shown, the system may at least include: a loose pairing module 610 , a feature extraction module 620 , a deep canonical correlation analysis module 630 , a feature fusion module 640 and a target classifier 650 .

[0170] The loose pairing module 610 may be used to obtain the OCT image and the CFP image to be classified, and pair the OCT image and the CFP image with the same AMD degeneration label using the loose pairing technique to obtain multiple groups of OCT-CFP training sample pairs;

[0171] The feature extraction module 620 can be used to extract features from the OCT-CFP training sample pair to obtain the OCT high-dimensional feature vector and the CFP high-dimensional feature vector after feature conversion based on the low-rank adaptive technology;

[0172] The deep canonical correlation analysis module 630 can be used to perform nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector to obtain the OCT target feature vector and the CFP target feature vector with the maximum correlation in the shared space;

[0173] The feature fusion module 640 may be used to perform feature fusion on the OCT target feature vector and the CFP target feature vector to obtain a target fusion feature;

[0174] The target classifier 650 may be configured to take the target fusion feature as input and output the AMD degeneration category of the OCT image and the CFP image to be classified;

[0175] Among them, AMD degeneration categories include normal, dry, wet, and polypoidal choroidal vasculopathy.

[0176] It should be noted that for other corresponding descriptions of the functional modules involved in the macular degeneration classification system provided in the embodiment of the present invention, reference can be made to Figure 1 The corresponding description of the method shown will not be repeated here.

[0177] Those skilled in the art can clearly understand that the specific working processes of the systems, devices, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and for the sake of brevity, they are not further described here.

[0178] In addition, the functional units in various embodiments of the present invention may be physically independent of each other, or two or more functional units may be integrated together, or all functional units may be integrated into one processing unit. The above integrated functional units may be implemented in the form of hardware, or in the form of software or firmware.

[0179] Those skilled in the art can understand that if the integrated functional unit is implemented in the form of software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can essentially or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, which includes a number of instructions to enable a computing device (such as a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention when running the instructions. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes.

[0180] Alternatively, all or part of the steps of implementing the aforementioned method embodiments may be accomplished by hardware associated with program instructions (such as a computing device such as a personal computer, a server, or a network device), and the program instructions may be stored in a computer-readable storage medium. When the program instructions are executed by a processor of a computing device, the computing device executes all or part of the steps of the methods described in the embodiments of the present invention.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate from the protection scope of the present invention.

Claims

1. A method for constructing a macular degeneration classification system, characterized in that: The method comprises: Acquire multiple training samples with different AMD degeneration labels to form a multimodal AMD image dataset; wherein each of the training samples includes an OCT image and a CFP image; and the AMD degeneration labels include normal, dry, wet, and polypoidal choroidal vasculopathy; Pairing the OCT images and CFP images in the multimodal AMD image dataset with the same AMD degeneration label using a loose pairing technique to obtain multiple groups of OCT-CFP training sample pairs; Constructing a feature extraction module integrating low-rank adaptive technology, and using the feature extraction module to extract features from the OCT-CFP training sample pair to obtain an OCT high-dimensional feature vector and a CFP high-dimensional feature vector; Constructing a deep canonical correlation analysis module based on a neural network, performing nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector respectively, and obtaining an OCT target feature vector and a CFP target feature vector with maximized correlation in a shared space; Constructing a multi-modal feature fusion module to perform feature fusion on the OCT target feature vector and the CFP target feature vector, and outputting target fusion features; Building a classifier based on a preset classification algorithm, taking the target fusion feature as the input of the classifier, the output subsex category as the prediction result, and the AMD degeneration label as the actual result, training and verifying the classifier to obtain a target classifier with a minimized loss function; The loose pairing technique, the feature extraction module, the deep canonical correlation analysis module, the feature fusion module and the target classifier are integrated to obtain a macular degeneration classification system for classifying AMD degeneration.

2. The method according to claim 1, characterized in that Before pairing the OCT images and CFP images in the multimodal AMD image dataset with the same AMD subtype label using the loose pairing technique, the method further includes: Performing image preprocessing on the multimodal image dataset to obtain a preprocessed target image dataset; the target image dataset includes a plurality of training samples carrying different AMD degeneration labels; each training sample includes a preprocessed OCT image and a CFP image; wherein the image preprocessing includes the following steps: S1: Deleting abnormal image data in the multimodal image dataset to obtain a first image dataset; S2: performing image normalization processing and image weighted enhancement processing on the first image data in the first image data set to obtain a second image data set; S3: Perform image enhancement processing on the second image data in the second image data set.

3. The method according to claim 2, characterized in that The image normalization processing includes at least one of normalization processing on the channel value range of image pixels, averaging processing on image pixels, standardization processing on image pixels, and uniform adjustment on image size; and / or, The processing formula of the image weighted enhancement is expressed as: I weight =I org *a+I blur *b+c I blur =I org *kernal h×w Among them, I weight I is the second image data; org I is the first image data; blur is the image after Gaussian blurring of the first image data, kernel h×w represents a Gaussian kernel of size h×w; α and β are weighting coefficients; γ is a constant; and / or, The image enhancement includes at least one of image rotation, image translation, random flipping, application of contrast-limited adaptive histogram equalization to CFP images, and application of median filtering to OCT images.

4. The method according to claim 2, characterized in that: The method uses the loose pairing technique to pair the OCT images and CFP images in the multimodal AMD image dataset with the same AMD degeneration label to obtain multiple groups of OCT-CFP training sample pairs, including: For the target image dataset I aug For each training sample i in the aug _f i ) and OCT images (I aug _o i ) as the original pair; Initialize an empty loose pairing list; For other training samples j (j≠i) in the target image dataset, if the AMD degeneration labels of the training samples i and j are the same, then (I aug _f i , I aug _o j ) and (I aug _f j , I aug _o i ) is added to the loose pairing list; The original pairings and the loose pairing lists are merged to obtain a multimodal AMD image set that expands multiple groups of OCT-CFP training sample pairs.

5. The method according to claim 1, characterized in that The constructing of a feature extraction module integrating low-rank adaptive technology, using the feature extraction module to extract features from the OCT-CFP training sample pair, and obtaining an OCT high-dimensional feature vector and a CFP high-dimensional feature vector, includes: Construct a feature extraction module consisting of two independent ViT-large encoders; the ViT-large encoder contains a patch embedding layer, 24 Transformer blocks, and a 1024-dimensional embedding vector; The CFP images (X cfp ∈R H×W×C ) and OCT images (X oct ∈R H×W×C ) is input into two independent ViT-large encoders; The patch embedding layer is used to split the image into fixed-size patches and map them into 1024-dimensional embedding vectors, which are denoted as P cfp ∈R 1×1024 and P oct ∈R 1×1024 ; The embedding vector is processed using position encoding to generate the corresponding input feature vectors, which are denoted as Z 0 cfp ∈R 1 ×1024 and Z 0 oct ∈R 1×1024 ; The input feature vector is input into the Transformer block for processing, and the output formula is expressed as: After 24 Transformer blocks, the high-dimensional feature vector Z to be transformed is obtained. cfp ∈R 1×1024 and Z oct ∈R 1×1024 .

6. The method according to claim 5, characterized in that The method further comprises: The query matrix WQ and the value projection matrix WV in the self-attention layer of the Transformer block are low-rank decomposed using the low-rank adaptive technology to optimize the weight update, which is expressed as follows: ΔW=BA Among them, B∈R d×r and A∈R r×d It is the matrix generated by low-rank decomposition, and the rank r is much smaller than the dimension d; The input feature vector is processed using the Transformer block after weight update optimization to obtain the OCT high-dimensional feature vector h oct ∈R 1×1024 and CFP high-dimensional eigenvector h cfp ∈R 1×1024 , the formula is as follows: h=W0Z+ΔWZ=W0Z+BAZ Among them, W0 is the initial pre-training weight; ΔW is the weight update term after low-rank decomposition; Z is the high-dimensional feature vector to be converted.

7. The method according to claim 1, characterized in that The deep canonical correlation analysis module based on the neural network is constructed to perform nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector respectively to obtain the OCT target feature vector and the CFP target feature vector with maximized correlation in the shared space; A deep canonical correlation analysis module including two independent deep neural networks is constructed, and the OCT high-dimensional feature vector and the CFP high-dimensional feature vector are respectively input into two independent deep neural networks for multiple nonlinear transformations to generate output feature vectors. cfp ∈R N×d and O oct ∈R N×d , the formula is as follows: Oh cfp =f cfp (h cfp ,W1),O oct =f oct (h oct ,W2) in, are the CFP instance feature matrix and the OCT instance feature matrix respectively; N is the number of instances; d1 is the feature dimension of the CFP modality; d2 is the feature dimension of the OCT modality; W1 and W2 are the parameters of the two deep neural networks for nonlinear transformation; d is the output dimension; The output features are centralized to construct a cross-modal covariance matrix: Among them, Σ 11 and Σ 22 Respectively represent the covariance matrix of the CFP mode and the covariance matrix of the OCT mode; Σ 12 is the cross-modal covariance matrix; Maximize the output feature O using the following formula cfp and O oct The correlation in the shared space is used to obtain the OCT target feature vector and the CFP target feature vector, which are expressed as follows:

8. The method according to claim 1, characterized in that The constructing of a multi-modal feature fusion module, performing feature fusion on the OCT target feature vector and the CFP target feature vector, and outputting target fusion features, includes: The OCT target feature vector and the CFP target feature vector are used as input features to initialize the sliding average variable y avg It is a zero matrix with the same size as the input features; Update the sliding average variable y avg , the formula is as follows: and avg =β·y avg +(1-β)·y t Where β is the sliding mean attenuation coefficient; y t is the input feature at the current moment; y t is the result of element-wise multiplication of the OCT target feature vector and the CFP target feature vector at time t; Output fusion feature y output =y avg ; y output As the input for the next time t+1; Select a specific time step t * As a termination, the final target fusion feature is output.

9. The method according to claim 7, characterized in that: The preset classification algorithm includes at least one of a support vector machine, a random forest, a K nearest neighbor algorithm, and a multi-layer perceptron; The loss function formula is expressed as: in, is the classification loss function; is the deep canonical correlation loss function; N is the number of training samples; C is the number of categories; y i,j is the actual result of training sample i; is the probability that the prediction result of training sample i is category j; λ is the loss weight used to balance the impact of classification loss and deep canonical correlation loss on classifier training.

10. A macular degeneration classification system constructed according to the method for constructing a macular degeneration classification system according to any one of claims 1 to 9, characterized in that: The system includes: a loose pairing module, a feature extraction module, a deep canonical correlation analysis module, a feature fusion module and a target classifier; The loose pairing module is used to obtain the OCT image and the CFP image to be classified, and use the loose pairing technology to pair the OCT image and the CFP image with the same AMD degeneration label to obtain multiple groups of OCT-CFP training sample pairs; The feature extraction module is used to extract features from the OCT-CFP training sample pair to obtain an OCT high-dimensional feature vector and a CFP high-dimensional feature vector after feature conversion based on a low-rank adaptive technology; The deep canonical correlation analysis module is used to perform nonlinear feature transformation on the OCT high-dimensional feature vector and the CFP high-dimensional feature vector to obtain the OCT target feature vector and the CFP target feature vector with maximized correlation in the shared space; The feature fusion module is used to perform feature fusion on the OCT target feature vector and the CFP target feature vector to obtain a target fusion feature; The target classifier is used to take the target fusion feature as input and output the AMD degeneration category of the OCT image and the CFP image to be classified; Among them, the AMD degeneration categories include normal, dry, wet and polypoidal choroidal vasculopathy.

Images