Glaucoma view progress prediction device and electronic apparatus

Abstract

The invention provides a glaucoma visual field progress prediction device and electronic equipment, and the method comprises the steps: obtaining a first original fundus image feature, a second original fundus image feature and primary clinical text data through preprocessing, and enabling the second original fundus image to be overturned relative to the first original fundus image; the twin bilateral difference network obtains bilateral blood vessel difference features of the first original eye fundus image features and the second original eye fundus image features, the bilateral blood vessel difference features and the first original image features are fused, and the text guide image feature enhancement module guides enhancement of image features output by the twin bilateral difference network through clinical text features. According to the glaucoma visual field progress prediction method and device, the trained prediction device is utilized to realize intelligent prediction of the glaucoma visual field progress, so that the problems of insufficient judgment accuracy, time consumption and low efficiency caused by dependence on subjective experience are avoided, and the prediction accuracy and timeliness are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a glaucoma visual field progression prediction device and electronic device. Background Technology

[0002] Glaucoma has become a major cause of irreversible blindness due to progressive damage to retinal ganglion cells and the resulting loss of visual field (VF). Early symptoms are often subtle (most patients experience no obvious discomfort), and some patients are diagnosed with moderate to severe visual impairment. Visual field progression is a key indicator for assessing disease deterioration. Clinically, doctors need to compare multiple visual field examinations to determine progression; relying on subjective experience leads to insufficient accuracy and is time-consuming and inefficient. Summary of the Invention

[0003] This application proposes a glaucoma visual field progression prediction device and electronic device, which can solve one of the problems existing in the background art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, a glaucoma visual progression prediction device is provided, comprising:

[0006] The preprocessing unit is used to perform a first preprocessing on the original fundus image to obtain a first original fundus image feature and a second original fundus image feature. The second original fundus image is flipped relative to the first original fundus image. The unit also performs a second preprocessing on the original clinical text data to obtain primary clinical text data.

[0007] The twin bilateral difference network includes: a first encoder for encoding features of a first raw fundus image, a second encoder for encoding features of a second raw fundus image, and a bilateral difference sensing module for obtaining bilateral vascular difference features from the first encoder and the second encoder. The first encoder is further configured to perform feature fusion processing on the bilateral vascular difference features during the encoding process.

[0008] The text-guided image feature enhancement module is used to extract features from the primary clinical text data to obtain text features, and the text features guide the first encoder to output enhanced image features to obtain enhanced features. The image features and the enhanced features are used to obtain the glaucoma visual field progression prediction result.

[0009] Based on the above technical solution, preprocessing yields first original fundus image features, second original fundus image features, and primary clinical text data. The second original fundus image is flipped relative to the first original fundus image. A twin bilateral difference network obtains bilateral vascular difference features between the first and second original fundus image features. The bilateral vascular difference features are then fused with the first original image features. A text-guided image feature enhancement module uses clinical text features to guide the twin bilateral difference network to output enhanced image features, resulting in enhanced features. This leads to the prediction result of glaucoma visual field progression. In this way, intelligent prediction of glaucoma visual field progression can be achieved using a trained prediction device, avoiding the problems of insufficient accuracy, time consumption, and low efficiency caused by relying on subjective experience, thus improving prediction accuracy and timeliness.

[0010] In one possible design of the first aspect, the first encoder and the second encoder adopt an LVM-Med model, which is composed of several Transformer layers connected in series. The Transformer layer includes, in sequence, a first normalization layer, a multi-head self-attention layer, a second normalization layer, and a first multilayer perceptron. The bilateral difference perception module is guided by visual question answering. The Transformer layer in the first encoder and the corresponding Transformer layer in the second encoder are both connected to the bilateral difference perception module. The bilateral difference perception module also takes the first original fundus image features as input.

[0011] In one possible design approach of the first aspect, the model parameters other than the bilateral difference perception module are frozen, and a low-rank convolutional fine-tuning term is used to fine-tune the query vector and value vector generated by the first normalization layer.

[0012] In one possible design of the first aspect, the bilateral difference sensing module includes: a bilateral difference sensing submodule, used to obtain distinguishing features by employing a cross-attention mechanism with intermediate results from the first encoder as keys and values ​​and intermediate results from the second encoder as queries;

[0013] The visual question-answering submodule includes: generating difference text based on the original fundus image and visual difference questions, encoding the difference text to obtain a visual question-answering prompt feature representation, and...

[0014] The feature fusion submodule is used to calculate the bilateral vascular difference features that represent the correlation between the distinguishing features and the visual question-answering prompt feature representations.

[0015] In one possible design approach of the first aspect, the visual question-answering submodule includes:

[0016] The LLaVa-Med model for generating differential text based on the original fundus images and visual difference issues, and,

[0017] The BioLinkBERT model is used to encode the differential text to obtain visual question-answering prompt feature representations.

[0018] The feature fusion submodule includes:

[0019] A matrix multiplication layer is used to calculate the correlation matrix between the distinguishing features and the visual question-answering prompt feature representation through matrix multiplication;

[0020] An activation function layer is used to calculate the normalized correlation graph corresponding to the correlation matrix;

[0021] Hadamard layers are used to combine the discriminative features with the normalized correlation map to obtain primary features of bilateral vascular differences; and...

[0022] The difference refinement block is used to refine the primary features of the bilateral blood vessel differences to obtain the bilateral blood vessel difference features.

[0023] In one possible design of the first aspect, the text-guided image feature enhancement module includes:

[0024] The text feature extraction module is used to extract text features from clinical text information.

[0025] A multi-scale image-text interaction module is used to guide the enhancement of image features based on the text features, thereby obtaining enhanced features, and...

[0026] A fully connected layer is used to obtain glaucoma visual field progression prediction results from the enhanced features.

[0027] In one possible design of the first aspect, the text feature extraction module includes:

[0028] The BioLinkBERT encoder is used to encode the structured clinical text information to obtain primary text features;

[0029] A second multilayer perceptron is used to obtain phrase-level embeddings with reduced-dimensional representations from the primary text features; and

[0030] An average pooling layer is used to obtain phrase embeddings from the primary features of the text.

[0031] In one possible design approach of the first aspect, the multi-scale graphic interaction module includes:

[0032] An initial convolutional layer is used to process the image features to obtain a low-dimensional representation;

[0033] A first enhancement network is used to enhance the low-dimensional representation of the image features using the phrase-level embeddings to obtain a first feature; and,

[0034] A second enhancement network is used to enhance the low-dimensional representation of the image features using the inter-phrase embeddings to obtain a second feature.

[0035] The image features, the first feature, and the second feature are all used as inputs to the fully connected layer.

[0036] In one possible design approach of the first aspect, preprocessing includes one or more combinations of the following processes: image scaling, normalization, flipping, and original feature extraction that preserves spatial location.

[0037] In a second aspect, an electronic device is provided, comprising: a processor, and a memory coupled to the processor, the memory for storing a computer program; the processor for executing the computer program stored in the memory, such that the electronic device performs the following processing:

[0038] The original fundus image is subjected to a first preprocessing to obtain the first original fundus image features and the second original fundus image features. The second original fundus image is flipped relative to the first original fundus image. The original clinical text data is subjected to a second preprocessing to obtain the primary clinical text data.

[0039] The first original fundus image features are encoded, and the second original fundus image features are encoded. Bilateral vascular difference features are obtained from the first and second original fundus image features. Feature fusion processing of the bilateral vascular difference features is performed during the encoding process.

[0040] The clinical text information is used to extract text features, which are then used to guide the encoding of image features to obtain enhanced features. Finally, the image features and the enhanced features are used to obtain the glaucoma visual field progression prediction result. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an overall method flowchart provided in the embodiments of this application;

[0043] Figure 2 This is a schematic diagram of the overall model structure provided in the embodiments of this application;

[0044] Figure 3 This is a schematic diagram of the bilateral difference perception module structure for visual questioning guidance provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of the multi-scale graphic interaction module provided in the embodiments of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0049] Before introducing the embodiments of this application, a brief description of the current stage of technical research of this application will be given:

[0050] With the continuous development of medical technology and the deepening application of deep learning in medical image analysis, artificial intelligence-assisted methods for prognostic analysis are gradually gaining attention. Deep learning technology can help doctors capture subtle structural damage and potential visual field progression features in fundus images that are difficult for humans to detect, thereby greatly improving the efficiency and accuracy of glaucoma prognostic analysis. Glaucoma visual field progression prediction uses a deep learning model to automatically predict whether future visual field progression will occur based on the patient's baseline fundus images (including fundus vascular structure information) and clinical text (such as age, intraocular pressure, corneal thickness, etc.). This can assist doctors in identifying high-risk patients in advance and developing personalized treatment plans.

[0051] Common perimetry examinations, including Standard Automated Perimetry (SAP), often require prolonged patient cooperation and high reliability. Furthermore, SAP results are frequently highly subjective, significantly hindering the analysis of visual field progression. In contrast, Color Fundus Photography (CFP) is a faster and more objective examination method. If CFP could be used to predict and analyze visual field progression in glaucoma patients, it would provide physicians with a faster and more accurate way to understand the patient's current condition and prognosis. A study from Sun Yat-sen University developed a clinically feasible deep learning system for predicting and stratifying the risk of glaucoma onset and progression based on CFP, and clinically validated its performance in an external population cohort. This demonstrated the feasibility of using deep learning algorithms to capture and learn glaucoma progression characteristics for visual field progression prediction.

[0052] To extract potential progression features, the Transformer architecture is often used as the encoder, enabling modeling from a higher global perspective. Initially applied to natural language processing, it achieved significant success, demonstrating competitive performance on a range of computer vision tasks. However, as research has progressed, some drawbacks of the Transformer have become apparent. The most significant is its weak inductive bias, which makes convergence during training difficult. Therefore, the Transformer typically requires large-scale datasets to achieve better performance, while medical image datasets are usually small, meaning that directly applying the Transformer may not yield better predictions than convolutional neural networks. To address this, researchers have combined convolutional neural networks with the Transformer to achieve better prediction results. Hu et al. proposed using polar convolutional neural networks to extract low-level features, allowing the model to converge quickly even with smaller datasets. However, due to the relatively simple labels on this dataset, the model's predictions only indicate whether the patient has the disease, offering little help in predicting visual field progression in glaucoma patients.

[0053] Because medical image datasets are typically small, to achieve better predictive performance, pre-trained large models are often used to learn more latent features and leverage their generalization capabilities to solve various problems, such as encoding textual information to extract various features. Common large medical models can process both image and clinical text information. In prognostic analysis, VisionFM, developed by the Chinese University of Hong Kong, serves as a multimodal, multi-task visual foundational model for general practice artificial intelligence. It uses data covering a wide range of eye diseases, various patterns, imaging devices, and demographics for learning. It not only demonstrates predictive accuracy equivalent to that of a mid-level physician but also exhibits high interpretability. This highlights the advantages of artificial intelligence over human intervention in extracting, learning, and utilizing the complex structural and indicative data features of eye diseases.

[0054] This application first uses Convolutional Low-Rank Adaptation (ConvLoRA) to fine-tune the Transformer block of the large-scale self-supervised learning visual model LVM-Med for medical images. Then, it forms a Siamese bilateral difference network with a Bilateral Difference Awareness Module (BDAM) guided by Visual Question Answering (VQA) to extract bilateral vascular difference features from the input image. Next, a Text-Guided Image Feature Enhancement (TG-IFE) module uses clinical text information to obtain phrase-level embeddings (PE) and inter-phrase embeddings (IE), guiding image feature enhancement at both fine-grained and coarse-grained levels. This achieves deep fusion of multimodal information, aiming to obtain better prediction results for glaucoma visual field progression. For example... Figure 1 As shown, the specific steps are as follows:

[0055] S1. Obtain existing fundus images and clinical information of glaucoma patients;

[0056] S2. The fundus images obtained in step S1 are preprocessed and combined with clinical information and progress labels to construct training, validation and test sets;

[0057] S3. A fundus image feature extraction module based on fine-tuned LVM-Med and a differential feature enhancement module based on a VQA large model are constructed together to form a Siamese bilateral differential network; a text-guided image feature enhancement module based on a medical large language model combined with multi-scale deep separable convolution and cross-modal attention mechanism is constructed; finally, a primary model for predicting glaucoma visual field progression is constructed, consisting of a Siamese bilateral differential network, a text-guided image feature enhancement module, and a prediction result output module.

[0058] S4. Using the training dataset obtained in step S2, train the primary model for predicting glaucoma visual field progression constructed in step S3 to obtain the glaucoma visual field progression prediction model.

[0059] The image preprocessing described in step S2 specifically includes the following steps:

[0060] To match the image format requirements of the LVM-Med data input, fundus images of all glaucoma patients in the dataset were preprocessed, mainly including image scaling and normalization. After scaling all data images to a uniform resolution of 224x224, the images were normalized using the standard score Z-score method. The normalized image data was then used for model training and performance evaluation using a three-fold cross-validation method.

[0061] To meet the input requirements of the Transformer encoder, the preprocessed image is flipped vertically, and the image data is then... Convert into initial features suitable for the Transformer encoder (Dimensions are B×196×768) where t=1 corresponds to the original preprocessed image, t=2 corresponds to the image after flipping, and B is the number of channels. The formula is as follows:

[0062] in, Image block embedding (dividing a 224×224 image into 14×14 16×16 pixel blocks). Location encoding (preserving the spatial location information of image blocks).

[0063] Step S3 includes the following steps:

[0064] The glaucoma visual field progression prediction model consists of a twin bilateral difference network, a text-guided image feature enhancement module, and a prediction result output module connected in series, as follows: Figure 2 As shown;

[0065] The twin bilateral difference network consists of a fundus image feature extraction module based on fine-tuned LVM-Med and a difference feature guided enhancement module based on VQA large model (VQA-BDAM). The twin bilateral difference network is used to extract the difference features of the upper and lower sides of the blood vessels in the fundus image for subsequent visual field progression prediction.

[0066] The TG-IFE module is based on a medical large language model combined with multi-scale deep separable convolution and cross-modal attention mechanism. TG-IFE is used to perform text-guided enhancement on extracted vascular differential features using clinical information for subsequent visual field progression prediction.

[0067] Prediction result output module: Uses a fully connected layer C to output the prediction results after extracting and guided enhancement of the feature encoding;

[0068] The aforementioned twin bilateral difference network specifically includes the following steps:

[0069] The constructed twin bilateral difference network consists of twelve layers ( Figure 2 The network consists of a 12-layer (×12) fine-tuned LVM-Med Transformer block and VQA-BDAM. Each Transformer block comprises a first normalization layer, a multi-head self-attention layer, a second normalization layer, and a multilayer perceptron. The twelve Transformer blocks are stacked, with the output of each layer serving as the input to the next. Furthermore, each layer incorporates VQA-BDAM, allowing for multi-level and multi-scale extraction of asymmetric information. The input to VQA-BDAM is the original image input and the intermediate results from both sides of the Transformer blocks, yielding bilateral vascular differential features. The features are fed back into the encoder and combined with the unique features of different images to produce the fused features at each stage. The twelve-layer progressive feature refinement process of the Siamese bilateral dissimilarity network can be generally described by the following formula:

[0070] in Representative level (1) ), t=1 corresponds to the original preprocessed image, and t=2 corresponds to the image after performing the up-and-down flip. MHSA stands for Multi-head Self-Attention, LN represents Layer Normalization, and MLP represents Multilayer Perception. This represents the bilateral vascular differential characteristics derived from VQA-BDAM. Representing the The final output of the layer is the corresponding fusion difference feature. represent The first layer input fusion differential features and vascular bilateral differential features Layer refinement and stacking features (when) hour, for ). It is the first The key vector corresponding to the layer, and Representing the first The query vector and value vector of the layer after ConvLoRA fine-tuning.

[0071] Regarding fine-tuning the Transformer encoder using ConvLoRA: We first freeze all parameters except for the bilateral difference perception module guided by visual question answering, TG-IFE, and the classification head. Then, we apply ConvLoRA to each Transformer encoder to effectively fine-tune the model. The query vector generated after the first normalization layer and the linear layer for each Transformer block ( ) and value vector ( Fine-tuning is performed by learning specific features related to bilateral differences using low-rank convolutional kernels, as shown in the following formula:

[0072] in This represents three different linear layers. This is a low-rank convolution fine-tuning term that modifies the Q and V vectors with a small number of parameters, focusing attention on the bilateral difference region. It is the first The key vector corresponding to the layer, and Representing the first The query vector and value vector of the layer after ConvLoRA fine-tuning.

[0073] The VQA-BDAM method yields bilateral vascular differential characteristics. The steps are as follows: Figure 3 As shown, VQA-BDAM consists of BDAM, VQA, and a feature fusion submodule. This embodiment proposes a bilateral difference perception module (BDAM) to capture bilateral vascular differences, and introduces visual descriptions of bilateral vascular differences generated by VQA as information cues to guide the model's attention to vascular-related features, thereby enhancing its ability to capture vascular asymmetry and mitigating the risks of background noise and improper attention allocation. The final output vascular difference features participate in the feature refinement process of each encoder layer.

[0074] First, bilateral difference features of the image are obtained through the BDAM submodule; taking the first layer as an example, the features are obtained from... of and and from of The input is cross-fused into BDAM, employing a traditional cross-attention mechanism, where... As a query As the key, As the value, we export the features representing the common information between the twin image pairs. We believe that represents the features, and by subtracting the common features, we obtain their difference features. This process can be expressed by the formula:

[0075] where is a non-linear activation function, represents the common features between the original and flipped images, represents the difference features between the two.

[0076] Secondly, we generate guided enhanced visual question answering prompt features through the sub-module VQA; we input the fundus image of each patient and the difference question "What is the vascular difference between the upper and lower parts of this fundus image?" into the VQA model LLaVa-Med to generate bilateral difference text as VQA prompts, which are then encoded by the large language model BioLinkBERT. It can be expressed as:

[0077] where the visual question answering prompt features are expressed as , represents the question, VQA represents the LLaVa-Med model, represents the pre-trained BioLinkBERT. And represents a difference refinement block with two linear layers, which can map text features to the image feature space for effective fusion through dimensionality reduction and dimensionality increase operations, while reducing the model parameters and complexity. The formula is as follows:

[0078] where, represents matrix multiplication, ∈ and ∈ (where d is the embedding dimension, r <d = 768) are the dimensionality reduction matrix and the dimensionality increase matrix respectively.

[0079] Finally, we fuse the two through the feature fusion sub-module; we first calculate the correlation matrix between the extracted bilateral difference features of the image and the VQA prompts through matrix multiplication, and then apply the softmax function to obtain a normalized correlation map representing the similarity between the difference features from the imaging modality and the VQA prompts. Next, we multiply the correlation matrix with The features are combined to guide image difference features based on their similarity. Finally, the features are refined through difference refinement blocks. This process can be described as follows:

[0080] in, Represents matrix multiplication. This represents the Hadamard product. It is a non-linear activation function. This represents dimensionality increase / decrease operations performed through a differential refinement block with two linear layers. This represents the bilateral difference features after text optimization. This represents the final bilateral vascular differential characteristics derived from VQA-BDAM.

[0081] The specific steps of the text-guided image feature enhancement module (TG-IFE) are as follows:

[0082] The constructed TG-IFE is a cascaded structure of a text feature extraction module and a multi-scale image-text interaction module. The text feature extraction module is mainly a BioLinkBERT encoder. The multi-scale image-text interaction module uses the phrase-level embedding (PE) and inter-phrase embedding (IE) output by the text feature extraction module to integrate image-text association, guiding image feature extraction at both fine-grained and coarse-grained levels. Finally, it combines the features output by the Siamese bilateral difference network to obtain fused features.

[0083] The text feature extraction module operates as follows: Collected clinical data from glaucoma patients includes age, gender, intraocular pressure, central corneal thickness, visual field progression prediction, glaucoma diagnosis type, and optical coherence tomography (OCT) retinal nerve fiber layer thickness. This textual information is first converted into structured phrases, and then feature-encoded using a BioLinkBERT encoder. Their dimensionality is then reduced using an MLP to obtain phrase-level embeddings (PEs), followed by average pooling to form inter-phrase embeddings (IEs). PEs reflect the characteristics of various influencing factors in the patient's clinical context and are typically associated with fine-grained image semantics, while IEs capture the overall clinical context and are implicitly associated with coarse-grained image features.

[0084] The steps of the multi-scale graphic-text interaction module are as follows: Figure 4 As shown, the input Feature representations are obtained by processing with a twin bilateral difference network. , represented as .enter The corresponding output This is a byproduct of this embodiment. Dimensionality is reduced using 1×1 convolution to obtain bilateral difference features of the image. .

[0085] Based on this, we process its branches. First, we use a 3×3 depthwise separable convolution (DSC) to obtain the branches containing low-level features. Secondly, we execute and The matrix product between the terms is used to obtain a phrase-based similarity matrix, which is then normalized using softmax and applied with... The Hadamard product is used to ultimately obtain phrase-enhanced image attention features. .

[0086] The 5×5 DSC contains a larger receptive field, which can be used to extract high-level semantic information and obtain contextual features. Ultimately combined and A similar approach was used to obtain phrase context-aware image attention features. .

[0087] in , and represents standard convolution, depthwise convolution, and pointwise convolution, respectively, with each convolution having a kernel size of i×i. represents matrix multiplication, and represents the Hadamard product. It is batch normalization with the ReLU activation function.

[0088] The steps of the prediction output module are as follows:

[0089] Output of TG-IFE , and These features are then connected to form more representative fusion features. This enhances image-text fusion by aligning semantic features across multiple scales. The prediction result is obtained using a fully connected layer, as shown in the following formula:

[0090] This indicates a concatenation operation along the channel dimension. Indicates a fully connected layer. This indicates the probability of predicting progress.

[0091] The training described in step S4 specifically includes the following steps:

[0092] The pre-trained medical basic model LVM-Med was loaded as the backbone network, and its backbone parameters were frozen; only the parameters of the VQA-BDAM, TG-IFE modules and the classification head were unfrozen, and the parameters of the Transformer layer were efficiently fine-tuned using the ConvLoRA technique.

[0093] To address the task class imbalance problem, a weighted cross-entropy loss function is used:

[0094] Where y is the actual progress label (0 or 1). It is the probability that the model predicts whether the field of vision will advance. and The weights are calculated using the following formula:

[0095] in and This represents the sample size for actual no progress and actual progress. This represents the total number of samples.

[0096] The Adam optimizer was used during training, with an initial learning rate of 8e-4, 200 training epochs, and a batch size of 8. The CosineAnnealingLR learning rate scheduler was used to achieve cosine decay of the learning rate as the training progressed.

[0097] Experimental verification

[0098] To evaluate the effectiveness of the prediction model proposed in this invention, this embodiment was compared with the following comparative methods:

[0099] BiomedCLIP: This model achieves high generalization ability by learning general cross-modal medical semantic representations through pre-training on large-scale biomedical and text pairs, enabling multi-task adaptation of a single pre-training. Its image encoder is based on an improvement of Vision Transformer, adding a "local medical feature enhancement module" to improve the capture of medical details. Its text encoder is based on an improvement of BioBERT, focusing on optimizing the understanding of "medical terminology, abbreviations, and complex sentence structures."

[0100] BLIP: Its encoders also include a Vision Transformer-based image encoder and a BioBERT-based text encoder, which complete the task through a three-part architecture of image encoder + text encoder + cross-modal decoder.

[0101] ReplykNet: It uses dynamic fusion of large and small kernels to perform convolution to extract image features and balances local and global features.

[0102] MOGANET introduces a multi-level feature aggregation module based on human vision. It encapsulates local perception and contextual aggregation into a unified spatial aggregation block. It uses gated aggregation to capture multi-level contextual features.

[0103] MambaVision: It redesigns the core Mamba formula, proves the feasibility of integrating Vision Transformer and Mamba, and equips the last few layers of the Mamba architecture with self-attention blocks, which can significantly improve its ability to capture long-range spatial dependencies.

[0104] MedVit: It is an efficient and highly robust CNN-Transformer hybrid model that uses efficient convolutional operations to construct a self-attention mechanism to reduce the impact of the self-attention mechanism on information in different representation subspaces. The overhead caused by level complexity.

[0105] Accuracy, sensitivity, specificity, and AUC (area under the ROC curve) were used to evaluate the model's performance in the experiment. The formulas for calculating each metric are shown below:

[0106] The core dimension of accuracy is the ability to judge whether the overall sample is correct or not, without distinguishing between positive and negative samples.

[0107] Sensitivity represents how many of the progress samples are correctly predicted as progress.

[0108] Specificity represents the proportion of samples that did not progress but were correctly predicted by the model. TP, FN, FP, and TN represent true positive, false negative, false positive, and true negative, respectively.

[0109] AUC (Area Under the Curve) is an important metric for evaluating binary classification tasks. The closer the AUC value is to 1, the better the classification performance of the classifier.

[0110] The data used in this experiment was the GRAPE dataset. This embodiment and the comparison method were evaluated on the same test set, and the experimental results are shown in Table 1.

[0111] Table 1: Comparison of prediction performance between this embodiment and the comparative method

[0112] In a comparison of methods for predicting glaucoma progression, BLIP, MambaVision, BiomedCLIP, and MedVit performed poorly in terms of sensitivity. The method proposed in this embodiment outperforms all other methods in terms of accuracy (83.02%), AUC (77.22%), sensitivity (75.00%), and specificity (84.44%), fully meeting the clinical need for "early detection and accurate diagnosis." The table shows that the sensitivity of the compared methods is generally low, while this embodiment combines local and global features at each level, significantly reducing the possibility of missed diagnoses. This indicates that this embodiment can more effectively extract features related to glaucoma progression, thereby obtaining more accurate prediction results.

[0113] In summary, this embodiment proposes a method for predicting the progression of glaucoma visual field based on fine-tuning of a large medical model. Experimental results show that this embodiment can effectively extract visual field loss features from fundus images of glaucoma patients, achieving more accurate prognostic progression prediction and analysis.

[0114] This application also provides an electronic device, including: a processor, and a memory coupled to the processor, the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory, such that the electronic device performs the following processing:

[0115] The original fundus image is subjected to a first preprocessing to obtain the first original fundus image features and the second original fundus image features. The second original fundus image is flipped relative to the first original fundus image. The original clinical text data is subjected to a second preprocessing to obtain the primary clinical text data.

[0116] The first original fundus image features are encoded, and the second original fundus image features are encoded. Bilateral vascular difference features are obtained from the first and second original fundus image features. Feature fusion processing of the bilateral vascular difference features is performed during the encoding process.

[0117] The clinical text information is used to extract text features, which are then used to guide the encoding of image features to obtain enhanced features. Finally, the image features and the enhanced features are used to obtain the glaucoma visual field progression prediction result.

[0118] Electronic devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. These electronic devices may include, but are not limited to, processors and memory.

[0119] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the device via various interfaces and lines.

[0120] The memory can be used to store the computer program, and the processor implements various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0121] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0122] This application also provides a storage medium, which is a computer-readable storage medium. The computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0123] This application also provides a computer program product, including: a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform any of the above possible implementation methods.

[0124] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A device for predicting the progression of glaucoma visual field, characterized in that, include: The preprocessing unit is used to perform a first preprocessing on the original fundus image to obtain a first original fundus image feature and a second original fundus image feature. The second original fundus image is flipped relative to the first original fundus image. The unit also performs a second preprocessing on the original clinical text data to obtain primary clinical text data. The twin bilateral difference network includes: a first encoder for encoding features of a first raw fundus image, a second encoder for encoding features of a second raw fundus image, and a bilateral difference sensing module for obtaining bilateral vascular difference features from the first encoder and the second encoder. The first encoder is further configured to perform feature fusion processing on the bilateral vascular difference features during the encoding process. The text-guided image feature enhancement module is used to extract features from the primary clinical text data to obtain text features, and the text features guide the first encoder to output enhanced image features to obtain enhanced features. The image features and the enhanced features are used to obtain the glaucoma visual field progression prediction result.

2. The glaucoma visual progression prediction device as described in claim 1, characterized in that, The first encoder and the second encoder adopt the LVM-Med model, which is composed of several Transformer layers connected in series. The Transformer layer includes, in sequence: a first normalization layer, a multi-head self-attention layer, a second normalization layer, and a first multilayer perceptron. The bilateral difference perception module is guided by visual question answering. The Transformer layer in the first encoder and the corresponding Transformer layer in the second encoder are both connected to the bilateral difference perception module. The bilateral difference perception module also uses the first original fundus image features as input.

3. The glaucoma visual progression prediction device as described in claim 2, characterized in that, Freeze the model parameters other than the bilateral difference perception module, and use a low-rank convolution fine-tuning term to fine-tune the query vector and value vector generated by the first normalization layer.

4. The glaucoma visual progression prediction device as described in claim 2, characterized in that, The bilateral difference perception module includes a bilateral difference perception submodule, which uses intermediate results from the first encoder as keys and values ​​and intermediate results from the second encoder as queries, and employs a cross-attention mechanism to obtain distinguishing features. The visual question-answering submodule includes: generating difference text based on the original fundus image and visual difference questions, encoding the difference text to obtain a visual question-answering prompt feature representation, and... The feature fusion submodule is used to calculate the bilateral vascular difference features that represent the correlation between the distinguishing features and the visual question-answering prompt feature representations.

5. The glaucoma visual progression prediction device as described in claim 4, characterized in that, The visual question-answering submodule includes: The LLaVa-Med model for generating differential text based on the original fundus images and visual difference issues, and, The BioLinkBERT model is used to encode the differential text to obtain visual question-answering prompt feature representations. The feature fusion submodule includes: A matrix multiplication layer is used to calculate the correlation matrix between the distinguishing features and the visual question-answering prompt feature representation through matrix multiplication; An activation function layer is used to calculate the normalized correlation graph corresponding to the correlation matrix; Hadamard layers are used to combine the discriminative features with the normalized correlation map to obtain primary features of bilateral vascular differences; and... The difference refinement block is used to refine the primary features of the bilateral blood vessel differences to obtain the bilateral blood vessel difference features.

6. The glaucoma visual progression prediction device as described in claim 1, characterized in that, The text-guided image feature enhancement module includes: The text feature extraction module is used to extract text features from clinical text information. A multi-scale image-text interaction module is used to guide the enhancement of image features based on the text features, thereby obtaining enhanced features, and... A fully connected layer is used to obtain glaucoma visual field progression prediction results from the enhanced features.

7. The glaucoma visual progression prediction device as described in claim 6, characterized in that, The text feature extraction module includes: The BioLinkBERT encoder is used to encode the structured clinical text information to obtain primary text features; A second multilayer perceptron is used to obtain phrase-level embeddings with reduced-dimensional representations from the primary text features; and An average pooling layer is used to obtain phrase embeddings from the primary features of the text.

8. The glaucoma visual progression prediction device as described in claim 7, characterized in that, The multi-scale graphic-text interaction module includes: An initial convolutional layer is used to process the image features to obtain a low-dimensional representation; A first enhancement network is used to enhance the low-dimensional representation of the image features using the phrase-level embeddings to obtain a first feature; and, A second enhancement network is used to enhance the low-dimensional representation of the image features using the inter-phrase embeddings to obtain a second feature. The image features, the first feature, and the second feature are all used as inputs to the fully connected layer.

9. The glaucoma visual progression prediction device as described in claim 1, characterized in that, Preprocessing includes one or more of the following processes in combination: image scaling, normalization, flipping, and original feature extraction that preserves spatial location.

10. An electronic device, characterized in that, The electronic device includes: a processor, and a memory coupled to the processor. The memory is used to store computer programs; and The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the following processes: The original fundus image is subjected to a first preprocessing to obtain the first original fundus image features and the second original fundus image features. The second original fundus image is flipped relative to the first original fundus image. The original clinical text data is subjected to a second preprocessing to obtain the primary clinical text data. The first original fundus image features are encoded, and the second original fundus image features are encoded. Bilateral vascular difference features are obtained from the first and second original fundus image features. Feature fusion processing of the bilateral vascular difference features is performed during the encoding process. The primary clinical text data is used to extract text features, which are then used to guide the encoding of image features to obtain enhanced features. Finally, the image features and the enhanced features are used to predict the progression of glaucoma visual field.

Images