Macular hole postoperative vision prediction method based on multi-modal feature fusion and electronic equipment

Abstract

The invention relates to a macular pore postoperative vision prediction method based on multi-modal feature fusion and electronic equipment. The prediction model integrates a feature extraction backbone network, a retina layering attention module, a crack hole positioning and enhancing module and a multi-modal fusion module. The method comprises the following steps: extracting multi-scale features of a preoperative OCT image through a backbone network; a retina layered attention module is utilized, and a layered mask guide model generated based on a segmentation network is focused on an ellipsoid zone, an external membrane and other key structures; meanwhile, a crack hole positioning and enhancing module is utilized to adaptively reinforce crack hole area features according to a positioning result of the target detection network; and finally, performing deep fusion on the processed image features and clinical parameters by adopting an attention mechanism through a multi-modal fusion module, and outputting an accurate postoperative vision prediction value. According to the method, automatic and high-precision personalized prediction is realized, and the defects of high subjectivity, low efficiency and insufficient information fusion of a traditional method are overcome.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and artificial intelligence technology, and in particular to a method and electronic device for predicting postoperative visual acuity after macular hole surgery based on multimodal feature fusion. Background Technology

[0002] Macular hole (MH) is a serious blinding fundus disease. Although surgical treatment can achieve a high rate of anatomical closure, the degree of postoperative visual recovery varies significantly. Accurate prediction of postoperative visual acuity is crucial for doctor-patient communication, surgical planning decisions, and the management of patient expectations.

[0003] Currently, postoperative visual acuity prediction mainly relies on physicians' manual measurement and analysis of morphological parameters in preoperative OCT images, such as the minimum diameter of the macular hole, the base diameter, the macular hole index (MHI), and the integrity of the ellipsoidal zone (EZ) and external endothelial membrane (ELM). However, this method has significant limitations: First, manual measurement is inefficient and introduces subjective bias and inter-observer variability, making it difficult to meet the needs of large-scale clinical applications; second, these pre-defined morphological parameters cannot comprehensively and deeply characterize the complex textures and structural features in OCT images that are related to prognosis; finally, this method is difficult to effectively integrate patients' clinical parameters (such as age, disease duration, etc.) to achieve personalized comprehensive prediction.

[0004] With the development of artificial intelligence technology, existing research has attempted to use machine learning or deep learning models for prediction. However, these existing technical solutions have one or more of the following drawbacks: Feature extraction relies on manual intervention: Most machine learning models still rely on manually measured parameters as input, failing to achieve end-to-end automated feature learning.

[0005] Insufficient attention to key areas: When processing OCT images, existing deep learning models lack specific attention mechanisms for lesion areas (holes) and key retinal structures (EZ, ELM layers), making it difficult for the models to focus on the subtle features most important for prognosis from complex backgrounds.

[0006] Shallow multimodal information fusion: It fails to achieve deep fusion of OCT image deep features and clinical parameters at the feature level. It is usually just simple feature splicing or decision-level fusion, and fails to fully explore the complementarity and correlation between multi-source information.

[0007] Insufficient model generalization ability: Due to the limited scale and uneven quality of training data, the model is prone to overfitting, and its prediction performance drops significantly when applied to data collected from different devices and medical institutions.

[0008] Therefore, there is an urgent need in this field for a new technological solution that can automatically, accurately, and personally predict postoperative visual acuity after macular hole surgery. Summary of the Invention

[0009] The purpose of this invention is to propose a method for predicting postoperative visual acuity after macular hole surgery based on multimodal feature fusion, so as to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: A method for predicting postoperative visual acuity after macular hole surgery based on multimodal feature fusion includes the following steps: Acquire preoperative OCT images and clinical parameters of the patient using optical coherence tomography (OCT). The preoperative OCT images and clinical parameters are input into the trained prediction model to obtain the postoperative visual acuity prediction results; The prediction model processes the preoperative OCT images and clinical parameters in the following manner: Multi-scale image features are extracted from the preoperative OCT images using a feature extraction backbone network; Using a retinal hierarchical attention module, the multi-scale image features are weighted and adjusted based on the hierarchical mask generated by the retinal segmentation network to focus on specific retinal structural regions. Using a crack localization and enhancement module, crack regions are located based on a target detection network, and the crack region features in the multi-scale image features are adaptively enhanced. Using a multimodal fusion module, the image features after weight adjustment and adaptive enhancement are deeply fused with the embedded and encoded clinical parameters to generate fused features; Based on the fusion features, the postoperative visual acuity prediction results are output through a regressor.

[0011] Optionally, the step utilizes a retinal hierarchical attention module to perform weight adjustment on the multi-scale image features based on the hierarchical mask generated by the retinal segmentation network, including: Obtain a layered mask generated by a retinal segmentation network based on the preoperative OCT image, the layered mask containing a probability map of key retinal layers of the ellipsoidal band and the outer membrane; The hierarchical mask and the image features output by the feature extraction backbone network are input into the attention subnetwork to generate a spatial weight map. By introducing learnable background suppression parameters and retinal enhancement parameters, differentiated weight suppression and enhancement are implemented for background regions and key retinal structure regions based on the spatial weight map. The spatial weight map, after differential weight suppression and enhancement, is used to weight the image features output by the feature extraction backbone network to achieve feature reconstruction.

[0012] Optionally, the step utilizes a crack localization and enhancement module to locate the crack region based on a target detection network, and adaptively enhances the crack region features in the multi-scale image features, including: Obtain the coordinates of the tear boundary box output by the target detection network based on the preoperative OCT image; Based on the coordinates of the crack bounding box, the crack region features are cropped from the image features output by the feature extraction backbone network through ROI alignment operation; The features of the crack region are processed using a parameterized convolutional network to generate an attention weight map of the crack region; The attention weight map of the hole region is restored to the full image size through interpolation, and the fusion strength between it and the image features output by the feature extraction backbone network is controlled by a learnable global enhancement factor, thereby achieving feature enhancement of the hole region.

[0013] Optionally, the step utilizes a multimodal fusion module to deeply fuse the weighted and adaptively enhanced image features with the embedded and encoded clinical parameters, including: The clinical parameters are converted into high-dimensional feature vectors through an embedding layer; The high-dimensional feature vector is dimensionally aligned with the image depth features after weight adjustment and adaptive enhancement. A multi-head self-attention mechanism is adopted to perform bidirectional interaction and fusion on the dimension-aligned high-dimensional feature vector and the image depth features, and output the fused features.

[0014] Optionally, the trained prediction model is obtained through a pre-training and fine-tuning strategy, wherein the fine-tuning strategy includes: The feature extraction backbone network is pre-trained on a general image recognition dataset; The overall prediction model was fine-tuned end-to-end on a medical dataset containing preoperative OCT images, clinical parameters, and corresponding postoperative visual acuity labels of patients with macular holes.

[0015] Optionally, before inputting the preoperative OCT images and clinical parameters into the trained prediction model, a data preprocessing step is also included, wherein the preprocessing includes one or more of reflection filling, center cropping, and random affine transformation.

[0016] Optionally, the retinal segmentation network is a U-Net3+ network.

[0017] Optionally, the target detection network is a YOLOv8 network.

[0018] Optionally, the feature extraction backbone network is a ConvNeXt network.

[0019] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for predicting postoperative visual acuity of macular hole based on multimodal feature fusion.

[0020] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described method.

[0021] Compared with the prior art, the present invention has the following significant advantages: High prediction accuracy: By automatically learning high-order features through an end-to-end deep convolutional neural network, it overcomes the bottleneck of traditional models' inability to characterize complex nonlinear relationships among multiple factors. An innovative attention mechanism ensures the model focuses on the most predictive key regions, thereby significantly improving prediction accuracy.

[0022] High degree of automation and objectivity: It completely avoids the tedious and subjective manual parameter measurement process, realizes fully automated processing from image input to result output, ensures the consistency and repeatability of feature extraction, and greatly improves the efficiency of clinical work.

[0023] Sufficient feature learning depth: Through the retinal layered attention and tear region enhancement module, the model can adaptively focus on the subtle features of key areas such as the ellipsoid zone, external membrane and tear boundary, capturing deep semantic information that is difficult for the human eye to recognize.

[0024] Deep information fusion: By utilizing a deep fusion module based on multi-head self-attention, bidirectional interaction and complementarity between image depth features and clinical parameters at the semantic level are achieved, providing a more comprehensive and accurate data foundation for prediction.

[0025] Good generalization and robustness: The pre-training and fine-tuning strategies and various data augmentation techniques adopted effectively improve the model's adaptability to OCT images from different sources and devices, ensuring stability and reliability in real and varied clinical environments. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the multi-stage retinal attention mechanism.

[0027] Figure 2 This is a schematic diagram of the crack feature enhancement module.

[0028] Figure 3 This is a schematic diagram of multi-scale feature fusion.

[0029] Figure 4 This is a schematic diagram of multimodal feature fusion.

[0030] Figure 5 This is a simplified architecture diagram of a method for predicting postoperative visual acuity after macular hole surgery based on multimodal feature fusion.

[0031] Figure 6 This is an architecture diagram of a method for predicting postoperative visual acuity after macular hole surgery based on multimodal feature fusion.

[0032] Figure 7 This is a schematic diagram of retinal segmentation.

[0033] Figure 8 This is a schematic diagram for crack detection.

[0034] Figure 9 This is a schematic diagram of a retinal attention heatmap.

[0035] Figure 10 This is a schematic diagram of the thermal imaging of crack enhancement.

[0036] Figure 11 This is a schematic diagram illustrating the combined effects of retinal attention and retinal tear enhancement. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Furthermore, elements in this invention are referred to as being "fixed to" or "set on" another element, which may be directly on the other element or may also include an intervening element. When an element is considered to be "connected" to another element, it may be directly connected to the other element or may also include an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementations.

[0039] To enable those skilled in the art to better understand the technical solution of this application, the following technical information is provided: Macular hole (MH) Definition: A macular hole is a complete or partial loss of the neuroepithelial layer of the macula in the retina of the eye, and is a common blinding fundus disease. The macula is the area of ​​the retina with the sharpest central vision, responsible for fine vision and central visual acuity; therefore, a macular hole can significantly affect a patient's daily functions such as reading, driving, and recognizing faces.

[0040] Etiology and pathogenesis: Idiopathic: The most common type, more common in elderly women, and associated with posterior vitreous cortex contraction and tangential traction on the retina.

[0041] Secondary: Can be caused by eye trauma, high myopia, diabetic macular edema, epiretinal membrane, etc.

[0042] Staging: The Gass staging system (stages I-IV) is used, ranging from foveal detachment (stage I) to full-thickness tear formation accompanied by posterior vitreous detachment (stage IV).

[0043] Clinical symptoms: Decreased central vision, distorted vision (straight lines are seen as curves), central scotoma, and abnormal color vision.

[0044] treat: Surgical treatment: Pars Plana Vitrectomy (PPV) combined with internal limiting membrane peeling is the standard procedure. During the procedure, gas or silicone oil is commonly used for intraocular tamponade. Patients are required to maintain a downward-facing position post-operatively to promote tear closure.

[0045] Prognosis: The success rate of surgical anatomy can reach over 90%, but the degree of functional recovery (visual improvement) varies from person to person and depends on a variety of factors such as the size of the tear, the duration of the disease, preoperative visual acuity, and the degree of damage to photoreceptor cells.

[0046] 2. Optical Coherence Tomography (OCT) Definition: OCT is a non-invasive, high-resolution ophthalmic imaging technique, known as "optical biopsy," which enables cross-sectional microscopic imaging of retinal structures.

[0047] Working principle: It employs the principle of low-coherence interferometry, similar to ultrasound, but uses light waves instead of sound waves.

[0048] An image is constructed by measuring the time delay and intensity of reflected or backscattered light.

[0049] Resolution: Axial resolution can reach 3-7 micrometers, far exceeding traditional ultrasound examination, and can clearly distinguish the structure of each layer of the retina.

[0050] Applications in macular holes: Diagnosis and staging: OCT is the gold standard for diagnosing MH, and can clearly show the shape and size of the hole, whether there is a tegmental membrane, and whether it is accompanied by vitreous traction or anterior macular membrane.

[0051] Preoperative assessment: Surgical prognosis is predicted by measuring a series of quantitative morphological parameters, which are the core indicators automatically analyzed by the model of this invention, including: Minimum Hole Diameter (MHD) Base Diameter (BD) Hole Height Macular Hole Index (MHI) = Height / Base Diameter Traction Hole Index (THI) = Height / Minimum Diameter Photoreceptor integrity: The degree of rupture and postoperative repair of the ellipsoid zone (EZ) and external limiting membrane (ELM) are the most important biological markers for visual prognosis.

[0052] Postoperative follow-up: assess whether the retinal tear has closed (classified as U-shaped closure, V-shaped closure, W-shaped closure, or flat closure), the recovery of the retinal layer structure, and monitor for complications.

[0053] Technological advancements: Spectral domain OCT (SD-OCT): Faster scanning speed and higher resolution, it has become the mainstream.

[0054] Sweep-source OCT (SS-OCT): Further improved speed, deeper imaging, and can be used for choroidal imaging.

[0055] OCT angiography (OCTA): It can display retinal and choroidal blood flow information without contrast agents, providing additional modal information for this invention.

[0056] 3. This paper summarizes the relevant indicators in OCT images that affect postoperative visual acuity prediction:

[0057] Currently, traditional models struggle to characterize the complex nonlinear relationships between multiple factors, leading to a bottleneck in prediction accuracy. 1. Existing machine learning research has improved performance by incorporating multiple parameters (7 OCT parameters + 3 clinical features), but these models are essentially shallow machine learning models based on handcrafted features. Such models struggle to effectively learn and express the complex nonlinear interactions and high-order correlations among multiple factors in macular hole prognosis (such as hole morphology, disease course, and microstructural integrity). Their predictive ability cannot automatically learn the optimal feature representation from the raw data through multi-layer nonlinear transformations, unlike deep learning.

[0058] 2. Feature extraction relies on manual intervention, is highly subjective and inefficient, and cannot capture deep semantic features: Researchers must manually measure the minimum diameter of the retinal tear, the base diameter, and calculate indices such as MHI and THI in OCT images. This process not only introduces significant subjective bias (different operators yield different results) but is also inefficient and unsuitable for large-scale clinical screening and preoperative assessment. More importantly, this method can only capture a limited set of pre-defined morphological parameters, failing to uncover deeper texture and structural features in the images. This results in the loss of a large amount of valuable diagnostic information, introducing subjective bias and inter-investigator variability, and failing to automatically and intelligently extract deep semantic features strongly correlated with prognosis. For example, the integrity of retinal structures (such as the ellipsoidal zone EZ and the external retinal membrane ELM) is crucial for prognosis, but existing methods cannot allow models to autonomously and discriminatively focus on these key areas. They are either represented by a few simple parameters measured manually or are buried in a large amount of irrelevant background information during image processing. The lack of a mechanism that can automatically focus on key retinal structures and suppress background noise is a fundamental limitation of current technology.

[0059] 3. Existing methods do not pay enough attention to the lesion area and have difficulty capturing subtle local features: Existing models lack attention to lesion areas and retinal layering when processing images. They cannot dynamically and adaptively focus on the most prognostic parts of the image (such as the morphology of the tear edge and the thickness of specific retinal layers). The models treat all image regions "equally," resulting in insufficient feature learning in key areas and excessive sensitivity to interference from non-key areas (such as the background), thus limiting their ability to learn features from complex images.

[0060] 4. The problem of shallow multimodal information fusion failing to achieve feature-level deep fusion: Most deep learning research uses only a single data source, either OCT images or limited clinical parameters. For example, Obata's model used only OCT images without incorporating any clinical data, resulting in extremely low accuracy (46%). Even when multimodal data was used, it was only a simple stitching together at the feature level, rather than end-to-end multimodal feature fusion and alignment within the deep network. The failure to organically integrate imaging features, precise pathological localization, and rich clinical information is a key limitation restricting its ability to achieve accurate and personalized predictions.

[0061] 5. Address the issue of insufficient model generalization ability and robustness caused by limitations in the scale and quality of training data: The training sample size is severely insufficient and the data quality is poor. Training with small samples easily leads to overfitting, meaning the model memorizes the noise specific to the training set rather than general patterns. Poor data quality and the presence of noise introduce invalid features from the images during training, resulting in poor predictive performance. These models may perform reasonably well during internal validation, but their performance deteriorates significantly and their generalization ability is poor when applied to external data from different devices, medical institutions, or populations.

[0062] This technical solution aims to directly address the core deficiencies in existing post-macular hole vision prediction technologies. By introducing an advanced deep learning architecture, retinal attention, hole enhancement, and fusing multimodal data, it achieves a prediction model with higher accuracy, stronger generalization ability, and better interpretability. The specific technical problems it addresses are as follows: 1. Addressing the bottleneck in prediction accuracy caused by the inability of traditional models to characterize complex nonlinear relationships among multiple factors: Existing machine learning methods rely on handcrafted features and shallow models, which cannot effectively learn the complex interactions between multiple factors (such as pore morphology, pathological course, and microstructural integrity). This technical solution employs deep convolutional neural networks (such as ConvNeXt) and an end-to-end learning paradigm. Through multi-layer nonlinear transformations, it automatically extracts high-order features from the original OCT images, capturing the nonlinear relationships and high-level semantic information between these factors, thereby significantly improving prediction accuracy and breaking through the performance ceiling of traditional models.

[0063] 2. Addresses the issues of feature extraction relying on manual intervention, being highly subjective, inefficient, and unable to capture deep semantic features: Existing technologies require physicians to manually measure morphological parameters, introducing subjective bias and resulting in low efficiency. This technical solution automates the feature extraction process, utilizing a pre-trained segmentation model and a deep learning backbone network to directly learn prognostic-related deep features (such as the integrity of retinal structures) from OCT images. This avoids manual intervention, ensures feature consistency and repeatability, and efficiently processes large-scale data to meet real-time clinical needs.

[0064] 3. To address the problem that existing methods do not pay enough attention to the lesion area and are unable to capture subtle local features: Existing models lack targeted attention to key regions (such as lesions and retinal layers). This technical solution innovatively integrates multi-scale attention mechanisms, including retinal layer attention (focusing on structures such as the ellipsoidal band and external membrane) and lesion region enhancement attention (guided by YOLOv8 detection boxes). This enables the model to dynamically focus on subtle local features of the lesion, strengthen the learning of relevant regions, and suppress background noise, thereby improving the accuracy and robustness of feature learning.

[0065] 4. Addressing the issue of shallow multimodal information fusion failing to achieve feature-level deep fusion: Existing methods often employ simple splicing or decision-level fusion, failing to fully utilize multi-source data. This technical solution achieves deep fusion at the feature level, integrating OCT image features, pore spatial information, and clinical data (such as age and disease duration) through attention mechanisms and a joint learning framework. This ensures complementarity and synergy between different modalities, enhancing the model's ability to predict personalized prognoses.

[0066] 5. Address the issues of insufficient model generalization ability and robustness caused by limitations in training data scale and quality: Existing models are prone to overfitting and have poor generalization ability due to training with small sample sizes. This technical solution adopts a pre-training and fine-tuning strategy, using a large-scale public dataset for model initialization, and then fine-tuning on a specific dataset. Combined with data augmentation and regularization techniques, it effectively reduces the risk of overfitting, improves the model's adaptability to different devices, medical institutions, and populations, and ensures stability and reliability in real clinical environments.

[0067] Specifically: Please see Figures 1-11 This invention provides a method for predicting postoperative visual acuity after macular hole surgery based on multimodal feature fusion, including steps S110-S120: Step S110. Acquire preoperative OCT images and clinical parameters of the patient. Specifically, OCT images are acquired using a high-resolution fundus scanner to ensure that image resolution and quality meet standards. Clinical parameters include key information such as patient age, gender, duration of disease, baseline visual acuity, and tear type. The data comes from the hospital's electronic health record system and has been anonymized to protect privacy. This raw data will be used in subsequent feature extraction modules to support deep fusion of multimodal information.

[0068] Step S120. Input the preoperative OCT image and the clinical parameters into the trained prediction model to obtain the postoperative visual acuity prediction result; The prediction model processes the preoperative OCT images and clinical parameters in the following manner, including steps S1201-S1205: Step S1201. Extract multi-scale image features from the preoperative OCT images using a feature extraction backbone network.

[0069] Step S1202. Using the retinal hierarchical attention module, the multi-scale image features are weighted and adjusted based on the hierarchical mask generated by the retinal segmentation network to focus on specific retinal structural regions.

[0070] In this embodiment, the multi-scale attention mechanism guided by retinal layer masks is a multimodal attention fusion architecture. The attention mechanism uses a multi-channel retinal layer probability map generated by a retinal segmentation network (U-Net3+) as a spatial prior mask. Through collaborative computation between a parallel spatial attention sub-network and a channel attention sub-network, the spatial attention sub-network generates a spatial weight map based on the retinal layer mask and introduces learnable background suppression parameters and retinal enhancement parameters. Differentiated weight suppression and enhancement are implemented for background regions and key retinal structural regions (such as the ellipsoidal band and retinal membrane). Finally, feature reconstruction is achieved through weighted fusion. A spatial prior-guided attention computation paradigm is designed to transform the semantic information output by the segmentation network into dynamic weights that drive feature selection and enhancement, achieving precise focusing on key retinal layers. This fundamentally overcomes the subjectivity, inefficiency, and inconsistency of manually extracting morphological parameters for shallow multimodal information fusion. This module innovatively designs a spatial prior-guided attention computation paradigm.

[0071] Specifically, the spatial attention subnetwork first performs spatial normalization on the input multi-channel retinal layer probability map to generate an initial spatial weight distribution. Based on this, two learnable modulation parameters—a background suppression coefficient α and a key structure enhancement coefficient β—are introduced to dynamically adjust the weight map: α acts on background regions not covered by the target retinal layer (such as the nerve fiber layer), implementing progressive weight decay; β focuses on anatomical structures closely related to postoperative visual recovery (such as the ellipsoidal band interruption area and the edge of the external retinal membrane tear), performing feature enhancement. This differential modulation process is achieved through the following formula: , Where Ws is the initial spatial weight map, and Menhance and Mbackground represent the key structure enhancement mask and background suppression mask, respectively. The modulated weight map W's is subjected to Hadamard product operation with the channel weights Wc output by the channel attention sub-network, and finally cross-modal feature reconstruction is achieved through weighted fusion. This attention mechanism guided by anatomical semantics transforms the hierarchical semantic information provided by the segmentation network into a dynamic regulatory signal that drives feature selection and enhancement, achieving precise focusing on lesion-related biomarkers. It fundamentally overcomes the subjectivity, inefficiency, and inconsistent annotation defects of traditional methods that rely on manual extraction of morphological parameters.

[0072] Step S1203. Using the crack localization and enhancement module, the crack region is located based on the target detection network, and the crack region features in the multi-scale image features are adaptively enhanced.

[0073] This embodiment includes a target detection-based crack localization and adaptive feature enhancement module; a lesion region enhancement method. The module receives the coordinates of the highest-confidence crack bounding box output by a target detection algorithm (YOLOv8), and precisely crops the corresponding region's features from the image feature map through ROI alignment. A small parameterized convolutional network processes the region's features to generate an attention weight map of the crack region, which is then interpolated to restore it to the full image size. Finally, a learnable global enhancement factor controls the fusion strength between the weight map and the original feature map, achieving targeted feature enhancement of the crack region. By combining the precise spatial localization capability of target detection with the deep semantic information of the feature map, and adaptively enhancing the representation strength of the lesion region through a learnable factor, the model can focus on subtle features crucial to prognosis, such as crack boundary morphology and surrounding tissue response, significantly improving the model's perception accuracy and robustness to complex cracks.

[0074] Furthermore, a multi-scale hierarchical attention fusion mechanism is proposed; a feature fusion method, wherein the mechanism introduces retinal hierarchical attention and tear region enhancement modules in multiple feature extraction stages (such as Stage 2, Stage 3, Stage 4) of the backbone network (ConvNeXt); through the synergistic effect of spatial attention and channel attention, adaptive focusing on key structures and suppression of irrelevant backgrounds are achieved at each level; finally, the processed multi-scale feature maps are upsampled to a uniform size and then subjected to channel splicing and dimensionality reduction fusion. This mechanism collaboratively fuses heterogeneous features (retinal structures and tear regions) at different abstraction levels of the deep network, utilizing the attention mechanism to achieve cross-scale adaptive weighting and information filtering, ensuring effective complementarity and integration of features at different levels (from details to semantics), thereby comprehensively improving the richness and discriminativeness of feature representation.

[0075] To address the issue of shallow multimodal information fusion, this solution further designs an efficient multimodal fusion module. This module receives a unified feature map processed by a multi-scale hierarchical attention fusion mechanism, as well as preprocessed non-image modality data (such as patient age, preoperative BCVA, and structured clinical indicators like hole type). In the feature-level fusion stage, a fully connected layer is first used to embed the structured clinical indicators into a high-dimensional feature vector, and a spatial copying operation is used to match its dimension with the image feature map. Subsequently, a gated attention mechanism is employed to calculate the adaptive weights of each spatial location of the image features to the clinical feature vector, achieving refined modulation of image features by clinical information. Finally, the weighted fused cross-modal features are input into the prediction head network to generate the final visual prognosis prediction. This deep fusion strategy effectively overcomes the insufficient intermodal information interaction caused by simple concatenation or average pooling in traditional methods, significantly improving the model's ability to perform joint inference using heterogeneous information.

[0076] Step S1204. Using the multimodal fusion module, the image features after weight adjustment and adaptive enhancement are deeply fused with the embedded and encoded clinical parameters to generate fused features.

[0077] This embodiment includes a deep fusion module for clinical parameters and image features; a multimodal fusion method, wherein the module first converts structured clinical measurement parameters (such as pore diameter, height, etc.) into high-dimensional feature vectors through an embedding network; after aligning this vector sequence with deep features extracted from the image in the feature dimension, it dynamically aggregates and reconstructs cross-modal information to achieve deep complementarity between quantitative data and visual information. This deep feature fusion framework enables bidirectional interaction and deep fusion of heterogeneous multi-source data (image features and structured clinical parameters) at the semantic level, dynamically adjusting the contribution of each modality, and providing a more comprehensive and accurate data foundation for prediction.

[0078] Specifically, this module achieves deep interaction through the Transformer's self-attention mechanism: features from two modalities are concatenated to form a unified sequence. Within the Transformer layer, features at each location can serve as query vectors, interacting with features at other locations to calculate cross-modal attention weights. This design enables bidirectional interaction and deep fusion of heterogeneous multi-source data (image features and structured clinical parameters) at the semantic level, dynamically adjusting the contribution of features from each modality, and providing a more comprehensive and accurate data foundation for prediction. During the fusion process, the Transformer's multi-layered structure captures cross-modal dependencies at different levels: shallow layers focus on local feature interactions, while deep layers capture global semantic associations. This hierarchical fusion mechanism effectively integrates the supplementary information of clinical parameters to image features and the verification role of image features for clinical parameters, forming a more complementary fusion representation.

[0079] Step S1205. Based on the fusion features, output the postoperative visual acuity prediction results through a regressor.

[0080] This regressor consists of one or more fully connected layers, each followed by a ReLU nonlinear activation function to progressively transform and abstract the fused features, capturing complex nonlinear relationships within the data. After processing by the fully connected layers, the fused features are passed through a linear output layer to generate continuous-value visual acuity prognosis predictions, directly reflecting accurate estimates of postoperative visual acuity. This design, through end-to-end optimization, significantly improves prediction accuracy, ensuring the model can reliably handle dynamic interactions of heterogeneous data in clinical applications and output highly robust predictions.

[0081] In one specific implementation, the step utilizes a retinal hierarchical attention module to perform weight adjustment on the multi-scale image features based on a hierarchical mask generated by a retinal segmentation network, including: Obtain a layered mask generated by a retinal segmentation network based on the preoperative OCT image, the layered mask containing a probability map of key retinal layers of the ellipsoidal band and the outer membrane; The hierarchical mask and the image features output by the feature extraction backbone network are input into the attention subnetwork to generate a spatial weight map. By introducing learnable background suppression parameters and retinal enhancement parameters, differentiated weight suppression and enhancement are implemented for background regions and key retinal structure regions based on the spatial weight map. The spatial weight map, after differential weight suppression and enhancement, is used to weight the image features output by the feature extraction backbone network to achieve feature reconstruction.

[0082] This feature reconstruction process significantly enhances the semantic information of key retinal layers (such as the ellipsoid and lateral membrane) by adaptively adjusting feature weights, while suppressing interference from irrelevant background regions, thereby improving the discriminative power of the features. The reconstructed features retain multi-scale contextual information, providing high-purity input for the subsequent multimodal fusion module. In the multimodal fusion module, these reconstructed image features interact with clinical parameters (such as patient age and retinal tear size) at the feature level, calculating intermodal correlation weights through a cross-attention mechanism to achieve dynamic alignment and deep fusion of heterogeneous data. This mechanism introduces learnable fusion parameters to weighted aggregate features from different modalities, generating robust joint representations that effectively capture complex nonlinear dependencies across modalities and address the semantic gap caused by shallow fusion. The fused features are then input into the prediction head network, where fully connected layers and nonlinear activation processing progressively abstract them into high-precision visual prognostic predictions, ensuring end-to-end optimization to improve the model's inference capabilities.

[0083] In one specific implementation, the step utilizes a crack localization and enhancement module to locate the crack region based on a target detection network, and adaptively enhances the crack region features in the multi-scale image features, including: Obtain the coordinates of the tear boundary box output by the target detection network based on the preoperative OCT image; Based on the coordinates of the crack bounding box, the crack region features are cropped from the image features output by the feature extraction backbone network through ROI alignment operation; The features of the crack region are processed using a parameterized convolutional network to generate an attention weight map of the crack region; The attention weight map of the hole region is restored to the full image size through interpolation, and the fusion strength between it and the image features output by the feature extraction backbone network is controlled by a learnable global enhancement factor, thereby achieving feature enhancement of the hole region.

[0084] By adaptively enhancing the features of the lesion region, the semantic weights of key pathological areas are significantly improved, reducing interference from irrelevant structures and thus optimizing the input quality of the multi-scale feature fusion module. Next, this enhanced feature is weighted and concatenated with the original image features output from the feature extraction backbone network. Learnable fusion coefficients are used to dynamically balance local and global information, ensuring that subtle changes in the lesion region are fully preserved. Simultaneously, the attention mechanism introduced during the enhancement process effectively captures the morphological features of the lesion edges, avoiding feature drift caused by image noise or background changes. Finally, these enhanced features are passed to the multi-scale feature fusion module, interacting with feature maps from other levels. Pyramid pooling operations are used to integrate contextual information from different resolutions, providing highly discriminative input to the multi-modal fusion module and improving the overall model's predictive robustness.

[0085] In one specific implementation, the step utilizes a multimodal fusion module to deeply fuse the image features after weight adjustment and adaptive enhancement with the embedded and encoded clinical parameters, including: The clinical parameters are converted into high-dimensional feature vectors through an embedding layer; The high-dimensional feature vector is dimensionally aligned with the image depth features after weight adjustment and adaptive enhancement. A multi-head self-attention mechanism is adopted to perform bidirectional interaction and fusion on the dimension-aligned high-dimensional feature vector and the image depth features, and output the fused features.

[0086] The fused features are normalized through a single layer of normalization to eliminate potential distributional differences between different modalities. Subsequently, a residual connection mechanism is applied to add the original image depth features to the fused features, preserving low-level information and enhancing feature expressiveness. Finally, the weights of the fused features are adjusted to ensure deep fusion of clinical parameters and image features, avoiding the loss of superficial information. The fused features are input into the prediction output module, undergo nonlinear transformation through a multilayer perceptron, and output the corresponding vision prediction result. The entire process optimizes the fusion coefficients and network parameters through end-to-end training, significantly improving the depth and robustness of feature-level interactions.

[0087] In this embodiment, the fusion module first adds modality-specific embedding vectors to the features of each modality to distinguish data from different sources. Then, the multimodal feature sequences are concatenated, and positional encoding is added to preserve sequence information. Deep feature interaction is achieved through multi-layer Transformer blocks, each layer containing a multi-head self-attention mechanism and a feedforward network, with residual connections and layer normalization applied. The final fused features are then input into a classifier after global average pooling to generate vision prediction results. The entire model achieves efficient fusion of multimodal features through end-to-end training and parameter optimization.

[0088] In one specific implementation, the trained prediction model is obtained through a pre-training and fine-tuning strategy, wherein the fine-tuning strategy includes: The feature extraction backbone network is pre-trained on a general image recognition dataset; The overall prediction model was fine-tuned end-to-end on a medical dataset containing preoperative OCT images, clinical parameters, and corresponding postoperative visual acuity labels of patients with macular holes.

[0089] This embodiment presents a pre-training and fine-tuning strategy and data preprocessing method for generalization capability; a model optimization and deployment method, wherein the strategy adopts a paradigm of pre-training on a large-scale public dataset to initialize model weights, and then fine-tuning on a specific target dataset; combined with a series of intelligent data preprocessing operations (such as reflection filling to maintain image edge continuity, center cropping to focus key regions, and random affine transformation to enhance data diversity), and an adaptive parameter optimization mechanism to dynamically adjust the intensity configuration of each attention module. Through the pre-training-fine-tuning strategy and carefully designed data augmentation, the adaptability and robustness of the model to medical image data collected from different sources and devices are effectively improved, ensuring the stable performance of the algorithm in the ever-changing real clinical environment, and providing a solid technical guarantee for large-scale clinical application and promotion.

[0090] Furthermore, this pre-training and fine-tuning strategy, combined with an adaptive parameter optimization mechanism, achieves feature-level deep fusion in the multimodal fusion module. Specifically, during the fine-tuning stage, by dynamically adjusting the weight configuration of the retinal layering attention module and the hole localization enhancement module, the model can adaptively learn the nonlinear correlation between OCT image features and clinical parameters, thereby completing deep interaction and complementarity in the feature space. This mechanism effectively overcomes the limitation of traditional methods where information fusion only reaches a shallow stitching level, significantly improving the synergistic effect of multi-scale feature fusion and providing a more refined and robust feature representation foundation for postoperative vision prediction. Simultaneously, combined with intelligent data preprocessing, such as reflection filling and center clipping, the model's ability to capture subtle local features is further enhanced, ensuring the stability and generalization performance of the fusion process under varying clinical scenarios.

[0091] In one specific implementation, before inputting the preoperative OCT image and the clinical parameters into the trained prediction model, a data preprocessing step is further included, the preprocessing including one or more of reflection filling, center cropping, and random affine transformation.

[0092] These preprocessing steps aim to standardize the input data, reduce noise interference, and enhance the model's generalization ability through data augmentation. Specifically, reflection filling effectively addresses the information loss problem in image boundary regions, ensuring the integrity of edge details; center cropping focuses on the core region of the macular hole, eliminating irrelevant background interference and strengthening the model's capture of subtle lesion features; randomized affine transformations introduce geometric transformations such as rotation, scaling, or translation to simulate image changes under different acquisition devices or angles, increasing the diversity of training data and thus improving the model's robustness in varied clinical scenarios. Through these intelligent preprocessing steps, combined with subsequent adaptive parameter optimization mechanisms, the model can more efficiently learn deep nonlinear correlations between multimodal features during the fine-tuning stage, avoiding the limitations of shallow stitching and achieving feature-level deep fusion, providing a more stable and refined foundation for postoperative visual acuity prediction.

[0093] In one specific implementation, the retinal segmentation network is a U-Net3+ network. Its unique full-scale skip connection mechanism can deeply fuse shallow spatial information and deep semantic features, achieving accurate segmentation of each layer of the retinal structure and providing high-quality input for subsequent hierarchical attention calculations.

[0094] In one specific implementation, the target detection network is a YOLOv8 network. This network, through an improved anchor frame mechanism and a cross-stage feature fusion strategy, efficiently locates the core region of the macular hole and enhances the feature response of the lesion edge, significantly improving the ability to capture subtle local lesions.

[0095] In one specific implementation, the feature extraction backbone network is a ConvNeXt network. This network employs a depthwise separable convolutional and hierarchical feature recombination design, which extracts deep feature representations with rich spatial context information while reducing computational complexity, effectively overcoming the limitations of traditional convolutional neural networks in long-distance dependency modeling.

[0096] Furthermore, the retinal segmentation network is a U-Net3+ network, the object detection network is a YOLOv8 network, and the feature extraction backbone network is a ConvNeXt network. The synergistic application of these specific network architectures lays a solid foundation for the deep fusion of multimodal features in spatial and semantic dimensions. The multi-level feature maps output by these networks will serve as core inputs and be fed into the multi-scale feature fusion module for cross-modal alignment and interaction. Finally, through a gating attention mechanism, feature-level adaptive weighting and nonlinear integration are achieved in the multimodal fusion module.

[0097] In this embodiment, refer to the appendix. Figure 1-11 This provides a more specific operational example: 1. Data preparation and preprocessing Collect preoperative OCT volumetric scan data (preferably linear scans of the fovea of ​​the macular region acquired by Spectralis SD-OCT or similar equipment) and corresponding clinical parameters from patients with a history of macular hole. Clinical parameters should include at least: minimum hole diameter, maximum base diameter, hole height, patient age, duration of symptoms, and preoperative best corrected visual acuity (BCVA).

[0098] Preprocessing OCT images to improve model robustness: Reflection Padding: Fills the edges of an image using mirror reflection to prevent the loss of critical information during random cropping.

[0099] CenterCrop: Crops the image to a fixed size (e.g., 512x512 pixels) to ensure consistent input size.

[0100] Random Affine Transformation: This includes small-amplitude random rotations (such as ±10°) and translations, used as a data augmentation technique to increase the diversity of training data and suppress overfitting.

[0101] Clinical parameters need to be standardized so that their mean is 0 and their standard deviation is 1.

[0102] 2. Prediction Model Construction and Training The core prediction model architecture of this invention is as follows: Figure 2 As shown, its construction and training steps are as follows: (1) Feature extraction backbone network We employ the ConvNeXt network as the backbone for feature extraction. First, this network is pre-trained on the large public dataset ImageNet to obtain powerful general visual feature extraction capabilities. Then, it is ported to this task to extract multi-scale feature maps from preprocessed OCT images. We utilize the outputs of its Stage2, Stage3, and Stage4, which correspond to different receptive fields and abstraction levels, respectively.

[0103] (2) Retinal layered attention module Segmentation Network: The U-Net3+ network is trained on a labeled retinal layer dataset. It can perform pixel-level segmentation on the input OCT image and output a probability map of the key retinal layers, including the elliptic zone (EZ) and the outer membrane (ELM), i.e., a spatial prior mask.

[0104] Attention computation: The spatial prior mask generated by U-Net3+ is fused with the feature map output from a certain stage of ConvNeXt (such as Stage3), and input into a lightweight attention subnetwork (usually composed of several convolutional layers and activation functions). This subnetwork outputs a spatial weight map.

[0105] Differential Enhancement and Suppression: Two learnable parameters are introduced into the network: a background suppression factor (β) and a retinal enhancement factor (α). During training, the model automatically learns to optimize these two factors, such that in the spatial weight map, the weights of background regions are suppressed (multiplied by β, β < 1), while the weights of key retinal structural regions are enhanced (multiplied by α, α > 1).

[0106] Feature reconstruction: The optimized spatial weight map is multiplied element-wise with the original feature map to focus on key structural regions and suppress background noise.

[0107] (3) Crack location and reinforcement module Detection network: The YOLOv8 object detection algorithm is used and trained on a dataset with labeled crack bounding boxes. It can quickly and accurately output the coordinates (x,y,w,h) of the crack bounding box with the highest confidence in the OCT image.

[0108] ROI feature alignment: Based on the coordinates output by YOLOv8, the ROI alignment (ROIAlign) operation is used to precisely crop the features of the corresponding region from the feature map extracted from the ConvNeXt backbone network, forming a fixed-size crack region feature block.

[0109] Local attention generation: Input the feature block of the crack region into a small parameterized convolutional network to generate an attention weight map that focuses on the morphology of the crack interior and edges.

[0110] Global Feature Enhancement: The generated local attention weight map is restored to the full-image size using bilinear interpolation. Subsequently, a learnable global enhancement factor (γ) controls the fusion strength between this full-image weight map and the original backbone feature map (e.g., through weighted summation), thereby achieving global feature enhancement for the crack region.

[0111] (4) Multi-scale feature fusion Feature maps from different stages (Stage 2, Stage 3, Stage 4) of ConvNeXt, after being processed by the retinal hierarchical attention module and the hole localization and enhancement module, are upsampled to a uniform size (usually the size of Stage 4) and then stitched together along the channel dimension. Subsequently, 1x1 convolutions are used to reduce and integrate the stitched high-dimensional features, forming a comprehensive image depth feature set. Dropout layers can be added during this process to prevent overfitting.

[0112] (5) Multimodal fusion module Clinical parameter embedding: Standardized structured clinical parameters are transformed into a high-dimensional feature vector through a fully connected layer (embedding layer), so that its dimension matches the depth features of the image.

[0113] Feature alignment: Aligning the clinical parameter vector with the comprehensive image depth features along the feature dimension.

[0114] Cross-modal interaction: The aligned image feature sequence and clinical parameter feature sequence are fed into a simplified Transformer encoder (i.e., a multi-head self-attention mechanism). Through query, key, and value operations, the model can dynamically calculate the correlation between image features and clinical features, achieving bidirectional information interaction and deep fusion, and ultimately outputting a fused representation rich in multimodal information.

[0115] (6) Predicted output The fusion representation output by the multimodal fusion module is input into a regressor (usually composed of several fully connected layers), and finally outputs a continuous value, which is the predicted postoperative best corrected visual acuity (BCVA, such as LogMAR visual acuity).

[0116] 3. Model Training and Validation The preprocessed dataset is divided into training, validation, and test sets in a ratio (e.g., 7:1:2).

[0117] Loss function: The mean squared error (MSE) or smoothed L1 loss is used as the loss function.

[0118] Optimizer: Use the AdamW optimizer.

[0119] Training Process: The constructed fusion model is trained end-to-end on the training set using a phased optimization strategy. During training, gradient accumulation and mixed precision training techniques are used to improve training stability, and validation loss and key performance indicators are monitored on the validation set. When the validation loss no longer improves for several consecutive epochs, an early stopping mechanism is used to select the optimal model weights. Finally, the model performance is comprehensively evaluated on an independent test set, calculating multi-dimensional indicators including AUC, accuracy, F1 score, sensitivity, and specificity to ensure the model has good generalization ability and clinical applicability.

[0120] 4. Model Application The trained model can be deployed to clinical diagnostic systems or cloud service platforms. When postoperative visual acuity prediction is needed for new patients with macular holes, only preoperative OCT images and relevant clinical parameters need to be collected and input into the model. Personalized postoperative visual acuity prediction results can be obtained in a short time, providing strong support for clinical decision-making.

[0121] As can be seen, this application provides a method for postoperative visual acuity prediction based on multimodal attention fusion after macular hole surgery. This method innovatively integrates the U-Net3+ retinal segmentation network, the YOLOv8 macular hole detection algorithm, and the ConvNeXt feature extraction backbone, achieving high-precision postoperative visual acuity prediction through a multi-scale attention mechanism. The model employs an end-to-end training approach, enabling automatic processing of OCT input images and output of visual acuity prognosis assessment results. Specific technical features are as follows: (a) To address the subjective bias and inefficiency caused by the reliance on manual measurement of morphological parameters in traditional methods, a multi-scale attention mechanism guided by retinal layer masking was designed. This mechanism utilizes the retinal layer probability map generated by U-Net3+ segmentation, and through the synergistic effect of spatial attention and channel attention, automatically focuses on key retinal structures such as the ellipsoidal zone and the lateral membrane. Furthermore, it employs learnable background suppression parameters and retinal enhancement parameters to suppress irrelevant regions and enhance attention to key anatomical structures.

[0122] (b) Since existing methods do not pay enough attention to the lesion area and are difficult to capture the subtle features of the tear region, a tear localization and adaptive enhancement module based on target detection was designed. This module integrates the coordinate information of the highest confidence tear box detected by YOLOv8, accurately extracts the tear region features through ROI alignment, and dynamically adjusts the attention weight of the tear region using a learnable enhancement factor to achieve targeted enhancement of subtle features such as tear boundary morphology and surrounding tissue response, significantly improving the model's perception accuracy and robustness of the lesion area.

[0123] (c) To address the problem of insufficient multimodal information fusion, a multi-scale hierarchical attention fusion mechanism is designed. This mechanism introduces retinal hierarchical attention and hole region enhancement into the four feature extraction stages of the ConvNeXt backbone network, respectively. Through the synergistic effect of spatial attention and channel attention, it achieves adaptive focusing on key retinal structures and intelligent suppression of irrelevant background regions. Furthermore, it employs feature concatenation and 1×1 convolution dimensionality reduction to achieve effective integration of multi-level features.

[0124] (d) Since features at different abstraction levels contribute differently to prediction, a cross-scale feature fusion and optimization module was designed. This module upsamples the feature maps of Stage2, Stage3, and Stage4 to a uniform size and then performs channel stitching. It uses 1×1 convolution for dimensionality reduction and adds a Dropout layer to prevent overfitting of retinal features, thereby achieving effective integration of multi-level features and removal of redundant information.

[0125] (e) To address the need for multimodal data fusion, a cross-modal feature fusion module was designed. This module first converts clinical measurement parameters (including minimum hole diameter, maximum base diameter, internal opening diameter, hole height, and hole hypotenuse length) into high-dimensional vector representations through a feature embedding layer. Then, it aligns and integrates these vector representations with the image depth features extracted by the ConvNeXt backbone network. A multi-head self-attention mechanism enables bidirectional interaction and deep fusion between image features and text-encoded features, dynamically aggregating multimodal information and ultimately outputting a fusion representation rich in multimodal information. This achieves deep interaction and complementary enhancement between image features and clinical parameters, providing a more comprehensive and accurate feature foundation for subsequent vision prediction.

[0126] (f) To address the overfitting and insufficient generalization capabilities caused by limited training data, a pre-training-fine-tuning training strategy and an adaptive parameter optimization mechanism are designed. The model weights are initialized through pre-training on a large-scale public dataset, followed by fine-tuning on a specific dataset. This is combined with learnable background suppression factors, retinal enhancement factors, and hole enhancement factors to dynamically optimize the intensity configuration of each attention module. Simultaneously, it supports various preprocessing methods for image data, such as reflection filling, center cropping, and random affine transformation.

[0127] The technical solution provided in this application has the following beneficial effects: An innovative multimodal attention fusion architecture significantly improves the overall performance of visual acuity prediction after macular hole surgery. A multi-scale attention mechanism guided by retinal layer masking enables the model to automatically identify and focus on key retinal structures that significantly impact visual prognosis, such as the ellipsoid and lateral membrane, while effectively suppressing interference from irrelevant background areas. This intelligent attention allocation mechanism not only overcomes the subjectivity and inconsistency of traditional methods that rely on manual measurement of morphological parameters, but more importantly, it achieves in-depth mining and precise representation of complex anatomical features. This allows the model to capture subtle pathological changes that are difficult for the human eye to detect, thus providing a more accurate and reliable predictive basis for postoperative visual recovery. By integrating advanced target detection technology with an adaptive enhancement module, the system effectively addresses the limitations of existing methods in focusing on lesion areas. The system can accurately locate the tear area and perform targeted feature enhancement, significantly improving the perception of subtle features such as tear boundary morphology and surrounding tissue response. The multi-scale hierarchical attention fusion mechanism achieves deep integration and optimization of multimodal information. By collaboratively fusing retinal structural features and tear region information at different network levels, the model can fully utilize the synergistic effect of spatial attention and channel attention to achieve adaptive weighting and focus on features of each modality. This fusion method not only ensures the effective integration of features at different levels of abstraction but also comprehensively enhances the richness and discriminativeness of feature representation, establishing a more comprehensive and reliable feature foundation for accurate prediction. This approach effectively integrates clinical measurement parameters with image depth features, significantly enhancing the model's ability to integrate multi-source information. By converting structured clinical parameters into high-dimensional vector representations and engaging in deep interaction with image features, the model can simultaneously utilize quantitative measurement data and visual feature information to achieve a more comprehensive characterization of lesion features. The application of a multi-head self-attention mechanism enables bidirectional information interaction between different modalities, dynamically adjusting the contribution weights of each modality, thereby enhancing the richness and discriminativeness of feature representation. This deep fusion mechanism effectively overcomes the limitations of single-modal information, improves the model's adaptability to complex cases, provides a more reliable feature basis for postoperative visual acuity prediction, and ultimately significantly improves the accuracy and clinical applicability of prediction results. It possesses excellent generalization ability and stable performance. The pre-training-fine-tuning strategy and adaptive parameter optimization mechanism employed enable the model to adapt well to medical imaging data from different sources and devices, while data preprocessing ensures consistent processing capabilities for diverse imaging data. This strong adaptability guarantees the model's stable performance in varied clinical environments, enabling its widespread application across different medical institutions and equipment platforms, providing a solid technical guarantee for large-scale clinical deployment.

[0128] According to one embodiment of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for predicting postoperative visual acuity of macular hole based on multimodal feature fusion.

[0129] According to one embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement a method for predicting postoperative visual acuity of macular hole based on multimodal feature fusion.

[0130] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0131] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0132] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for predicting postoperative visual acuity after macular hole surgery based on multimodal feature fusion, characterized in that, Includes the following steps: Acquire preoperative OCT images and clinical parameters of the patient using optical coherence tomography (OCT). The preoperative OCT images and clinical parameters are input into the trained prediction model to obtain the postoperative visual acuity prediction results; The prediction model processes the preoperative OCT images and clinical parameters in the following manner: Multi-scale image features are extracted from the preoperative OCT images using a feature extraction backbone network; Using a retinal hierarchical attention module, the multi-scale image features are weighted and adjusted based on the hierarchical mask generated by the retinal segmentation network to focus on specific retinal structural regions. Using a crack localization and enhancement module, crack regions are located based on a target detection network, and the crack region features in the multi-scale image features are adaptively enhanced. Using a multimodal fusion module, the image features after weight adjustment and adaptive enhancement are deeply fused with the embedded and encoded clinical parameters to generate fused features; Based on the fusion features, the postoperative visual acuity prediction results are output through a regressor.

2. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 1, characterized in that, The steps utilize a retinal hierarchical attention module to adjust the weights of the multi-scale image features based on the hierarchical mask generated by the retinal segmentation network, including: Obtain a layered mask generated by a retinal segmentation network based on the preoperative OCT image, the layered mask containing a probability map of key retinal layers of the ellipsoidal band and the outer membrane; The hierarchical mask and the image features output by the feature extraction backbone network are input into the attention subnetwork to generate a spatial weight map. By introducing learnable background suppression parameters and retinal enhancement parameters, differentiated weight suppression and enhancement are implemented for background regions and key retinal structure regions based on the spatial weight map. The spatial weight map, after differential weight suppression and enhancement, is used to weight the image features output by the feature extraction backbone network to achieve feature reconstruction.

3. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 2, characterized in that, The steps utilize a crack localization and enhancement module to locate crack regions based on a target detection network, and adaptively enhance the crack region features in the multi-scale image features, including: Obtain the coordinates of the tear boundary box output by the target detection network based on the preoperative OCT image; Based on the coordinates of the crack bounding box, the crack region features are cropped from the image features output by the feature extraction backbone network through ROI alignment operation. The features of the crack region are processed using a parameterized convolutional network to generate an attention weight map of the crack region; The attention weight map of the hole region is restored to the full image size through interpolation, and the fusion strength between it and the image features output by the feature extraction backbone network is controlled by a learnable global enhancement factor, thereby achieving feature enhancement of the hole region.

4. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 3, characterized in that, The steps utilize a multimodal fusion module to deeply fuse the weighted and adaptively enhanced image features with the embedded and encoded clinical parameters, including: The clinical parameters are converted into high-dimensional feature vectors through an embedding layer; The high-dimensional feature vector is dimensionally aligned with the image depth features after weight adjustment and adaptive enhancement. A multi-head self-attention mechanism is adopted to perform bidirectional interaction and fusion on the dimension-aligned high-dimensional feature vector and the image depth features, and output the fused features.

5. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 1, characterized in that, The trained prediction model is obtained through a pre-training and fine-tuning strategy, the fine-tuning strategy including: The feature extraction backbone network is pre-trained on a general image recognition dataset; The overall prediction model was fine-tuned end-to-end on a medical dataset containing preoperative OCT images, clinical parameters, and corresponding postoperative visual acuity labels of patients with macular holes.

6. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 1, characterized in that, Before inputting the preoperative OCT images and clinical parameters into the trained prediction model, a data preprocessing step is also included, which includes one or more of reflection filling, center cropping, and random affine transformation.

7. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 1, characterized in that, The retinal segmentation network is a U-Net3+ network.

8. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 1, characterized in that, The target detection network is the YOLOv8 network.

9. The method for predicting postoperative visual acuity of macular hole surgery based on multimodal feature fusion according to claim 1, characterized in that, The feature extraction backbone network is the ConvNeXt network.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the postoperative visual acuity prediction method based on multimodal feature fusion as described in any one of claims 1 to 9.

Images