Auxiliary detection method for carotid plaque based on multi-modal fusion of external eye images

Abstract

This invention discloses a carotid artery plaque-assisted detection method based on multimodal fusion of external eye images. First, external eye image data and corresponding electronic health record data are collected to construct a multimodal dataset. Then, anatomical masks are generated through offline preprocessing to obtain clean ocular surface images. An anatomically guided multimodal fusion network is constructed, including a visual branch, a clinical branch, a two-stage fusion module, and a classification layer. The trained anatomically guided multimodal fusion network model is used to infer the risk probability of carotid artery plaques from the external eye images and clinical data to be detected. This invention improves detection accuracy and robustness by preserving high-frequency texture features and fusing clinical information, achieving non-invasive and low-cost carotid artery plaque-assisted screening.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and artificial intelligence-assisted diagnosis, specifically involving a framework for a carotid artery plaque auxiliary detection method based on multimodal fusion of external eye images. Background Technology

[0002] Carotid atherosclerotic plaques are a major risk factor for ischemic stroke. Early detection and assessment of carotid plaques are of great clinical significance for stroke prevention. Currently, commonly used clinical detection methods include carotid ultrasound, CT angiography (CTA), and magnetic resonance angiography (MRA). Although these methods have high diagnostic accuracy, ultrasound examination is highly dependent on operator experience, and CTA and MRA equipment are expensive, complex to operate, and have significant limitations due to radiation risks or contraindications, making them difficult to popularize in primary healthcare institutions or large-scale population screening.

[0003] In recent years, ocular omics research has revealed the potential of the eye as a "window" into the body's vascular system. Anatomical and pathophysiological studies have shown that the bulbar conjunctival microvessels covering the sclera are supplied by branches of the ophthalmic artery, which is the first branch of the internal carotid artery. Therefore, morphological abnormalities of bulbar conjunctival microvessels (such as decreased density and increased tortuosity) are significantly associated with the risk of cardiovascular diseases such as hypertension and stroke. However, existing analysis methods based on external eye images face two major challenges: first, consumer-grade external eye images contain significant anatomical noise (such as eyelashes and eyelids), interfering with the extraction of pathological features; second, bulbar conjunctival microvessels exhibit high-frequency, fine-grained textures, which traditional convolutional neural networks are prone to smoothing when processing, leading to the loss of crucial information. Furthermore, single image analysis often lacks key clinical background information such as age and blood pressure, limiting diagnostic accuracy.

[0004] Therefore, there is an urgent need for a high-precision, non-invasive, and easily accessible method for assisting in the detection of carotid plaques that can effectively filter out anatomical noise, accurately capture high-frequency vascular textures in the scleral region, and deeply integrate clinical information. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a carotid artery plaque-assisted detection method based on multimodal fusion of external eye images. It aims to simulate the comprehensive diagnostic process of clinicians through deep learning technology, thereby achieving high-precision and non-invasive screening of carotid artery plaques.

[0006] In a first aspect, embodiments of this application provide a carotid artery plaque-assisted detection method based on multimodal fusion of external eye images, comprising the following steps:

[0007] First, external eye image data and corresponding electronic health record data were collected to construct a multimodal dataset.

[0008] Then, an anatomical mask is generated through offline preprocessing to obtain a clean ocular surface image.

[0009] An Anatomy-Guided Multi-Modal Network (AG-MMNet) was constructed, comprising a visual branch, a clinical branch, a two-stage fusion module, and a classification layer. The visual branch employs an improved frequency-domain perception backbone network, termed the Anatomy-Guided Spectral Former (AG-SpectFormer).

[0010] Anatomy-guided training and optimization of multimodal fusion network models based on multimodal datasets

[0011] Finally, the trained anatomy-guided multimodal fusion network model is used to infer the risk probability of carotid artery plaque from the external eye images and clinical data to be tested.

[0012] In one possible implementation, the construction of the multimodal dataset is specifically achieved as follows: A scleral shadowless imaging system is used to acquire images of the subject's left eye in five standard gaze directions. The images are stored in RGB format with an original resolution of 960×640 pixels. After acquisition, samples with defocus, closed eyes, or severe occlusion are removed. Simultaneously, the subject's electronic health records are collected, including age, gender, height, weight, systolic blood pressure, and diastolic blood pressure, and three derived features—body mass index, pulse pressure, and mean arterial pressure—are calculated. All continuous variables are standardized using Z-Score, and missing values ​​are filled with the number of digits in the training set, ultimately forming a 9-dimensional clinical feature vector.

[0013] In addition, a unique identifier (UUID) is assigned to each subject to link all external eye images and electronic health record data of the same subject, ensuring no data leakage during subsequent dataset partitioning.

[0014] In one possible implementation, the offline preprocessing is specifically implemented as follows: Several images are randomly selected from the dataset, and pixel-level annotation tools are used to annotate the background, sclera, and iris regions of each image to form a training subset; a lightweight U-Net segmentation network is trained based on this training subset; the trained lightweight U-Net segmentation network is used as an offline preprocessing module for anatomical mask preprocessing, inferring from the original external eye images in the dataset to generate a binary anatomical mask; the original external eye images and the anatomical mask are multiplied pixel-by-pixel, forcing the background region pixels to zero to obtain a pure ocular surface image retaining only the sclera and iris.

[0015] In one possible implementation, the frequency-domain aware backbone network comprises three stages: the shallow layers (Stage 1 and Stage 2) output high-resolution feature maps, incorporating spectral gating units to preserve high-frequency microvascular texture; the deep layers (Stage 3) output low-resolution feature maps, employing self-attention modules to capture global semantic features. An anatomy-guided multi-scale aggregator processes the outputs of Stage 2 and Stage 3 to fuse shallow texture features with deep semantic features, and performs masked global pooling using an anatomical mask to ultimately generate a purified visual feature vector. The clinical branch employs a TabNet network based on a sequential attention mechanism to perform instance-level feature encoding on electronic health record data, generating high-dimensional clinical feature embeddings.

[0016] In one possible implementation, after obtaining features from both the visual and clinical branches, a two-stage fusion module performs multimodal interaction. The two-stage fusion module consists of feature-level modulation and decision-level gating: the first stage is feature-level modulation, which uses the FiLM mechanism, conditioned on clinical feature embeddings, to perform channel-level affine transformations on the feature maps output from Stage 2 and Stage 3 of the visual branch, achieving dynamic enhancement of visual features. The enhanced feature maps replace the original feature maps and continue flowing to subsequent network layers in the visual branch (including multi-scale feature fusion and masked global pooling); the second stage is decision-level gating, which uses a learnable gating network to dynamically calculate the fusion weights of the purified visual feature vectors and clinical feature embeddings in the final decision, and after weighted summation, outputs the final disease prediction probability through a classification layer.

[0017] In one possible implementation, image preprocessing is performed on the training data before training the multimodal fusion network, including spatial and structural processing and numerical normalization. The spatial and structural processing includes converting the clean ocular surface image obtained after offline preprocessing in step 2 into a standard RGB three-channel format; downsampling to 224×224 resolution through random region cropping; performing random horizontal flipping with a 50% probability; performing random rotation; and perturbing brightness and contrast to enhance generalization ability. The numerical normalization uses the mean and standard deviation of the ImageNet dataset for cross-channel normalization, first scaling the pixel values ​​from [0,255] to [0,1], and then using the RGB three-channel mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225], according to... The formula adjusts the pixel value distribution to a standard normal distribution range centered at zero.

[0018] In one possible implementation, during model training, the multimodal dataset is proportionally divided into a training set and an independent test set based on the subject's unique identifier (UUID), and hierarchical 5-fold cross-validation is used for hyperparameter tuning within the training set. The optimizer uses AdamW, combined with linear warm-up and cosine annealing strategies, employing gradient accumulation and global gradient clipping. Training uses a composite objective function, including a primary task classification loss (square root smoothed weighted cross-entropy, used to alleviate class imbalance) and an auxiliary supervision loss (calculated based on clinical branch outputs, used to prevent modality collapse).

[0019] In one possible implementation, the reasoning is specifically implemented as follows: for the subject to be screened, the acquired multi-directional external eye images are preprocessed with anatomical masks and then input together with the corresponding standardized clinical feature vectors into the trained AG-MMNet model to output the carotid plaque risk probability.

[0020] Secondly, embodiments of this application provide a carotid plaque-assisted detection device based on multimodal fusion of external eye images, comprising the following modules:

[0021] Dataset construction module: used to collect external eye image data and corresponding electronic health record data of clinical populations undergoing carotid plaque assessment to construct a multimodal dataset.

[0022] Offline preprocessing module: Generates anatomical masks through offline preprocessing to obtain clean ocular surface images.

[0023] Model building module: Used to build the Anatomy-Guided Multimodal Fusion Network AG-MMNet, including a visual branch, a clinical branch, a two-stage fusion module, and a classification layer. Among them, the visual branch adopts an improved frequency-domain aware backbone network, called Anatomy-Guided Spectral Former (AG-SpectFormer).

[0024] Training module: Training and optimization of multimodal fusion network models based on the constructed dataset under dissection guidance.

[0025] Risk prediction module: Used to infer the risk probability of carotid artery plaques from the external eye images and clinical data to be tested using a trained anatomy-guided multimodal fusion network model.

[0026] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory.

[0027] The memory is used to store computer programs.

[0028] When the processor executes the program stored in the memory, it implements any of the carotid plaque-assisted detection methods described in this application.

[0029] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the carotid plaque-assisted detection methods described in this application.

[0030] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the carotid plaque-assisted detection methods described in this application.

[0031] The beneficial effects of this invention are as follows:

[0032] 1. This invention proposes for the first time a carotid plaque-assisted detection method based on multimodal fusion of external eye images. By constructing a large-scale multimodal dataset containing 12,448 subjects, it demonstrates for the first time at a large sample level the strong correlation between bulbar conjunctival microvascular texture features and carotid plaques.

[0033] 2. To address the high-frequency texture characteristics of microvessels and environmental noise interference, this invention innovatively proposes a combined architecture of AG-SpectFormer frequency domain backbone network and anatomically guided multi-scale aggregator. The former keenly captures high-frequency vascular morphological cues through spectral gating mechanism, while the latter explicitly removes background noise through hard-space attention mechanism, thereby achieving purification and decoupling of visual features.

[0034] 3. This invention designs a hierarchical multimodal fusion strategy that includes feature-level modulation and decision-level gating. It utilizes the nonlinear clinical features encoded by TabNet to dynamically calibrate the visual network, effectively solving the modality imbalance problem in multimodal learning.

[0035] In large-scale validation, the method of this invention achieved excellent performance of 92.14% AUC and 73.18% sensitivity on independent test sets, significantly outperforming the single-modal baseline. This invention requires only a regular camera to acquire external eye images, eliminating the need for mydriasis, is radiation-free, easy to operate, and low-cost. It can be widely deployed in community health service centers, physical examination centers, and mobile health applications, providing a novel tool for the early prevention and treatment of carotid artery plaques, with significant clinical application value and socio-economic benefits. Attached Figure Description

[0036] Figure 1 This is a model framework diagram in an example of the present invention.

[0037] Figure 2 A heatmap for visualizing model attention in an example of this invention.

[0038] Figure 3 This is a bar chart showing the results of the physical occlusion experiment in an embodiment of the present invention.

[0039] Figure 4 This is a comparative schematic diagram of the activation patterns of different backbone networks in the scleral region in an embodiment of the present invention. Detailed Implementation

[0040] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0041] This application provides a method for assisting in the detection of carotid plaques based on multimodal fusion of external eye images, comprising the following steps:

[0042] Step 1: Collect external eye image data and corresponding electronic health record data to construct a multimodal dataset. The specific implementation is as follows:

[0043] 1-1. The data for this example came from a comprehensive health assessment cohort comprising 12,448 individuals. All participants underwent a comprehensive health examination at a hospital. The data included ocular surface image data and structured electronic health records (EHRs).

[0044] (1) Collection of electronic health records: A structured electronic health record is collected for each subject. This includes: age, gender, height, weight, systolic blood pressure and diastolic blood pressure.

[0045] (2) Acquisition of ocular surface images: A standardized set of ocular surface images was acquired for each subject using a scleral shadowless imaging system. All images were acquired at a fixed high resolution (960×640 pixels) and with calibrated illumination and focus settings including five different gaze positions: downward, upward, left, right, and emmetropia. In the specific implementation of this invention, the dataset from the left eye was primarily used for subsequent analysis.

[0046] (3) Acquisition of diagnostic labels: To achieve supervised deep learning, each subject must undergo a carotid artery plaque diagnostic examination. In this embodiment, all subjects underwent carotid ultrasound scans, and professional physicians labeled the scan results to determine whether carotid artery plaques were present. These labels will serve as target variables for subsequent model training.

[0047] 1-2 Clinical Data Collection and Characterization

[0048] Electronic health records were collected for each subject, including the following raw indicators: age, sex, height (cm), weight (kg), systolic blood pressure (SBP, mmHg), and diastolic blood pressure (DBP, mmHg).

[0049] To enhance the model's ability to capture nonlinear physiological relationships, three key derived features are introduced based on medical priors:

[0050] Body Mass Index (BMI): It is used to assess metabolic load.

[0051] Pulse pressure difference: It reflects the degree of arteriosclerosis.

[0052] Mean arterial pressure: This reflects the organ perfusion status.

[0053] All continuous variables were standardized using Z-scores, and missing values ​​were padded with the number of bits in the training set. Ultimately, the clinical characteristics of each subject were organized into a 9-dimensional clinical feature vector for subsequent multimodal fusion.

[0054] Step 2: Generate an anatomical mask using the offline preprocessing module to obtain a clean ocular surface image. The specific steps are as follows:

[0055] To eliminate interference from non-anatomical noise such as eyelids, eyelashes, and periocular skin, this invention employs a deep learning-based region of interest extraction strategy, which is implemented in three steps: first, a labeled subset is constructed for supervised training; second, a lightweight segmentation model is trained; and finally, the trained model is used as an offline preprocessing module to generate anatomical masks from the full dataset and obtain clean images.

[0056] (1) Construction of labeled subset: 3,000 images are randomly selected from the dataset, and the background, sclera and iris regions are labeled to form a training subset.

[0057] (2) Segmentation model training: A lightweight U-Net network was trained based on the above training subset. The network achieved a foreground Dice coefficient of 97.12% and an intersection-over-union ratio of 94.40% on the validation set.

[0058] (3) Offline preprocessing: The trained U-Net is used to infer and generate a binary anatomical mask (1 represents the sclera and iris, and 0 represents the background) from the original external eye image. The original image is then multiplied pixel by pixel with the mask, and the background pixels are set to zero to obtain a clean image that retains only the ocular surface. This operation provides an accurate spatial prior for the subsequent model to focus on the pathological vascular texture.

[0059] Step 3: Construct an end-to-end deep learning network as an Anatomy-Guided Multi-Modal Network (AG-MMNet) to fuse clean ocular surface images with clinical features for carotid artery plaque risk assessment. Figure 1 As shown, AG-MMNet consists of three parts: a visual branch, a clinical branch, and a two-stage fusion module.

[0060] 3-1 Overall Architecture

[0061] The network input is the purified ocular surface image after step 2 processing. and the 9-dimensional clinical feature vector standardized by step 1. The visual branch is responsible for extracting microvascular texture features from ocular surface images, while the clinical branch is responsible for non-linear encoding of clinical features. The two-stage fusion module then performs deep interaction between the two to ultimately output the plaque risk probability.

[0062] 3-2 Visual Branch: Anatomy-Guided Frequency Domain Perceptual Network (AG-SpectFormer)

[0063] The vision branch employs an improved SpectFormer architecture, using a hybrid design of "early frequency domain, late spatial domain" to overcome the smoothing effect of traditional convolutional networks on high-frequency textures. In this embodiment, the vision branch only uses the first three stages of SpectFormer (Stage 1~Stage 3), discarding Stage 4 and the original classification head. Stage 1 and Stage 2 are defined as shallow layers, outputting high-resolution feature maps (56×56→28×28); Stage 3 is defined as a deep layer, outputting low-resolution feature maps (14×14).

[0064] (1) Shallow layer: spectral gating unit

[0065] The original SpectFormer network uses a standard spectral transformation module to process input features in Stages 1 and 2. This invention improves upon these two shallow stages by replacing the original spectral transformation module with an Anatomy-Guided Spectral Gating Unit (AGU). This introduces a learnable frequency-wise gated modulation mechanism while retaining frequency domain processing capabilities, enabling the network to adaptively preserve high-frequency microvascular textures and suppress irrelevant low-frequency noise. Given an input feature tensor, the AGU first maps it to the frequency domain using a two-dimensional real-valued Fast Fourier Transform:

[0066] in Let H and W be the frequency domain coordinates, representing the height and width of the feature map, respectively. Then, a learnable complex weight parameter is introduced. Frequency-by-frequency gated modulation of the spectrum:

[0067]

[0068] in This represents the Hadamard product. The modulated spectrum is restored to the spatial domain via inverse Fourier transform, and residual connections are added:

[0069]

[0070] The aforementioned frequency domain modulation mechanism (i.e., the operation of restoring the spectrum after frequency-gated modulation by learnable complex weights) enables the network to adaptively preserve high-frequency vascular texture and suppress irrelevant low-frequency noise.

[0071] (2) Deep layer: Bullish self-attention

[0072] The network retains the multi-head self-attention module configured in the deep layers of the original SpectFormer network to capture global semantic associations. This module employs the self-attention mechanism from the standard Vision Transformer:

[0073]

[0074] This self-attention mechanism plays a semantic aggregation role at a deeper level, which helps to form a global understanding of the lesion.

[0075] 3-3 Anatomically Guided Multiscale Aggregator (AG-MSA)

[0076] To fuse shallow texture features with deep semantic features and to filter out background noise using dissecting masks, an AG-MSA module is designed, which includes two sub-modules: a multi-scale feature fusion unit and a mask global pooling unit.

[0077] (1) Multi-scale feature fusion (MFF)

[0078] make For high-frequency texture features output by Stage 2, The semantic features output for Stage 3. First, through... Convolution projects both onto the same channel dimension. and to Perform bilinear upsampling to (i.e., 28×28), and then spliced ​​along the channel dimension:

[0079]

[0080] in Represents a 1×1 convolution. Indicates bilinear upsampling. This indicates channel splicing. The intermediate feature map is assembled from multiple parts, and its spatial resolution is [missing information]. The number of channels is This feature map preserves both high-frequency details and semantic information.

[0081] Next, depthwise separable convolutional blocks are used to process the intermediate feature maps. Refining:

[0082]

[0083] in This is channel-wise spatial convolution. For pointwise convolution, This is the ReLU activation function. Output ( The fused feature map combines semantic consistency with high texture fidelity.

[0084] (2) Masked Global Pooling

[0085] Using the generated binary dissection mask The mask is downsampled to the fused feature map. The same resolution (28×28) is denoted as Then, weighted spatial integration is performed:

[0086]

[0087] in This is a smoothing term. The final purified visual feature vector is obtained. (Pick ).

[0088] 3-4 Clinical Branches: Instance-level Coding Based on TabNet

[0089] The TabNet network was used as the clinical feature encoder. TabNet employs a sequential attention mechanism for instance-level selection of input features, effectively capturing non-linear interactions. The resulting 9-dimensional clinical feature vector... Inputting TabNet, it outputs a high-dimensional clinical feature embedding after multiple decision steps. (Pick This embedding not only preserves the information of the original metrics, but also uncovers deep correlations between features.

[0090] 3-5 Two-stage multimodal fusion

[0091] The two-stage fusion module of the present invention consists of feature-level modulation (first stage) and decision-level gating (second stage) connected in series, which together realize progressive multimodal interaction from feature map to feature vector.

[0092] (1) First stage: EHR-guided feature modulation (FiLM)

[0093] Clinical embeddings are used to dynamically calibrate the feature maps of the intermediate layers (stage 2 and stage 3) of the visual branch. Let the output of a certain intermediate layer of the visual branch be... (FiLM is applied to the outputs of Stage 2 and Stage 3 respectively in this invention). Scaling vectors are generated through two linear layers. and offset vector :

[0094]

[0095] in The number of channels is the same as that of the feature map.

[0096] Then to Perform channel-level affine transformation:

[0097]

[0098] in, , , These represent the channel index, height index, and width index of the feature map, respectively. and These are the scaling factor and offset factor for the c-th channel, respectively. For the transformed enhanced feature map at the location The element value at the specified location is used to replace the original feature map of the intermediate layer of the visual branch with the transformed enhanced feature map, which is then input into the subsequent network layers of the visual branch.

[0099] (2) Second stage: Decision-level gating

[0100] The purified visual feature vector is the final output of the visual branch. (448-dimensional) and clinical embedding The 64-dimensional arrays are mapped to the same dimension (448-dimensional) through linear projection, and then concatenated into an input gating network to generate balance coefficients. :

[0101]

[0102] in For the Sigmoid function, This represents vector concatenation. The final multimodal representation. (The final output of the two-stage fusion module) is obtained by weighted summation using gating weights:

[0103]

[0104] The gating network adaptively adjusts the weights of the two modalities, automatically reducing the contribution of visual features when noise is high, thus improving model robustness. Furthermore, the visual branch not only outputs purified visual feature vectors... The intermediate feature maps of Stage 2 and Stage 3 are further enhanced by the FiLM module in the first stage, which replaces the original feature maps and continues to flow to subsequent network layers (including multi-scale feature fusion and mask global pooling). See the description of the first stage for details.

[0105] 3-6 Classification Layers and Output

[0106] The fused features The input is a fully connected layer (with Softmax activation), and the output is a binary classification probability, representing "plaque risk" and "no plaque risk," respectively. The entire network is jointly trained end-to-end to achieve optimal fusion of imaging and clinical information.

[0107] Step 4: Anatomy-guided training and optimization of the multimodal fusion network model

[0108] Using the multimodal dataset constructed in step 1, the AG-MMNet network designed in step 2 is trained end-to-end, and the generalization ability and robustness of the model are improved through optimization strategies.

[0109] 4-1. Dataset Partitioning

[0110] Testing on multiple models demonstrated that the extracted feature combinations possess universal predictive power, and their effectiveness is not limited to specific model structures. To ensure the generalization of the evaluation and prevent data leakage, a strict subject-level hierarchical independent partitioning strategy was adopted. All multi-view images and electronic health records of the same subject were forcibly grouped into the same subset. The dataset was divided into a training set and an independent test set in an 8:2 ratio, with the prevalence rate in both groups strictly controlled at 22.9%, consistent with the original cohort.

[0111] 4-2. Training Environment and Hyperparameter Settings

[0112] All experiments were conducted using the PyTorch deep learning framework on a single NVIDIA RTX 3080 Ti (12GB VRAM) GPU. During training, the AdamW optimizer was used with a weight decay of 0.05 and an initial learning rate of 5×10⁻⁻⁶. 5The system employs a linear warm-up and cosine annealing strategy for the first 10 rounds, training for a total of 100 rounds. Due to memory limitations, the batch size is set to 16, and gradient accumulation (with a cumulative step count of 2) is used to simulate the training effect of larger batches. Simultaneously, a global gradient pruning threshold of 0.5 is implemented to stabilize the joint training of heterogeneous networks. Furthermore, to improve the model's generalization ability to spatial transformations, data augmentation operations such as rotations ranging from -15° to 15°, horizontal and vertical flips are randomly applied to the input images.

[0113] Five metrics were used to evaluate model performance: accuracy (Acc), sensitivity (Sens), specificity (Spec), AUC (area under the ROC curve), and Matthews correlation coefficient (MCC). Acc represents the overall prediction accuracy, Sens measures the ability to detect positive samples, Spec measures the ability to identify negative samples, AUC reflects the model's ability to distinguish between positive and negative samples, and MCC comprehensively evaluates the true prediction performance under class imbalance. Furthermore, the maximum, upper quartile, median, lower quartile, and minimum values ​​of the results from multiple experiments (such as cross-validation) were statistically analyzed to describe the distribution of each metric and the model's stability.

[0114] 4-3. Composite Objective Function

[0115] To address the inherent problem of extreme imbalance between positive and negative samples in medical datasets, and the "modal collapse" phenomenon commonly seen in multimodal joint training, a composite objective function is designed:

[0116]

[0117] in The balance coefficient is set to 1.0 in this embodiment.

[0118] (1) Classification loss of the main task: √smooth weighted cross-entropy

[0119] For category The dynamic reweighting coefficient is defined as:

[0120]

[0121] in The total number of samples in the training set. For category The number of samples, This represents the total number of categories. The formula uses a square root operation to smooth the weight distribution, preventing the minority classes from having excessively large weights.

[0122] Introducing tag smoothing regularization, soft tags are defined as:

[0123] ,

[0124] in This is a smoothing factor.

[0125] The final main task loss is calculated as follows:

[0126]

[0127] in For batch size, This is the predicted probability output by the fusion branch.

[0128] (2) Auxiliary supervision loss: Single-modal regularization

[0129] To prevent the visual branch from dominating gradients during multimodal training and causing insufficient optimization of the clinical branch, a separate classification head (structure: LayerNorm → Fully Connected → ReLU → Dropout) is introduced after the clinical encoder (TabNet). The auxiliary loss is defined as the cross-entropy loss between the output of this branch and the true label.

[0130]

[0131] This loss, acting as a regularization term, forces clinical branches to learn discriminative features independently without relying on visual information, ensuring that clinical features can provide effective information gain during the fusion phase.

[0132] Step 5: Use the trained anatomy-guided multimodal fusion network model to infer the risk probability of carotid artery plaque from the external eye images and clinical data to be tested.

[0133] Experimental results

[0134] (1) Comparative analysis of clinical coding methods and ablation experiments

[0135] To determine the optimal configuration of each component in the three-tiered hierarchical multimodal architecture, this invention conducted ablation experiments on different fusion strategies, encoding methods, and vision modules. Table 1 shows the specific configurations and evaluation results.

[0136] Table 1 Comparative analysis and ablation experiment results

[0137]

[0138] In terms of fusion strategies, the concat operation outperforms the attention mechanism, with AUC increasing from 86.77% to 90.18% and MCC increasing from 51.20% to 58.28%, indicating that simple feature concatenation is more effective in this multimodal fusion scenario.

[0139] In terms of coding methods, the Hybrid Expert Model (MoE) significantly outperformed the Multilayer Perceptron (MLP), with AUC increasing from 86.99% to 90.64%, sensitivity from 60.78% to 69.39%, and MCC from 50.68% to 59.69%, validating the importance of stronger nonlinear coding capabilities for extracting discriminative clinical features.

[0140] Further ablation analysis was performed on the core modules of the visual branch: after removing feature-level modulation (No FiLM), the sensitivity and MCC decreased to 69.66% and 59.55%, respectively, confirming that the FiLM module plays a key role in dynamically modulating visual features based on clinical priors and improving the ability to identify positive samples; removing masked global pooling (No Masked Gap) caused the AUC to drop significantly to 86.45%, indicating that anatomical masks are indispensable in suppressing background noise and focusing on vascular texture in the scleral region; removing multi-scale feature fusion (No MMF) reduced the AUC to 89.98% and the sensitivity to 68.21%, indicating that the effective fusion of superficial high-frequency texture features and deep semantic features is crucial to the overall performance.

[0141] In summary, the complete AG-MMNet model achieved the best performance across all evaluation metrics (AUC 92.14%, MCC 64.64%, sensitivity 73.18%), fully validating the dissection guidance mechanism and hierarchical multimodal fusion strategy as the core design principles for achieving optimal performance.

[0142] (2) Quantitative classification performance

[0143] The overall classification performance of the AG-MMNet model proposed in this invention and the baseline model on the independent test set is shown in Table 2.

[0144] Table 2 Results of Single-Mode Comparison Experiment

[0145]

[0146] The unimodal evaluation results showed that while models using only electronic health records (SVM, MLP) exhibited high specificity (>88%), their sensitivity was extremely low (<14%), with a Matthews correlation coefficient close to zero, limiting their ability to identify plaque-positive patients. The random forest model, despite having an area under the curve (AUC) of 95.02%, had a sensitivity of only 4.94%, highlighting the potential for high AUC to mask low sensitivity under class imbalance conditions. TabNet's sensitivity of 20.87% remained far below clinical screening requirements. Deep learning models based solely on external eye images (ResNet50, SpectFormer-s) improved sensitivity to approximately 35%, but their AUC remained below 78%, indicating that a single ocular surface morphological feature is insufficient for reliable diagnosis. These results collectively reveal the inherent limitations of unimodal methods in plaque screening, highlighting the necessity of multimodal fusion.

[0147] (3) Comparative analysis of backbone network and fusion strategy

[0148] As shown in Table 3, the fusion strategy proposed in this invention (i.e., "two-stage fusion module combined with TabNet clinical encoder") achieved consistent performance improvements across different visual backbone networks. Compared to the traditional method of stitching together with a multilayer perceptron, the combination of gated fusion and TabNet encoding improved the area under the curve (AUC) of each model. Taking ResNet50 as an example, the AUC increased from 85.26% to 90.67%, and the sensitivity increased from 54.55% to 67.88%, demonstrating the effectiveness of the two-stage gating mechanism in managing heterogeneous data interaction and preventing modality suppression.

[0149] Furthermore, with TabNet fixed as the clinical encoder, different fusion strategies were compared. The traditional splicing combined with a multilayer perceptron approach outperformed attention-based fusion, achieving an area under the curve of 90.18% and a sensitivity of 70.59%, significantly higher than the latter's 86.77% and 62.81%, respectively. This indicates that direct feature splicing can better preserve cross-modal complementary information.

[0150] Within the improved group, frequency domain learning demonstrated a significant advantage. SpectFormer-s achieved the best performance (area under the curve 91.93%, sensitivity 72.82%), surpassing traditional convolutional networks and Swin Transformer, supporting the hypothesis that frequency domain learning can effectively capture high-frequency vascular texture features that are easily smoothed by conventional convolution.

[0151] Table 3. Experimental Results Comparing Fusion Strategies and Visual Backbone Networks

[0152]

[0153] 4-5 Model Interpretability and Mechanism Verification

[0154] (1) Thermograph and physical shielding experiment

[0155] To explore the anatomical basis of the model's decision-making, this invention first visualizes its attention using Grad-CAM. For example... Figure 2 As shown, the model's attention is primarily focused on the scleral region, highlighting its diagnostic importance. To further quantify the contributions of different anatomical regions, this invention conducted a physical occlusion experiment, the results of which are as follows: Figure 3 As shown, the input configuration using only the scleral region achieved nearly comparable performance to the full-image input in terms of AUC and accuracy, while its sensitivity was significantly better than the configuration using only the iris region. Furthermore, color image input consistently outperformed its corresponding grayscale image across all metrics. These findings collectively constitute a complete chain of evidence confirming that the AG-SpectFormer model's decisions are highly dependent on the microvascular morphology (texture) and congestion status (color) of the scleral region.

[0156] (2) Texture Preservation Analysis

[0157] To further explore how frequency domain modeling affects the perception of vascular texture, this invention visualizes the feature activation maps of different backbone networks at multiple stages, such as... Figure 4 As shown, the activation maps of ResNet-50 in the first two layers appear diffuse and spatially smooth, failing to capture fine microvascular details. In contrast, AG-SpectFormer maintains clear and vessel-aligned activation patterns in both stages (stage 1 and stage 2), significantly highlighting the elongated branching structures in the bulbar conjunctival microvessels. This qualitative comparison supports the ability of frequency domain learning to effectively preserve high-frequency vascular features, which are typically suppressed by traditional convolutional filters.

[0158] Further analysis revealed that within the feature space of Stage 3, there existed a subset of clinically healthy individuals located near diseased samples; this invention defines these as the "borderline healthy population." This indicates that the model captures a continuous risk spectrum, rather than a simple binary classification. Calculating the Euclidean distance from each group to the healthy control center showed that the distance between the borderline healthy group and the healthy center was significantly greater than that between the core healthy group and the healthy control center. Furthermore, the group's condition was similar to that of the diseased group, forming a clear gradient among the three groups (core healthy group: Borderline healthy group: ; Disease group: This gradient validates the model's ability to characterize continuous risk. In summary, AG-MMNet's identification of critical samples reflects its advantage in sensing early pathological changes and has the potential to identify individuals with hidden risks and achieve refined stratification.

[0159] In summary, this invention, by constructing a complete technical process from data acquisition, anatomical mask generation, multimodal fusion network construction to risk prediction, successfully verified the feasibility and efficiency of non-invasive auxiliary diagnosis of carotid artery plaques using external eye images and electronic health record data, providing strong technical support for early screening of cardiovascular and cerebrovascular diseases and their application in primary healthcare.

[0160] This application also provides a carotid plaque auxiliary detection device based on multimodal fusion of external eye images, including the following modules:

[0161] Dataset construction module: used to collect external eye image data and corresponding electronic health record data of clinical populations undergoing carotid plaque assessment to construct a multimodal dataset.

[0162] Offline preprocessing module: Generates anatomical masks through offline preprocessing to obtain clean ocular surface images.

[0163] Model building module: Used to build the Anatomy-Guided Multimodal Fusion Network AG-MMNet, including a visual branch, a clinical branch, a two-stage fusion module, and a classification layer. Among them, the visual branch adopts an improved frequency-domain aware backbone network, called Anatomy-Guided Spectral Former (AG-SpectFormer).

[0164] Training module: Training and optimization of multimodal fusion network models based on the constructed dataset under dissection guidance.

[0165] Risk prediction module: Used to infer the risk probability of carotid artery plaques from the external eye images and clinical data to be tested using a trained anatomy-guided multimodal fusion network model.

[0166] In one possible implementation, the dataset construction module is specifically implemented as follows: A scleral shadowless imaging system is used to acquire external eye images of the subject's left eye in five standard fixation directions. The images are stored in RGB format with an original resolution of 960×640 pixels. After acquisition, samples with defocus, closed eyes, or severe occlusion are removed. Simultaneously, the subject's electronic health records are collected, including age, gender, height, weight, systolic blood pressure, and diastolic blood pressure, and three derived features—body mass index, pulse pressure, and mean arterial pressure—are calculated. All continuous variables are standardized using Z-Score, and missing values ​​are filled with the number of digits in the training set, ultimately forming a 9-dimensional clinical feature vector.

[0167] In addition, a unique identifier (UUID) is assigned to each subject to link all external eye images and electronic health record data of the same subject, ensuring no data leakage during subsequent dataset partitioning.

[0168] In one possible implementation, the offline preprocessing module is specifically implemented as follows: Several images are randomly selected from the dataset, and pixel-level annotation tools are used to annotate the background, sclera, and iris regions of each image to form a training subset; a lightweight U-Net segmentation network is trained based on this training subset; the trained lightweight U-Net segmentation network is used as an offline preprocessing module for anatomical mask preprocessing, inference is performed on the original external eye images in the dataset, and a binary anatomical mask is generated; the original external eye images and the anatomical mask are multiplied pixel-by-pixel, and the background region pixels are forcibly set to zero to obtain a pure ocular surface image that retains only the sclera and iris.

[0169] In one possible implementation, the frequency-domain aware backbone network comprises three stages: the shallow layers (Stage 1 and Stage 2) output high-resolution feature maps, incorporating spectral gating units to preserve high-frequency microvascular texture; the deep layers (Stage 3) output low-resolution feature maps, employing self-attention modules to capture global semantic features. An anatomy-guided multi-scale aggregator processes the outputs of Stage 2 and Stage 3 to fuse shallow texture features with deep semantic features, and performs masked global pooling using an anatomical mask to ultimately generate a purified visual feature vector. The clinical branch employs a TabNet network based on a sequential attention mechanism to perform instance-level feature encoding on electronic health record data, generating high-dimensional clinical feature embeddings.

[0170] In one possible implementation, after obtaining features from both the visual and clinical branches, a two-stage fusion module performs multimodal interaction. The two-stage fusion module consists of feature-level modulation and decision-level gating: the first stage is feature-level modulation, which uses the FiLM mechanism, conditioned on clinical feature embeddings, to perform channel-level affine transformations on the feature maps output from Stage 2 and Stage 3 of the visual branch, achieving dynamic enhancement of visual features. The enhanced feature maps replace the original feature maps and continue flowing to subsequent network layers in the visual branch (including multi-scale feature fusion and masked global pooling); the second stage is decision-level gating, which uses a learnable gating network to dynamically calculate the fusion weights of the purified visual feature vectors and clinical feature embeddings in the final decision, and after weighted summation, outputs the final disease prediction probability through a classification layer.

[0171] In one possible implementation, image preprocessing is performed on the training data before training the multimodal fusion network, including spatial and structural processing and numerical normalization. The spatial and structural processing includes converting the clean ocular surface image obtained after offline preprocessing in step 2 into a standard RGB three-channel format; downsampling to 224×224 resolution through random region cropping; performing random horizontal flipping with a 50% probability; performing random rotation; and perturbing brightness and contrast to enhance generalization ability. The numerical normalization uses the mean and standard deviation of the ImageNet dataset for cross-channel normalization, first scaling the pixel values ​​from [0,255] to [0,1], and then using the RGB three-channel mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225], according to... The formula adjusts the pixel value distribution to a standard normal distribution range centered at zero.

[0172] In one possible implementation, the training module divides the multimodal dataset into a training set and an independent test set in an 8:2 ratio based on the subject's unique identifier (UUID), and performs hyperparameter tuning within the training set using hierarchical 5-fold cross-validation. The optimizer uses AdamW with an initial learning rate of 5×10⁻⁻⁶. 5 The training employs a combination of linear preheating and cosine annealing strategies; a batch size of 16; and gradient accumulation with a global gradient pruning threshold of 0.5. Training utilizes a composite objective function, including a primary task classification loss (square root smoothed weighted cross-entropy, used to alleviate class imbalance) and an auxiliary supervision loss (calculated based on clinical branch outputs, used to prevent modality collapse).

[0173] In one possible implementation, the risk prediction module is specifically implemented as follows: for the subject to be screened, the acquired multi-directional external eye images are preprocessed with anatomical masks and then input together with the corresponding standardized clinical feature vectors into the trained AG-MMNet model to output the carotid plaque risk probability.

[0174] This application also provides an electronic device, including a processor and a memory.

[0175] The memory is used to store computer programs.

[0176] When the processor executes a program stored in the memory, it implements any of the methods described in this application.

[0177] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0178] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0179] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0180] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0181] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0182] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.

[0183] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.

[0184] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0185] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0186] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0187] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A carotid plaque auxiliary detection method based on multi-modal fusion of external eye images, characterized in that, Includes the following steps: First, external eye image data and corresponding electronic health record data were collected to construct a multimodal dataset; Then, an anatomical mask is generated through offline preprocessing to obtain a clean ocular surface image; An anatomy-guided multimodal fusion network was constructed, including a visual branch, a clinical branch, a two-stage fusion module, and a classification layer; among which, the visual branch adopted an improved frequency domain perception backbone network. Anatomy-guided training and optimization of multimodal fusion network models based on multimodal datasets Finally, the trained anatomy-guided multimodal fusion network model is used to infer the risk probability of carotid artery plaque from the external eye images and clinical data to be tested.

2. The carotid plaque auxiliary detection method based on multi-modal fusion of external eye images according to claim 1, characterized in that, The construction of the multimodal dataset is specifically implemented as follows: A scleral shadowless imaging system is used to acquire images of the subject's left eye in five standard gaze directions. The images are stored in RGB format with an original resolution of 960×640 pixels. After acquisition, samples with defocus, closed eyes, or severe occlusion are removed. Simultaneously, the subject's electronic health records are collected, including age, gender, height, weight, systolic blood pressure, and diastolic blood pressure, and three derived features—body mass index, pulse pressure, and mean arterial pressure—are calculated. All continuous variables were standardized using Z-Score, and missing values ​​were filled with the number of bits in the training set, ultimately forming a 9-dimensional clinical feature vector. In addition, each subject was assigned a unique identifier to link all external eye images and electronic health record data of the same subject.

3. The carotid plaque auxiliary detection method based on multi-modal fusion of external eye images according to claim 1, characterized in that, The offline preprocessing is implemented as follows: Several images are randomly selected from the dataset, and pixel-level annotation tools are used to annotate the background, sclera, and iris regions of each image to form a training subset; a lightweight U-Net segmentation network is trained based on this training subset; the trained lightweight U-Net segmentation network is used as an offline preprocessing module for anatomical mask preprocessing, and inference is performed on the original external eye images in the dataset to generate a binary anatomical mask; the original external eye images and the anatomical mask are multiplied pixel by pixel, and the background region pixels are forced to zero to obtain a pure ocular surface image that retains only the sclera and iris.

4. The carotid plaque auxiliary detection method based on multi-modal fusion of external eye images according to claim 1, characterized in that, The frequency domain sensing backbone network consists of three stages: the shallow layers, namely Stage 1 and Stage 2, output high-resolution feature maps and introduce spectral gating units to preserve the high-frequency texture of microvessels; the deep layers, namely Stage 3, output low-resolution feature maps and use self-attention modules to capture global semantic features. The anatomically guided multi-scale aggregator processes the outputs of Stage 2 and Stage 3 to fuse shallow texture features with deep semantic features, and performs masked global pooling in conjunction with the anatomical mask to finally generate a purified visual feature vector. The clinical branch uses a TabNet network based on the sequential attention mechanism to perform instance-level feature encoding on electronic health record data to generate high-dimensional clinical feature embeddings.

5. The carotid plaque auxiliary detection method based on multi-modal fusion of external eye images according to claim 4, characterized in that, After obtaining features from both the visual and clinical branches, a two-stage fusion module performs multimodal interaction. This two-stage fusion module consists of feature-level modulation and decision-level gating: The first stage is feature-level modulation, which uses the FiLM mechanism with clinical feature embedding as a condition to perform channel-level affine transformation on the feature maps output from Stage 2 and Stage 3 of the visual branch, thereby dynamically enhancing the visual features. The enhanced feature maps replace the original feature maps and continue to flow to subsequent network layers of the visual branch. The second stage is decision-level gating, which uses a learnable gating network to dynamically calculate the fusion weights of the purified visual feature vector and the clinical feature embedding in the final decision. After weighted summation, the final disease prediction probability is output through the classification layer.

6. The carotid plaque auxiliary detection method based on multi-modal fusion of external eye images according to claim 1, characterized in that, Before training the multimodal fusion network, image preprocessing is performed on the training data, including spatial and structural processing and numerical standardization. The spatial and structural processing includes converting clean ocular surface images to a standard RGB three-channel format; downsampling to 224×224 resolution through random region cropping; performing random horizontal flipping with a 50% probability; performing random rotation; and perturbing brightness and contrast to enhance generalization ability. The numerical standardization uses the mean and standard deviation of the ImageNet dataset for cross-channel normalization, first scaling pixel values ​​from [0,255] to [0,1], then using the RGB three-channel mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225], and then... The formula adjusts the pixel value distribution to a standard normal distribution range centered at zero.

7. The carotid plaque auxiliary detection method based on multi-modal fusion of external eye images according to claim 6, characterized in that, In model training, the multimodal dataset is divided into training set and independent test set proportionally based on the unique identifier of the subject. Within the training set, hierarchical 5-fold cross-validation is used for hyperparameter tuning. The optimizer is AdamW, combined with linear warm-up and cosine annealing strategies, and gradient accumulation and global gradient clipping are employed. The training uses a composite objective function, which includes the main task classification loss and the auxiliary supervision loss.

8. The carotid plaque auxiliary detection method based on multi-modal fusion of external eye images according to claim 1, characterized in that, The reasoning is specifically implemented as follows: For the subjects to be screened, the multi-directional external eye images are preprocessed with anatomical masks and then input together with the corresponding standardized clinical feature vectors into the trained AG-MMNet model to output the carotid plaque risk probability.

9. An external eye image-based multi-modal fusion carotid plaque auxiliary detection device, characterized in that, Includes the following modules: Dataset construction module: used to collect external eye image data and corresponding electronic health record data to construct a multimodal dataset; Offline preprocessing module: Generates anatomical masks through offline preprocessing to obtain clean ocular surface images; Model building module: used to build the anatomically guided multimodal fusion network AG-MMNet, including a visual branch, a clinical branch, a two-stage fusion module, and a classification layer; among which, the visual branch adopts an improved frequency domain perception backbone network; Training module: Training and optimization of multimodal fusion network models based on the constructed dataset using anatomy-guided methods; Risk prediction module: Used to infer the risk probability of carotid artery plaques from the external eye images and clinical data to be tested using a trained anatomy-guided multimodal fusion network model.

10. An electronic device, comprising: Including processor and memory; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the carotid plaque auxiliary detection method according to any one of claims 1-8.

Images