Epilepsy prediction system based on multi-modal biological images and image data processing method

Abstract

The present application relates to the technical field of intelligent medical treatment, and particularly relates to a seizure prediction system based on multi-modal biological images and an image data processing method. The present application first collects multi-modal images, then extracts features from the multi-modal images respectively, then aggregates image information by using an attention pooling mechanism, then weights and fuses feature vectors of different modes by using a fusion module based on a gated attention mechanism to generate a unified multi-modal feature vector as an input of a prediction module, so that the seizure can be predicted based on the multi-modal feature vector. Through the fusion of multi-modal information, the complementary features of eye images, tongue images and face images are fully utilized, the accuracy and robustness of prediction are significantly improved, and the present application has a good clinical application prospect.

Description

Technical Field

[0001] This invention relates to the technical field of epilepsy prediction in smart healthcare, and in particular to an epilepsy prediction system and image data processing method based on multimodal biological images. Background Technology

[0002] Epilepsy is a chronic neurological disorder caused by sudden, abnormal, and hypersynchronous discharges of neuronal groups in the brain, leading to transient brain dysfunction. Its seizures are characterized by randomness, repetition, and stereotyped patterns, placing a heavy burden on the lives and health of patients. Epilepsy not only causes sudden loss of control over physical and cognitive functions but can also trigger mental health problems such as anxiety or depression, and in extreme cases, even sudden epileptic death (SUDEP). Delayed diagnosis of epilepsy is common, increasing the risk of accidental injuries (such as falls, burns, and drowning) and increasing the risk of SUDEP by 2-3 times, placing a heavy burden on patients, families, and society. Therefore, early and accurate diagnosis of epilepsy is crucial for improving patient prognosis.

[0003] Currently, early diagnosis of epilepsy mainly relies on detailed medical history and eyewitness descriptions. However, the sudden, transient, and diverse nature of epileptic seizures poses a significant challenge to diagnosis, especially for patients with low seizure frequency, nocturnal seizures, or atypical symptoms, which can easily lead to missed or misdiagnosis. With technological advancements, modern medical technologies such as electroencephalography (EEG), neuroimaging (e.g., MRI), and genetic testing can provide objective evidence for diagnosis. Among these, EEG signals are considered the gold standard for monitoring epilepsy.

[0004] For example, invention patent CN118452948B discloses a method for preprocessing EEG data for auxiliary diagnosis of epilepsy, including: acquiring several EEG signals from an epilepsy patient and key EEG signal values; acquiring the final epilepsy characteristic values ​​of the key EEG signal values; obtaining EEG signals for mild and severe epilepsy based on the fluctuations of the final epilepsy characteristic values ​​of the key EEG signal values; acquiring epilepsy characteristic values ​​of the mild and severe epilepsy EEG signals; correcting the key EEG signal values ​​based on the final epilepsy characteristic values ​​and constructing an original EEG signal matrix; acquiring the source signal matrix and performing dimensionality reduction processing to obtain several dimensionality-reduced source signals. This method can improve the reflection of epilepsy characteristics in the dimensionality-reduced source signals while reducing the high dimensionality of the data.

[0005] For example, the invention patent with announcement number CN118557145B discloses an automatic epilepsy signal recognition system and method based on multimodal information fusion. The EEG acquisition unit is used to acquire the patient's EEG signals in real time; the video acquisition unit is used to simultaneously acquire the patient's monitoring video signals; the data preprocessing unit performs noise reduction processing on the EEG signals and monitoring video signals, and adds the epileptic seizure start and end timestamps to the EEG signals and monitoring video signals for time-series calibration to obtain epileptic EEG data and epileptic video data; the multi-scale convolutional network uses deep learning feature extraction methods to combine the time domain, frequency domain, time-frequency domain, and spatiotemporal domain features of the epileptic EEG data to obtain multi-feature vectors of epileptic EEG; the multi-level feature pyramid network processes the data using a spatial pyramid pooling method. The method obtains multi-scale features of epilepsy video data from epilepsy video data; the human optical flow estimation module obtains epilepsy video optical flow vectors by capturing epilepsy video data sequences; the multimodal feature fusion module uses a multilayer perceptron mapping method to fuse epilepsy EEG multi-feature vectors, epilepsy video multi-scale features, and video optical flow vectors to obtain epilepsy fusion feature data; the epilepsy signal classification module uses a bidirectional long short-term memory network with multiple attention mechanisms to identify epilepsy fusion feature data to obtain epilepsy signal categories; the epilepsy data automatic recognition model optimization module uses a cross-entropy loss function to train the epilepsy signal automatic recognition model. This method makes full use of various data from patients undergoing epilepsy detection, uses multimodal information of different dimensions to capture the spatiotemporal correlation of epilepsy signals, and improves the robustness of the system.

[0006] However, EEG signal monitoring generally suffers from problems such as a short detection window, limited positive rate (e.g., the EEG positive rate is often below 11%), high cost, and demanding equipment requirements. These limitations are particularly pronounced in areas with relatively scarce medical resources, severely restricting the widespread and timely diagnosis of epilepsy and highlighting the urgent clinical need to develop new low-cost, high-efficiency, and easy-to-operate non-invasive auxiliary diagnostic technologies.

[0007] In view of this, existing technologies have proposed detection methods based on non-EEG signals, i.e., based on physiological signals. For example, invention patent CN113907707B discloses a system and method for detecting epileptic seizure time periods based on multimodal information. The detection system includes four modules: physiological signal acquisition module A, preprocessing module B, feature extraction module C, and classification and recognition module D. Based on these four modules, multimodal physiological signals during and outside seizure periods of epileptic patients are first acquired; then, the multimodal physiological signals are divided into moving windows according to time length and quality screening is performed; the seizure and non-seizure time periods are initially identified based on accelerometer signals, and signal processing is performed to remove noise interference; then, the time domain and frequency domain features of each modality signal in the moving window are extracted; finally, cost-sensitive support vector machines are used for recognition and classification in combination with cross-validation. It uses multimodal physiological signals, such as acceleration, electromyography signals, and skin conductance signals, to characterize the physiological changes during epileptic seizure time periods.

[0008] In summary, while existing epilepsy diagnostic technologies have improved accuracy and efficiency to some extent, significant limitations remain. Electroencephalography (EEG), as the gold standard for epilepsy diagnosis, suffers from a short detection window, limited positive rate (often below 11%), high cost, and demanding equipment requirements—problems particularly acute in regions with limited medical resources. Neuroimaging (such as MRI) and genetic testing, while providing auxiliary diagnostic information, also face challenges including high cost, complex operation, demanding equipment requirements, and limited positive rates. Summary of the Invention

[0009] The purpose of this invention is to provide an epilepsy prediction system and image data processing method based on multimodal biological images, which partially solves or alleviates the above-mentioned shortcomings in the prior art. The system predicts epilepsy based on non-EEG signals (such as predicting whether the current user is an epilepsy patient). Specifically, it collects and analyzes multimodal biological image data such as eye images, tongue images, and facial images, and uses deep learning technology to achieve non-invasive and efficient prediction of epilepsy. It has higher accuracy and clinical operability, and provides a new technical path for the early screening of epilepsy.

[0010] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution:

[0011] A first aspect of the present invention is to provide a method for processing multimodal biological image data in epilepsy detection, characterized by comprising the following steps:

[0012] Acquire multimodal image data, which includes: an eye image set: including first images of multiple designated regions of the eyes, the first images including visible light images; a tongue image set: including first images and infrared images of the tongue surface and first images and infrared images of the underside of the tongue; and a face image set: including first images and infrared images of the face.

[0013] Feature extraction is performed on the eye image set, tongue image set, and face image set respectively to obtain eye feature vector, tongue feature vector, and face feature vector;

[0014] The eye feature vector and the tongue feature vector are aggregated using an attention-weighted aggregation mechanism to obtain aggregated feature vectors for the eye and tongue respectively.

[0015] The aggregated feature vectors of the eyes, tongue, and face are fused together, and a fusion weight for each modality is obtained through a gating attention mechanism. A joint feature vector is obtained based on the fusion weights. The joint feature vector is used as input data for an epilepsy prediction model to predict epilepsy.

[0016] Furthermore, the feature extraction from the tongue image set and the facial image set includes:

[0017] Input the RGB channels of the tongue image set / face image set into the pre-trained ResNet model to extract the feature vectors of the RGB channels;

[0018] The infrared channels of the tongue image set / face image set are input into a randomly initialized feature extraction network to extract the feature vectors of the infrared channels;

[0019] The feature vectors of the RGB channels and infrared channels of the tongue image set and the face image set are fused respectively to generate the fused tongue feature vector and face feature vector.

[0020] Furthermore, the attention-weighted aggregation mechanism is specifically as follows:

[0021] Attention weights are calculated for each feature vector using an attention pooling mechanism. :

[0022] ;

[0023] Among them, h i h is the feature vector of the i-th image; jLet be the feature vector of the j-th image; w be the learnable weight vector used to calculate the similarity score of each feature vector; V be the learnable weight matrix used to map the feature vectors to the hidden space; tanh(·) be the activation function; and n be the number of images.

[0024] Then, the aggregated global feature representation is obtained by weighted summation:

[0025] ;

[0026] Among them, h MIL This represents the aggregated global feature representation.

[0027] Furthermore, the fusion weights for each modality obtained through the gating attention mechanism include:

[0028] The feature vector of each modality is input into a gated attention network, which includes at least one linear layer and a sigmoid activation function to generate weights ranging from 0 to 1.

[0029] The generated weights are normalized so that the sum of the weights of all modes is 1.

[0030] The feature vectors of each modality are weighted and summed using normalized weights to generate a multimodal fusion feature vector.

[0031] Furthermore, the eye feature vector, tongue feature vector, facial feature vector, and multimodal fusion feature vector are concatenated to form the final joint feature vector.

[0032] Furthermore, the method outputs epilepsy prediction results through a pre-trained epilepsy prediction model, the training part of which includes:

[0033] The encoder takes multimodal image data as input and outputs eye feature vectors, tongue feature vectors, and facial feature vectors.

[0034] The attention-weighted aggregation module takes either an eye feature vector or a tongue feature vector as input and outputs either an aggregated feature vector of the eye or an aggregated feature vector of the tongue.

[0035] The fusion layer takes as input the aggregated feature vectors of the eyes, the tongue, and the face, and outputs a joint feature vector.

[0036] The classifier takes a joint feature vector as input and outputs an epilepsy prediction result (such as an epilepsy classification result or an epilepsy prediction probability value).

[0037] Furthermore, the epilepsy prediction model is trained using category-weighted cross-entropy loss:

[0038] ;

[0039] in, The category weights for epilepsy; The category weights are for non-epilepsy classes; Let y be the true label of the sample, and y∈{0,1}; The predicted probability output by the model;

[0040] The category weights are obtained as follows:

[0041] ;

[0042] in, Indicates category weight; This represents the total number of samples in the training set; This represents the number of samples in the k-th class.

[0043] Furthermore, a two-stage training strategy is used for model training, including:

[0044] In the first stage, the weight parameters of all modules except the classifier are frozen, and only the classifier is trained;

[0045] In the second phase, all modules are unfrozen, learning rates are set for different modules, and regularization and early stopping strategies are introduced for training.

[0046] Furthermore, the visible light images of the multiple designated regions of the binoculars include:

[0047] Visible light images viewed from the front, directly above, directly below, directly to the left, and directly to the right.

[0048] Secondly, this application also discloses an epilepsy prediction system based on multimodal biological images, comprising:

[0049] The data acquisition module is configured to acquire multimodal image data, which includes: an eye image set including first images of multiple designated regions of the eyes, the first images including visible light images; a tongue image set including first images and infrared images of the tongue surface and first images and infrared images of the underside of the tongue; and a face image set including first images and infrared images of the face.

[0050] The feature extraction module is configured to extract features from the eye image set, tongue image set, and face image set respectively, to obtain eye feature vectors, tongue feature vectors, and face feature vectors;

[0051] The weighted aggregation module is configured to use an attention-weighted aggregation mechanism to aggregate the eye feature vector and the tongue feature vector respectively, to obtain the aggregated feature vector of the eye and the aggregated feature vector of the tongue.

[0052] The fusion module is configured to perform fusion processing based on the aggregated feature vectors of the eyes, the aggregated feature vectors of the tongue, and the facial feature vectors, and to obtain the fusion weights of each modality through a gating attention mechanism, and to perform fusion processing based on the fusion weights to obtain a joint feature vector;

[0053] The prediction module is configured to input the joint feature vector into a pre-trained epilepsy prediction model (such as the classifier mentioned above) to output epilepsy prediction results.

[0054] Furthermore, the feature extraction module specifically includes:

[0055] The first feature extraction unit is configured to input the RGB channels of the tongue image set / face image set into a pre-trained ResNet model to extract the feature vectors of the RGB channels;

[0056] The second feature extraction unit is configured to input the infrared channels of the tongue image set / face image set into a randomly initialized feature extraction network to extract the feature vectors of the infrared channels;

[0057] The feature fusion unit is configured to fuse the feature vectors of the RGB channels and the feature vectors of the infrared channels of the tongue image set and the face image set, respectively, to generate fused tongue feature vectors and face feature vectors.

[0058] Furthermore, the attention-weighted aggregation mechanism is specifically as follows:

[0059] Attention weights are calculated for each feature vector using an attention pooling mechanism. ;

[0060] ;

[0061] Among them, h i h is the feature vector of the i-th image; j Let be the feature vector of the j-th image; w be the learnable weight vector used to calculate the similarity score of each feature vector; V be the learnable weight matrix used to map the feature vectors to the hidden space; tanh(·) be the activation function; and n be the number of images.

[0062] Then, the aggregated global feature representation is obtained by weighted summation:

[0063] ;

[0064] Among them, hMIL This represents the aggregated global feature representation.

[0065] Furthermore, the fusion module specifically includes:

[0066] The weight processing unit is configured to input the feature vector of each modality into a gated attention network, the gated attention network including at least one linear layer and a sigmoid activation function to generate weights ranging from 0 to 1;

[0067] The normalization unit is configured to normalize the generated weights so that the sum of the weights of all modes is 1.

[0068] The feature fusion unit is configured to use normalized weights to perform a weighted summation of the feature vectors of each modality to generate a multimodal fusion feature vector.

[0069] Beneficial Effects: Electroencephalography (EEG) is the gold standard for epilepsy diagnosis, but it suffers from a short detection window, limited positive rate (often below 11%), high cost, and demanding equipment requirements. Existing technologies have also proposed some methods based on physiological signals during epileptic seizures, such as acceleration, electromyography, and skin conductance, but these physiological signals are actually quite difficult to collect. Furthermore, in addition to these physiological signals, the external facial features of epilepsy patients and healthy patients differ significantly. For example, the physiological characteristics of the periocular / eyelid color and vascular network in epilepsy patients are very different from those in healthy individuals; similarly, the tongue color, texture, and coating distribution in epilepsy patients differ from those in healthy individuals; and the facial color and texture distribution in epilepsy patients also differ. Therefore, this application proposes a novel system for detecting epilepsy based on the physiological characteristics of the tongue, face, and eyes, along with its corresponding image processing method. Compared to existing technologies, it is less expensive.

[0070] 1. By collecting non-invasive bio-image data such as eye images, tongue images, and facial images, the discomfort and risks that traditional invasive detection methods such as electroencephalography (EEG) may bring to patients are avoided, thus improving patient acceptance and compliance.

[0071] 2. By fusing multimodal biological image data, the complementary nature of eye, tongue, and facial images is fully utilized (e.g., facial images have more features but may have lower information density, while eye and tongue images have relatively fewer features but may have higher information density), significantly improving the accuracy of epilepsy prediction. Compared to single-modal models, the multimodal fusion model maintains high sensitivity while greatly improving specificity and reducing the false positive rate. Furthermore, for facial and tongue images, visible light images provide easily identifiable physiological features, while infrared images provide subtle physiological features that are not easily identifiable (infrared images can also provide easily identifiable physiological features), thereby improving prediction accuracy.

[0072] 3. By utilizing multimodal biological image data, this method aggregates image information through attention pooling and employs a gated attention mechanism to weightedly fuse feature vectors from different modalities, thereby facilitating efficient prediction of epilepsy and advancing non-invasive auxiliary diagnosis. This method boasts high accuracy and specificity, enhances model interpretability, and provides an efficient and non-invasive solution for early epilepsy screening. As mentioned earlier, for the same epilepsy patient, compared to facial feature vectors (i.e., global features), the number of feature vectors for the eyes and tongue (i.e., local region features) is very small, but their information density may be higher than that of the face, making them indispensable. If they are not distinguished and the entire facial image is used directly for prediction and recognition, the feature vectors for the eyes and tongue may be diluted by background feature vectors (such as normal feature vectors) in the facial feature vectors, thus reducing prediction accuracy. Even if they are distinguished, directly fusing the feature vectors corresponding to the three modalities may still lead to the feature vectors for the eyes and tongue being diluted by background feature vectors (such as normal feature vectors) in the facial feature vectors, thus reducing prediction accuracy. Therefore, this application first aggregates the feature vectors of the eye and tongue images separately using an attention pooling mechanism to find key features (i.e., aggregated feature vectors) in the patient's corresponding region that reflect the epileptic pattern (i.e., distinguishes them from non-epilepsy patients / healthy individuals). Then, the aggregated feature vectors of the eye and tongue images are fused with the feature vector of the facial image. However, traditional feature fusion methods (such as concatenation, averaging, and weighted summation) typically use fixed weights or rules to combine features from different branches. In practical applications, different samples may have different levels of information and reliability in different locations. For example, different epilepsy patients may have different features reflected in their eyes, tongue, and face (for instance, epilepsy patient A's physiological features reflecting epileptic symptoms are more concentrated in the tongue and face, while epilepsy patient B's physiological features reflecting epileptic symptoms may be more concentrated in the eyes and tongue), and correspondingly, the amount of information and reliability provided by the eyes, tongue, and face also differs. Therefore, this application utilizes a gated attention mechanism to learn and generate dynamic weights for each branch of each sample (the aggregated feature vectors of the eye image, the aggregated feature vectors of the tongue image, and the feature vector of the face image). This allows the model to automatically adjust the degree of dependence on each branch according to the specific input situation (for example, patient A may mainly exhibit transient eyelid myoclonus; the gated mechanism allocates 90% of the weights to the aggregated feature vector of the eye image). This avoids the large prediction error caused by blurring individual differences after using fixed weights or fixed rules for feature processing.

[0073] 4. This method is based on bio-image data that is not derived from electroencephalograms (EEGs), resulting in lower equipment requirements and costs, making it easy to promote and apply in areas with relatively scarce medical resources. Furthermore, this system can be combined with other diagnostic methods to provide a more comprehensive basis for the integrated diagnosis of epilepsy. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0075] Figure 1 This is a flowchart of a multimodal biological image data processing method for epilepsy detection according to this application;

[0076] Figure 2 This is a schematic diagram of a multimodal attention fusion network model framework for epilepsy prediction in one embodiment of this application;

[0077] Figure 3 This is a schematic diagram of the accuracy variation curve of a multimodal fusion model in one embodiment of this application;

[0078] Figure 4 This is a schematic diagram of the loss function change curve of a multimodal fusion model during the training process in one embodiment of this application;

[0079] Figure 5 This is a schematic diagram of the receiver operating characteristic curve of a multimodal fusion model in one embodiment of this application;

[0080] Figure 6 This is a schematic diagram of the module structure of an epilepsy prediction system based on multimodal biological images in one embodiment of this application. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0082] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.

[0083] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0084] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0085] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0086] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0087] As used in this specification, the term "about" typically means + / -5% of the value, more typically + / -4%, more typically + / -3%, more typically + / -2%, even more typically + / -1%, even more typically + / -0.5%.

[0088] In this specification, certain embodiments may be disclosed in a range-bound format. It should be understood that this "range-bound" description is merely for convenience and brevity and should not be construed as a rigid limitation on the disclosed range. Therefore, the description of a range should be considered as having specifically disclosed all possible subranges and the individual numerical values ​​within those ranges. For example, a description of the range 1-6 should be considered as having specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and the individual numbers within those ranges, such as 1, 2, 3, 4, 5, and 6. This rule applies regardless of the breadth of the range.

[0089] Figure 1 A flowchart of the multimodal biological image data processing method for epilepsy prediction in this application is shown. (Refer to...) Figure 1 The method specifically includes the following steps:

[0090] S1. Acquire multimodal image data.

[0091] In some embodiments, the multimodal image data includes: an eye image set: a first image of multiple designated regions of both eyes; a tongue image set: a first image and an infrared image of the tongue surface, and a first image and an infrared image of the sublingual region; and a face image set: a first image and an infrared image of the face. Preferably, the first image includes a visible light image (visible light image in this document refers to an image that can acquire the numerical information of the three RGB channels of the image, and is not limited to its format), such as an RGB image, a JPG image, or an image in other formats. Subsequent embodiments will use RGB images as an example for illustration.

[0092] In one specific embodiment, the data acquisition process is as follows:

[0093] 1. Visual Diagnosis Objective Information Collection Process

[0094] Data collection equipment: Bio-Crystal Eye Diagnostic Instrument.

[0095] Data Acquisition Steps: Positioning: Instruct the subject to sit upright, placing their eye against the eye diagnostic instrument's support, ensuring each eye is centered on the lens. Exposing the Sclera: Instruct the subject to gently open the upper and lower eyelids with their fingers, ensuring full exposure of the inner and outer canthi, as well as the upper and lower sclera. Focusing and Imaging: The operator adjusts the focus until the eye image on the screen is clear. Multi-directional Acquisition: Instruct the subject to move their eyes sequentially to five positions: straight ahead, directly above, directly below, directly to the left, and directly to the right. After stabilizing in each position, the operator presses the shutter to take the picture. Binocular Acquisition: Following the above steps, complete the multi-directional acquisition for one eye first, then acquire data for the other eye, ensuring complete data for both eyes. Built-in Light Source: The entire acquisition process utilizes the eye diagnostic instrument's built-in shadowless imaging device, simulating natural light, eliminating interference from external light.

[0096] Data quality control: After each image is taken, the operator must check it immediately. For photos that are blurry or have incomplete exposure of the sclera due to eyelid obstruction, blinking, inaccurate focus, or improper rotation, they must be discarded immediately and retaken until a complete and clear set of visual examination images is obtained.

[0097] 2. Objective information collection process for tongue diagnosis

[0098] Data collection equipment: Bio-Crystal Tongue Diagnostic Instrument.

[0099] Data Acquisition Procedure: Equipment Setup: Turn on the power of the tongue diagnostic instrument and the D65 standard light source, and select the fixed shooting mode. Subject Positioning: Instruct the subject to sit upright in front of the instrument, leaning slightly forward, ensuring that the chin is firmly against the chin rest within the circle and the forehead is against the upper edge of the instrument. Exposure Confirmation: Instruct the subject to close their eyes. The operator confirms that the subject's face is fully exposed under the light source and checks the exposure indicator scale. Tongue Extension Requirements: Instruct the subject to open their mouth and extend their tongue naturally and relaxedly. The tongue surface should be flat, the tongue body relaxed, and the tip of the tongue naturally drooping. Excessive force or tense curling is not recommended. Focusing and Shooting: The operator fine-tunes the acquisition distance until the exposure compensation indicator (EV) scale reaches zero, and then clicks the center of the screen to complete the shooting.

[0100] Data quality control: After shooting, the operator must immediately check the image quality. Photos with problems such as tongue tension, incomplete tongue extension, severe reflection of saliva in the mouth, or blurry images must be discarded immediately, and the subject must be instructed to adjust and retake the photo until a qualified standard tongue image is obtained.

[0101] 3. Objective Information Collection Process for Face-to-Face Consultation

[0102] Facial diagnosis aims to standardize the acquisition of facial images of subjects, providing objective evidence for observing facial color, shape, etc. The acquisition equipment used is the Bio-Crystal Tongue and Face Diagnosis Instrument.

[0103] Data Acquisition Steps: Equipment Setup: Use the same equipment as for tongue diagnosis, turn on the D65 standard light source, and select the facial diagnosis shooting mode. Subject Positioning: The positioning method is exactly the same as for tongue diagnosis. Instruct the subject to sit upright in front of the instrument, leaning slightly forward, ensuring that the chin is firmly against the inner jaw support and the forehead is close to the upper edge of the instrument. Expression Requirements: Instruct the subject to completely relax their facial muscles, keep their lips naturally closed, and maintain a natural, expressionless state, with their eyes looking straight ahead or closed as instructed. Light Source and Shooting: The operator confirms that the face is evenly illuminated under the standard light source, without obvious shadows or reflections. After the exposure compensation indicator (EV) reaches zero, tap the screen to complete the shooting.

[0104] Data quality control: After shooting, the operator must immediately check the image quality. Photos with unnatural expressions, uneven lighting, closed eyes, blurry images, or misaligned positions must be discarded immediately, and the subject must be instructed to adjust and retake the photo until a qualified standard facial image is obtained.

[0105] In this embodiment, the acquired multimodal image data are images of epileptic patients during the interictal period, including an interictal eye image set: including first images of multiple designated areas of both eyes; an interictal tongue image set: including first images and infrared images of the tongue surface and first images and infrared images of the sublingual area; and an interictal facial image set: including first images and infrared images of the face.

[0106] S2. Extract features from the eye image set, tongue image set, and face image set respectively to obtain eye feature vector, tongue feature vector, and face feature vector.

[0107] In one specific embodiment, for tongue diagnosis and facial images, since both contain visible light and infrared images, this application designs a multispectral fusion encoder for feature extraction. Specifically, this application uses the infrared image as a fourth channel in addition to RGB, forming a four-channel input (RGB+I).

[0108] The RGB channels are initialized using a ResNet model pre-trained on ImageNet to leverage its powerful representation capabilities; the infrared channel (I) uses a randomly initialized feature extraction network and is optimized together with the RGB channels during training to achieve effective fusion of multispectral features.

[0109] S3. The eye feature vector and the tongue feature vector are aggregated using an attention-weighted aggregation mechanism to obtain aggregated feature vectors for the eye and tongue respectively.

[0110] In a specific embodiment, for cases where each sample (such as a test subject, or each healthy subject / each epileptic subject in the training sample) contains multiple images, especially eye and tongue diagnosis images, this application introduces a Multiple Instance Learning (MIL) framework to fully utilize the local information carried by each image. In MIL, each sample is regarded as a "bag" composed of several "instances," and the model learns to dynamically assign weights to each instance, thereby obtaining a discriminative representation of the entire sample.

[0111] Specifically, this application employs attention pooling as the MIL aggregation strategy. Let a sample contain... An image instance, whose features are represented as {h1, h2, ..., h...} n}, each Indicates the first Feature vectors of an image; Let represent a d-dimensional real vector space.

[0112] First, the attention weights for each instance are learned through a trainable attention network. :

[0113] ;

[0114] in, and These are learnable parameters; w is used to calculate the similarity score for each feature vector; V is used to map the feature vectors to the hidden space. Let h be an l-dimensional real vector space; i Let i be the feature vector of the i-th image. For Transpose; Indicates the first Feature vectors of an image; For Transpose; tanh is the activation function.

[0115] Finally, the aggregated package-level feature representation is obtained through weighted summation:

[0116] ;

[0117] The statement This represents the final feature of the sample in that modality, which can integrate the importance of each instance and capture the overall contribution of local image information to the prediction.

[0118] S4. Based on the aggregated feature vectors of the eyes, tongue, and face, perform fusion processing, and obtain the fusion weights of each modality through a gating attention mechanism. Based on the fusion weights, perform fusion processing to obtain a joint feature vector.

[0119] In one specific embodiment, during the multimodal feature fusion stage, this application designs a fusion module whose output includes four vectors: three single-modal feature vectors (eye, tongue, and face) and a fusion feature vector. This fusion feature vector is obtained by the fusion module through learning the normalized weights of each modality's features and then weighting and summing them, thereby capturing complementary information between different modalities.

[0120] Specifically, let the feature vectors of the three modalities of eye diagnosis, tongue diagnosis, and face diagnosis be represented as follows: , , The fusion module first learns the weights of the i-th mode. Furthermore, the weights of the three modes satisfy the normalization constraint:

[0121] ;

[0122] Then calculate the fused feature vector:

[0123] ;in, For example, the feature vector of the i-th mode. , , .

[0124] Subsequently, this application concatenates the three types of single-modal feature vectors with the fused feature vector to form a joint representation:

[0125] ;

[0126] The concatenated feature vector The data is input into the classification head, which consists of several fully connected layers and ultimately outputs a prediction result for epilepsy or non-epilepsy. For example, based on the concatenated features, a prediction probability value for epilepsy is obtained. If the prediction probability value is greater than a preset probability threshold, it is considered to be epilepsy, or the risk of epilepsy is high; if it is less than the preset probability threshold, it is considered not to be epilepsy, or the risk of epilepsy is low.

[0127] In another specific embodiment, the fused feature vector can also be obtained as follows:

[0128] For the eigenvector f of the i-th mode i Initialize a learnable weight vector w i ;

[0129] Calculate the dot product of the feature vector of the i-th mode and its corresponding weight vector to obtain the attention score S. i :

[0130] ;

[0131] The attention score is normalized to weights using the Softmax function, ensuring that the sum of all weights is 1;

[0132] The feature vectors of each modality are weighted and summed using normalized weights to generate a multimodal joint feature vector.

[0133] Based on the above method, this application also provides an embodiment of an epilepsy prediction method based on multimodal biological image data. Specifically, it includes the above steps S1-S4, but with the difference that it also includes step S5: inputting the joint feature vector in step S4 into the epilepsy prediction model to output the epilepsy prediction probability.

[0134] In one specific embodiment, this application designs and implements a deep learning network based on multimodal information fusion, the technical framework of which is as follows: Figure 2 As shown, this framework aims to predict or intelligently assist in the diagnosis of epilepsy by integrating biometric information from a patient's eye, tongue, and facial features. It consists of four core components: modality-specific feature extraction, attention-based feature aggregation, multimodal feature fusion, and classification prediction.

[0135] Input and Feature Extraction Module: The model uses three different biological visual data as input sources: (1) eye image modality, with an input of N1×224×224×3 RGB images; (2) tongue image modality, with an input of N2×224×224×4 multispectral images; (3) face image modality, with an input of N3×224×224×4 multispectral images; where N1, N2, and N3 are the number of images for the three modalities, respectively. In order to handle the characteristics of different modal data, this application designs an independent encoder branch for each input source. In this process, eye image data is processed using a standard convolutional neural network (CNN) encoder to extract features. Specifically, the visible light image of the eye is input into this encoder (i.e., a multimodal encoder) for feature extraction, resulting in feature sequence I. Tongue and face image data are processed using a multispectral encoder optimized for four-channel images. Correspondingly, the visible light image (e.g., RGB image) and infrared image of the tongue are input into this multispectral encoder for feature extraction, resulting in feature sequences II and III for the tongue, respectively. Similarly, the visible light image (e.g., RGB image) and infrared image of the face are input into this multispectral encoder for feature extraction, resulting in feature sequences IV and V for the tongue, respectively. In other words, these encoders map high-dimensional image data into a series of compact depth feature vectors.

[0136] Attention Feature Aggregation Module: To filter the most discriminative information from the feature sequences of each modality, the model introduces an attention pooling module in the eye and tongue image branches. This module learns the attention weights assigned to each feature vector through a small neural network (consisting of two linear layers and a Tanh activation function), and then performs a weighted summation of the feature sequences to generate a single aggregated feature vector with higher information density and better representative of the core features of that modality. For example, each feature vector in the tongue image feature sequences II-III is assigned a corresponding attention weight, and then the feature sequences II-III are weighted and summed; each feature vector in the eye image feature sequence I is assigned a corresponding attention weight, and then the feature sequences I are weighted and summed. It is worth noting that the feature vectors of the face image branch (such as all feature vectors in feature sequences IV-V) are directly fed into the next stage, a design that preserves their original global features.

[0137] Multimodal Attention Fusion Module: This is the core innovation of the model. Feature vectors from three branches (aggregated features from eye and tongue images, and direct features from facial images) are jointly input into an attention fusion layer. This fusion layer dynamically learns the relative importance of different modalities in the prediction of a specific sample through a gated attention mechanism. It generates a weight between 0 and 1 (using a sigmoid function) for each modality's feature vector, and then fuses the weighted features (e.g., through concatenation or element-wise addition) to finally generate a unified multimodal feature vector containing all the key information. For each modality's feature vector, its corresponding weight is calculated through a gated attention network. In some examples, this network may include: one or more fully connected layers (or convolutional layers, depending on the feature dimension); an activation function, such as ReLU, to introduce non-linearity; and a final sigmoid layer that converts the output into values ​​between 0 and 1, representing the relative importance of each modality. These weights are ensured to be non-negative by the Sigmoid function and can be further normalized by the softmax function so that the sum of all weights is 1.

[0138] Prediction Module: Finally, the deeply fused multimodal feature vector is passed to a Multi-Layer Perceptron (MLP) classifier. The MLP performs a non-linear transformation on the high-level features and outputs the final prediction probability (i.e., the epilepsy prediction probability). In other embodiments, the prediction module may further compare the predicted probability with a preset probability threshold (e.g., 85% or 93%) to determine whether the test subject is a person with epilepsy (PWE) or a healthy control group (HC).

[0139] Model training:

[0140] To effectively train the multimodal fusion model and alleviate overfitting under small sample conditions, this application adopts a phased training strategy, combined with appropriate optimizers and regularization methods, to improve the model's stability and generalization ability.

[0141] The model training employs a class-weighted cross-entropy loss function to mitigate training bias caused by imbalanced class distribution. This loss function is suitable for the binary classification task (epilepsy vs. non-epilepsy) in this application, introducing class weights on top of the standard cross-entropy to impose a higher loss penalty on classes with fewer samples.

[0142] Specifically, let the sample label be y∈{0,1}, and the predicted probability output by the model is:

[0143] ;in, denoted as the predicted probability output by the model, where x is the original output of the model, typically the output of the last fully connected layer of the neural network.

[0144] If the class weights are w0 (corresponding to the non-epilepsy class) and w1 (corresponding to the epilepsy class), then the loss function is defined as:

[0145] ;

[0146] Among them, category weight The calculation is based on the number of samples of each class in the training set, defined as follows:

[0147] ;

[0148] in:

[0149] Total number of samples in the training set (including (1) visible light images of the eye modality of healthy subjects; (2) multispectral images of the tongue modality of healthy subjects; (3) multispectral images of the facial modality of healthy subjects; and (4) visible light images of the eye modality of epileptic subjects; (5) multispectral images of the tongue modality of epileptic subjects; and (6) multispectral images of the facial modality of epileptic subjects).

[0150] : No. Number of samples per class (k=0 represents non-epilepsy or healthy subjects, k=1 represents epilepsy).

[0151] This weighting strategy can balance the contribution of each class to training in the loss function, avoiding excessive bias of the model towards the dominant class when the class distribution is unbalanced, thereby improving the model's discriminative ability and generalization performance.

[0152] In one specific embodiment, the model training adopts a two-stage training strategy, and the training process is divided into two stages:

[0153] Phase 1: Warm-up. The encoder is frozen, and only the classifier is trained. In this initial phase, the encoder, attention pooling module, and fusion layer are frozen, and only the parameters of the classifier head are updated. The Adam optimizer is used in this phase, with a learning rate set to [value missing]. Train for 10 epochs to ensure that the classifier can quickly adapt to the initial representation of multimodal features.

[0154] Phase Two: Full-Finetune. In this phase, the encoder, attention pooling module, and fusion layer are unfrozen, and the entire model is jointly trained. This phase employs the AdamW optimizer and sets the learning rates for different modules to achieve fine-grained training control.

[0155] Encoder learning rate: ;

[0156] Classification head learning rate: ;

[0157] To further mitigate overfitting, this application introduces a regularization strategy of random neuron deactivation (Dropout=0.6) and weight decay (weightdecay=0.05) during training. Simultaneously, an EarlyStopping strategy is employed, terminating training early if there is no improvement after 10 consecutive rounds on the validation set.

[0158] To effectively address the overfitting and low specificity issues commonly found in single-modal models, this application constructs a multimodal fusion model that fully utilizes the complementary features of eye, tongue, and facial images, significantly improving prediction accuracy and robustness. The results are as follows: Figures 3-5 As shown. First, the model training process ( Figure 3 and Figure 4 It exhibits excellent generalization ability, and its validation set accuracy and loss value maintain a highly synchronized optimization trend with the training set, fundamentally overcoming the overfitting phenomenon that occurs during single-modality training. Figure 5 The ROC curve in the model, with an area under the curve (AUC) value of 0.73, surpasses all single-modal models, providing decisive quantitative evidence for the superiority of the multimodal fusion strategy and proving that fusing complementary information from multiple sources is an effective way to improve the accuracy and reliability of epilepsy prediction.

[0159] This application constructs a multimodal deep learning method that integrates images from the eyes, tongue, and face to assist in the prediction of epilepsy. By introducing a multi-instance learning strategy, the model can fully utilize multiple image information from each site; simultaneously, by combining a multispectral fusion mechanism of visible light and infrared images, the ability to perceive subtle physiological changes is enhanced. Finally, the model achieves intelligent discrimination and prediction between individuals with and without epilepsy through feature fusion and weighted classification. This method provides a new technical path for non-invasive auxiliary prediction / early screening of epilepsy based on representative image features, and has good clinical scalability and interpretability.

[0160] Further reference Figure 6 Based on the above method, this application also provides an embodiment of an epilepsy prediction system based on multimodal biological image data, which is similar to... Figure 1 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0161] refer to Figure 6 An epilepsy prediction system based on multimodal biological images, comprising:

[0162] The data acquisition module 101 is configured to acquire multimodal image data, which includes: an eye image set including first images of multiple designated regions of the eyes, the first images including visible light images; a tongue image set including first images and infrared images of the tongue surface and first images and infrared images of the underside of the tongue; and a face image set including first images and infrared images of the face.

[0163] The feature extraction module 102 is configured to extract features from the eye image set, tongue image set, and face image set respectively to obtain eye feature vectors, tongue feature vectors, and face feature vectors; for example, the encoder in the classifier processes multimodal images to output eye feature vectors, tongue feature vectors, and face feature vectors.

[0164] The weighted aggregation module 103 is configured to use an attention-weighted aggregation mechanism to aggregate the eye feature vector and the tongue feature vector respectively, to obtain an aggregated feature vector of the eye and an aggregated feature vector of the tongue; for example, the attention-weighted aggregation module in the classifier takes an eye feature vector or a tongue feature vector as input and outputs an aggregated feature vector of the eye or an aggregated feature vector of the tongue.

[0165] The fusion module 104 is configured to perform fusion processing based on the aggregated feature vectors of the eyes, the aggregated feature vectors of the tongue, and the facial feature vectors, and to obtain the fusion weights of each modality through a gating attention mechanism. The fusion processing is then performed based on the fusion weights to obtain a joint feature vector. For example, the fusion layer in the classifier takes the aggregated feature vectors of the eyes, the aggregated feature vectors of the tongue, and the facial feature vectors as inputs and outputs a joint feature vector.

[0166] Prediction module 105 is configured to input the joint feature vector into a classifier to output epilepsy prediction results.

[0167] In some embodiments, the feature extraction module specifically includes:

[0168] The first feature extraction unit is configured to input the RGB channels of the tongue image set / face image set into a pre-trained ResNet model to extract the feature vectors of the RGB channels;

[0169] The second feature extraction unit is configured to input the infrared channels of the tongue image set / face image set into a randomly initialized feature extraction network to extract the feature vectors of the infrared channels;

[0170] The feature fusion unit is configured to fuse the feature vectors of the RGB channels and the feature vectors of the infrared channels of the tongue image set and the face image set, respectively, to generate fused tongue feature vectors and face feature vectors.

[0171] In some embodiments, the above attention-weighted aggregation mechanism is specifically as follows:

[0172] Attention weights are calculated for each feature vector using an attention pooling mechanism. ;

[0173] ;

[0174] Among them, h i h is the feature vector of the i-th image; j Let be the feature vector of the j-th image; w be the learnable weight vector used to calculate the similarity score of each feature vector; V be the learnable weight matrix used to map the feature vectors to the hidden space; tanh(·) be the activation function; and n be the number of images.

[0175] Then, the aggregated global feature representation is obtained by weighted summation:

[0176] ;

[0177] Among them, h MIL This represents the aggregated global feature representation.

[0178] In some embodiments, the fusion module specifically includes:

[0179] The weight processing unit is configured to input the feature vector of each modality into a gated attention network, the gated attention network including at least one linear layer and a sigmoid activation function to generate weights ranging from 0 to 1;

[0180] The normalization unit is configured to normalize the generated weights so that the sum of the weights of all modes is 1.

[0181] The feature fusion unit is configured to use normalized weights to perform a weighted summation of the feature vectors of each modality to generate a multimodal fusion feature vector.

[0182] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following... Figure 1 The method shown.

[0183] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0185] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An epilepsy prediction system based on multimodal biological images, characterized in that, include: The data acquisition module is configured to acquire multimodal image data, which includes: an eye image set of epileptic patients during interictal periods: including first images of multiple designated areas of both eyes, the first images including visible light images; a tongue image set of epileptic patients during interictal periods: including first images and infrared images of the tongue surface and first images and infrared images of the sublingual area; and a facial image set of epileptic patients during interictal periods: including first images and infrared images of the face. The feature extraction module is configured to extract features from the eye image set, tongue image set, and face image set respectively, to obtain eye feature vectors, tongue feature vectors, and face feature vectors; The weighted aggregation module is configured to use an attention-weighted aggregation mechanism to aggregate the eye feature vector and the tongue feature vector respectively, to obtain the aggregated feature vector of the eye and the aggregated feature vector of the tongue. The fusion module is configured to perform fusion processing based on the aggregated feature vector of the eye, the aggregated feature vector of the tongue, and the facial feature vector, and obtain the fusion weight of each modality through a gating attention mechanism, and perform fusion processing based on the fusion weight to obtain a joint feature vector; The prediction module is configured to input the joint feature vector into a pre-trained epilepsy prediction model to output an epilepsy prediction result; wherein, the epilepsy prediction model is trained using a class-weighted cross-entropy loss, the class-weighted cross-entropy loss being specifically as follows: ； in, The category weights for epilepsy; The category weights are for non-epilepsy classes; Let y be the true label of the sample, and y∈{0,1}; The predicted probability output by the model; The category weights are obtained as follows: ； in, Indicates category weight; This represents the total number of samples in the training set; This represents the number of samples in the k-th class.

2. The epilepsy prediction system based on multimodal biological images according to claim 1, characterized in that, The feature extraction module specifically includes: The first feature extraction unit is configured to input the RGB channels of the tongue image set / face image set into a pre-trained ResNet model to extract the feature vectors of the RGB channels; The second feature extraction unit is configured to input the infrared channels of the tongue image set / face image set into a randomly initialized feature extraction network to extract the feature vectors of the infrared channels; The feature fusion unit is configured to fuse the feature vectors of the RGB channels and the feature vectors of the infrared channels of the tongue image set and the face image set, respectively, to generate fused tongue feature vectors and face feature vectors.

3. The epilepsy prediction system based on multimodal biological images according to claim 1, characterized in that, The attention-weighted aggregation mechanism is specifically as follows: Attention weights are calculated for each feature vector using an attention pooling mechanism. : ； Among them, h i h is the feature vector of the i-th image; j Let be the feature vector of the j-th image; w be the learnable weight vector used to calculate the similarity score of each feature vector; V be the learnable weight matrix used to map the feature vectors to the hidden space; tanh(·) be the activation function; and n be the number of images. Then, the aggregated global feature representation is obtained by weighted summation: ； Among them, h MIL This represents the aggregated global feature representation.

4. The epilepsy prediction system based on multimodal biological images according to claim 1, characterized in that, The fusion module specifically includes: The weight processing unit is configured to input the feature vector of each modality into a gated attention network, the gated attention network including at least one linear layer and a sigmoid activation function to generate weights ranging from 0 to 1; The normalization unit is configured to normalize the generated weights so that the sum of the weights of all modes is 1. The feature fusion unit is configured to use normalized weights to perform a weighted summation of the feature vectors of each modality to generate a multimodal fusion feature vector.

5. An image data processing method, characterized in that, Including the following steps: Acquire multimodal image data, which includes: an eye image set of epileptic patients during interictal periods: including first images of multiple designated regions of both eyes; the first images include visible light images; a tongue image set of epileptic patients during interictal periods: including first images and infrared images of the tongue surface and first images and infrared images of the sublingual area; and a facial image set of epileptic patients during interictal periods: including first images and infrared images of the face. Feature extraction is performed on the eye image set, tongue image set, and face image set respectively to obtain eye feature vector, tongue feature vector, and face feature vector; The eye feature vector and the tongue feature vector are aggregated using an attention-weighted aggregation mechanism to obtain aggregated feature vectors for the eye and tongue respectively. The aggregated feature vectors of the eyes, tongue, and face are fused together, and a fusion weight for each modality is obtained through a gated attention mechanism. A joint feature vector is then obtained based on these weights. This joint feature vector serves as input data for an epilepsy prediction model. The epilepsy prediction model is trained using a class-weighted cross-entropy loss, specifically: ； in, The category weights for epilepsy; The category weights are for non-epilepsy classes; Let y be the true label of the sample, and y∈{0,1}; The predicted probability output by the model; The category weights are obtained as follows: ； in, Indicates category weight; This represents the total number of samples in the training set; This represents the number of samples in the k-th class.

6. The image data processing method according to claim 5, characterized in that, The feature extraction from the tongue image set and the facial image set includes: Input the RGB channels of the tongue image set / face image set into the pre-trained ResNet model to extract the feature vectors of the RGB channels; The infrared channels of the tongue image set / face image set are input into a randomly initialized feature extraction network to extract the feature vectors of the infrared channels; The feature vectors of the RGB channels and infrared channels of the tongue image set and the face image set are fused respectively to generate the fused tongue feature vector and face feature vector.

7. The image data processing method according to claim 5, characterized in that: The attention-weighted aggregation mechanism is specifically as follows: Attention weights are calculated for each feature vector using an attention pooling mechanism. : ； Among them, h i h is the feature vector of the i-th image; j Let be the feature vector of the j-th image; w be the learnable weight vector used to calculate the similarity score of each feature vector; V be the learnable weight matrix used to map the feature vectors to the hidden space; tanh(·) be the activation function; and n be the number of images. Then, the aggregated global feature representation is obtained by weighted summation: ； Among them, h MIL This represents the aggregated global feature representation.

8. The image data processing method according to claim 5, characterized in that, The fusion weights for each modality obtained through the gating attention mechanism include: The feature vector of each modality is input into a gated attention network, which includes at least one linear layer and a sigmoid activation function to generate weights ranging from 0 to 1. The generated weights are normalized so that the sum of the weights of all modes is 1. The feature vectors of each modality are weighted and summed using normalized weights to generate a multimodal fusion feature vector.

9. An image data processing method according to any one of claims 5-8, characterized in that: The method outputs epilepsy prediction results through a pre-trained epilepsy prediction model, the training part of which includes: The encoder takes multimodal image data as input and outputs eye feature vectors, tongue feature vectors, and facial feature vectors. The attention-weighted aggregation module takes either an eye feature vector or a tongue feature vector as input and outputs either an aggregated feature vector of the eye or an aggregated feature vector of the tongue. The fusion layer takes as input the aggregated feature vectors of the eyes, the tongue, and the face, and outputs a joint feature vector. The classifier takes a joint feature vector as input and outputs an epilepsy prediction result.

Images