Epileptic seizure identification method and device based on multi-modal data

Abstract

The invention provides an epileptic seizure identification method and device based on multi-modal data, and can be applied to the technical field of artificial intelligence and biomedical information processing. The method comprises the following steps: in response to acquired multi-modal data of a target object, performing segmentation processing on electroencephalogram signal data in the multi-modal data according to a preset granularity to obtain a plurality of electroencephalogram signal segments; for each electroencephalogram signal segment, extracting a time sequence dependence feature of the electroencephalogram signal segment; processing an identification prompt vector and the plurality of time sequence dependence features to obtain a global signal feature; determining a multi-modal feature of the target object based on the global signal feature and a text feature of the text data; and carrying out classification based on the multi-modal features, and outputting an epileptic seizure recognition result about the target object for assisting a user in judgment.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and biomedical information processing technology, and specifically to a method and device for epileptic seizure recognition based on multimodal data. Background Technology

[0002] In the field of auxiliary diagnostic technology for neurodevelopmental disorders, electroencephalogram (EEG) signals and clinical text data are two key information sources. Currently, deep learning-based multimodal classification methods mainly rely on large-scale labeled data for model training.

[0003] In the process of implementing this application, at least the following problems were found in the related technologies: In actual clinical settings, high-quality labeled data is often scarce, especially in complex comorbidity scenarios. Only a limited amount of labeled data can be used to train multimodal models, which makes existing multimodal models prone to overfitting in scenarios with few samples, resulting in poor generalization ability and difficulty in learning robust semantic correspondence between electroencephalogram (EEG) signals and text descriptions. Summary of the Invention

[0004] In view of the above problems, this application provides a method and apparatus for epileptic seizure recognition based on multimodal data.

[0005] According to a first aspect of this application, a method for epileptic seizure recognition based on multimodal data is provided, comprising: in response to acquiring multimodal data of a target object, segmenting the electroencephalogram (EEG) signal data in the multimodal data according to a preset granularity to obtain multiple EEG signal segments, wherein the multimodal data includes EEG signal data and text data corresponding to the EEG signal data; for each EEG signal segment, extracting temporal dependency features of the EEG signal segment, wherein the temporal dependency features are obtained by concatenating multiple temporal dependency sub-features, and the multiple temporal dependency sub-features are obtained by extracting features from the EEG signal segment using convolutional kernels of different sizes; processing the recognition prompt vector and the multiple temporal dependency features to obtain global signal features, wherein the global signal features are obtained by fusing multiple global signal sub-features, and the global signal sub-features are obtained by processing the recognition prompt vector and the temporal dependency features through a bidirectional cross-attention aggregation mechanism; determining the multimodal features of the target object based on the global signal features and the text features of the text data; and classifying based on the multimodal features and outputting an epileptic seizure recognition result for assisting the user in judgment.

[0006] According to embodiments of this application, the identification cue vector and multiple temporal dependency features are processed to obtain global signal features, including: using the identification cue vector as a query and the temporal dependency features as keys and values, and determining the cue-driven features among the temporal dependency features through cross-attention calculation; using the temporal dependency features as queries and the identification cue vector as keys and values, and calibrating the semantic bias introduced into the temporal dependency features through the identification cue vector through cross-attention calculation to obtain calibrated signal features; and fusing the cue-driven features and the calibrated signal features to obtain global signal features.

[0007] According to embodiments of this application, the identification prompt vector includes at least one of the following: a preset prompt vector and a learnable prompt vector, wherein the learnable prompt vector is obtained by fitting through an end-to-end model training process; and the preset prompt vector is set based on the signal characteristics of EEG signal data during an epileptic seizure.

[0008] According to embodiments of this application, the epileptic seizure recognition method further includes: performing structured processing on text data based on a preset cue vector to obtain structured text data; and using a text encoder to extract features from the structured text data to obtain text features of the text data.

[0009] According to an embodiment of this application, the structured text data includes multiple text pairs; the text encoder is determined by: randomly arranging the multiple text pairs in the structured text data to obtain disordered text data; using the text encoder to extract features from the disordered text data to obtain disordered text features; and if the vector similarity between the disordered text features and the global signal features is less than a first similarity threshold, adjusting the parameters of the text encoder until the vector similarity is greater than or equal to the first similarity threshold, thereby determining the text encoder.

[0010] According to an embodiment of this application, classification based on multimodal features outputs an epileptic seizure identification result for assisting the user in judgment, including: determining the similarity between multimodal features and multiple reference features using a classifier, wherein the classifier is trained using multiple sample data, the sample data including sample text data and labels corresponding to the sample text data, and the reference features are obtained by feature extraction from the sample text data; and determining the epileptic seizure identification result based on the label corresponding to the reference feature with the highest similarity.

[0011] According to embodiments of this application, convolutional kernels of different sizes include at least a first convolutional kernel and a second convolutional kernel, wherein the kernel length of the second convolutional kernel is greater than the kernel length of the first convolutional kernel; for each EEG signal segment, the temporal dependency features of the EEG signal segment are extracted, including: using the first convolutional kernel to extract features from the EEG signal segment to obtain a first temporal dependency sub-feature of the EEG signal segment; using the second convolutional kernel to extract features from the EEG signal segment to obtain a second temporal dependency sub-feature of the EEG signal segment; concatenating the first temporal dependency sub-feature and the second temporal dependency sub-feature in the channel dimension to obtain a temporal dependency concatenated feature; and mapping the temporal dependency concatenated feature to a feature space of a preset dimension to obtain a temporal dependency feature.

[0012] According to an embodiment of this application, the dimension of the text features is higher than the preset dimension; based on the global signal features and the text features of the text data, the multimodal features of the target object are determined, including: mapping the text features to a feature space of a preset dimension to obtain dimensionality-reduced text features; and fusing the dimensionality-reduced text features with the global signal features to obtain multimodal features.

[0013] According to embodiments of this application, the epileptic seizure recognition method further includes: adjusting the parameters of a signal encoder and / or a text encoder when the vector similarity between global signal features and text features is less than a second similarity threshold, wherein the signal encoder is used to extract temporal dependency features of each EEG signal segment, and the text encoder is used to extract features from the text data to obtain text features of the text data; and adjusting the parameters of the signal encoder and / or the text encoder when the vector similarity between the global signal features of the target object and the text features of other objects is greater than a third similarity threshold, or the vector similarity between the global signal features of other objects and the text features of the target object is greater than the third similarity threshold.

[0014] A second aspect of this application provides an epileptic seizure recognition device based on multimodal data, comprising: a signal segmentation module, configured to, in response to acquiring multimodal data of a target object, segment the electroencephalogram (EEG) signal data in the multimodal data according to a preset granularity to obtain multiple EEG signal segments, wherein the multimodal data includes EEG signal data and text data corresponding to the EEG signal data; and a feature extraction module, configured to extract temporal dependency features for each EEG signal segment, wherein the temporal dependency features are obtained by concatenating multiple temporal dependency sub-features, and the multiple temporal dependency sub-features are obtained by applying convolutional kernels of different sizes to the EEG signals. The feature extraction module is used to extract features from the segment; the feature processing module is used to process the recognition prompt vector and multiple temporally dependent features to obtain global signal features, wherein the global signal features are obtained by feature fusion of multiple global signal sub-features, and the global signal sub-features are obtained by processing the recognition prompt vector and temporally dependent features through a bidirectional cross-attention aggregation mechanism; the feature fusion module is used to determine the multimodal features of the target object based on the global signal features and the text features of the text data; and the result output module is used to classify based on the multimodal features and output the epileptic seizure recognition results of the target object to assist the user in judgment.

[0015] According to embodiments of this application, in response to acquiring multimodal data of the target object, the EEG signal data in the multimodal data is segmented according to a preset granularity to obtain multiple EEG signal segments. Features are extracted from these EEG signal segments using convolutional kernels of different sizes, capturing temporal dependency information at different time scales and enhancing the representational ability of the EEG signals. Dynamic sampling is performed from a cue pool including multiple manual cues to construct multiple sets of sample pairs of manual cues and EEG signals, enhancing the model's robustness to cue combinations and solving the problem of poor model generalization ability caused by insufficient data samples. The recognition cue vector and multiple temporal dependency features are processed to obtain global signal features that integrate the recognition cue vector and multiple temporal dependency features, enhancing the overall representational and understanding capabilities of the EEG signals and improving the robustness and discriminative power of the global signal features. Based on the global signal features and the text features of the text data, the multimodal features of the target object are determined. Classification is performed based on the multimodal features, and an epileptic seizure recognition result for the target object is output to assist the user in making a judgment. It enables multimodal recognition of epileptic seizures, improving the model's adaptability and recognition accuracy in complex clinical scenarios. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0017] Figure 1The diagram illustrates an application scenario of the epileptic seizure recognition method and apparatus based on multimodal data according to embodiments of this application.

[0018] Figure 2 A flowchart of a method for identifying epileptic seizures based on multimodal data according to an embodiment of this application is shown.

[0019] Figure 3 A data flow diagram of a seizure identification method based on multimodal data according to an embodiment of this application is shown.

[0020] Figure 4 A structural block diagram of a multimodal data-based epileptic seizure recognition device according to an embodiment of this application is shown.

[0021] Figure 5 A block diagram of an electronic device suitable for implementing a multimodal data-based method for epileptic seizure recognition, according to an embodiment of this application, is shown. Detailed Implementation

[0022] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, application, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0027] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0028] Related technologies rarely utilize prior knowledge such as diagnostic cues to guide feature learning, making it difficult for models to quickly adapt to new categories or samples, and thus hindering joint recognition based on multimodal data. Furthermore, epilepsy seizure recognition based on single-modal data suffers from data bias and difficulty in fully utilizing available information, resulting in low recognition accuracy. Clinical texts are typically unstructured descriptions; in limited sample settings, mechanisms for textual information enhancement and semantic expansion are lacking, limiting the model's information utilization efficiency.

[0029] To address the aforementioned issues, embodiments of this application provide a method for epileptic seizure recognition based on multimodal data, comprising: in response to acquiring multimodal data of a target object, segmenting the electroencephalogram (EEG) signal data in the multimodal data according to a preset granularity to obtain multiple EEG signal segments, wherein the multimodal data includes EEG signal data and text data corresponding to the EEG signal data; for each EEG signal segment, extracting temporal dependency features of the EEG signal segment, wherein the temporal dependency features are obtained by concatenating multiple temporal dependency sub-features, and the multiple temporal dependency sub-features are obtained by extracting features from the EEG signal segment using convolutional kernels of different sizes; processing the recognition prompt vector and the multiple temporal dependency features to obtain global signal features, wherein the global signal features are obtained by fusing multiple global signal sub-features, and the global signal sub-features are obtained by processing the recognition prompt vector and temporal dependency features through a bidirectional cross-attention aggregation mechanism; determining the multimodal features of the target object based on the global signal features and the text features of the text data; and classifying based on the multimodal features and outputting an epileptic seizure recognition result for the target object to assist the user in making a judgment.

[0030] This application responds to the acquisition of multimodal data of the target object by segmenting the EEG signal data in the multimodal data according to a preset granularity to obtain multiple EEG signal segments. Features are extracted from these segments using convolutional kernels of different sizes, capturing temporal dependency information at different time scales and enhancing the representational ability of the EEG signals. Dynamic sampling is performed from a cue pool including multiple manual cues to construct multiple sets of manual cue and EEG signal sample pairs, enhancing the model's robustness to cue combinations and solving the problem of poor model generalization ability caused by insufficient data samples. The recognition cue vector and multiple temporal dependency features are processed to obtain global signal features that integrate the recognition cue vector and multiple temporal dependency features, enhancing the overall representation and understanding ability of the EEG signals and improving the robustness and discriminativeness of the global signal features. Based on the global signal features and the text features of the text data, the multimodal features of the target object are determined. Classification is performed based on these multimodal features, outputting epileptic seizure recognition results for the target object to assist user judgment. This achieves multimodal recognition of epileptic seizures, improving the model's adaptability and recognition accuracy in complex clinical scenarios.

[0031] Figure 1 The diagram illustrates an application scenario of the epileptic seizure recognition method and apparatus based on multimodal data according to embodiments of this application.

[0032] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0033] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] It should be noted that the epileptic seizure recognition method based on multimodal data provided in this application embodiment can generally be executed by server 105. Correspondingly, the epileptic seizure recognition device based on multimodal data provided in this application embodiment can generally be located in server 105. The epileptic seizure recognition method based on multimodal data provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the epileptic seizure recognition device based on multimodal data provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0037] It should be understood that Figure 1The number of first terminal devices, second terminal devices, third terminal devices, networks, and servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of first terminal devices, second terminal devices, third terminal devices, networks, and servers can be included.

[0038] The following will be based on Figure 1 The described scene, through Figures 2-3 The epileptic seizure recognition method based on multimodal data according to the embodiments of this application will be described in detail.

[0039] Figure 2 A flowchart of a method for identifying epileptic seizures based on multimodal data according to an embodiment of this application is shown.

[0040] like Figure 2 As shown, the epileptic seizure recognition method based on multimodal data in this embodiment includes operations S210 to S250.

[0041] In operation S210, in response to acquiring the multimodal data of the target object, the EEG signal data in the multimodal data is segmented according to a preset granularity to obtain multiple EEG signal segments. The multimodal data includes EEG signal data and text data corresponding to the EEG signal data.

[0042] Multimodal data can include various types of data, such as EEG signal data collected by EEG devices and corresponding text data. The text data can include descriptive text for the EEG signal data obtained by analyzing the EEG signal data, and can also include text used to describe the target object corresponding to the EEG signal data, such as the target object's age, gender, etc.

[0043] After acquiring multimodal data of the target object, the collected EEG signal data is segmented according to a preset granularity to obtain multiple shorter EEG signal segments. For example, with a predetermined time length of 5 seconds, 60 minutes of collected EEG signal data is segmented to obtain several temporally adjacent 5-second EEG signal segments.

[0044] In operation S220, for each EEG signal segment, the temporal dependency features of the EEG signal segment are extracted. The temporal dependency features are obtained by concatenating multiple temporal dependency sub-features, which are obtained by extracting features from the EEG signal segment using convolution kernels of different sizes.

[0045] Smaller convolutional kernels excel at capturing rapidly changing local details, while larger kernels are better at sensing slower-changing overall trends. In the embodiments of this application, smaller convolutional kernels are used to capture rapidly changing local details in EEG signals, such as transient discharges like spikes.

[0046] Temporal-dependent features can be extracted from EEG signal segments that reflect the changing patterns of the signal over time, including but not limited to short-term fluctuations in detail and long-term background activity trends. By using convolutional kernels of different sizes to extract features from the same EEG signal segment, multiple temporal-dependent sub-features with different time dimensions are obtained, and these features are then concatenated to obtain the final temporal-dependent feature.

[0047] In operation S230, the recognition cue vector and multiple temporally dependent features are processed to obtain global signal features. The global signal features are obtained by feature fusion of multiple global signal sub-features. The global signal sub-features are obtained by processing the recognition cue vector and temporally dependent features through a bidirectional cross-attention aggregation mechanism.

[0048] The recognition cue vector can be a semantic representation that integrates expert prior knowledge and learnable parameters. The recognition cue vector and multiple temporally dependent features are input into a bidirectional cross-attention structure. Through a symmetrical interactive computation method, the recognition cue vector and temporally dependent features mutually filter and calibrate to extract complementary information, resulting in global signal sub-features. Similarly, the cue vector and EEG features mutually filter and calibrate to extract complementary information. These features are then fused to obtain global signal features.

[0049] In operation S240, based on global signal features and text features of text data, the multimodal features of the target object are determined.

[0050] Global signal features are representations extracted from EEG signals that incorporate diagnostic information, while text features are semantic representations extracted from clinical text data. Organically combining these two different modalities forms a unified joint representation that includes both EEG signal information and clinical semantic information—that is, multimodal features.

[0051] In operation S250, classification is performed based on multimodal features, and the output is the epileptic seizure recognition result of the target object to assist the user in making a judgment.

[0052] Multimodal features are input into a classification model, which calculates the probability or score of whether a target object belongs to a certain category, such as whether it has epileptic seizures and what type of seizure it is. This result is then presented to the user in a visual form to provide a reference for the user's judgment.

[0053] According to embodiments of this application, in response to acquiring multimodal data of the target object, the EEG signal data in the multimodal data is segmented according to a preset granularity to obtain multiple EEG signal segments. Features are extracted from these EEG signal segments using convolutional kernels of different sizes, capturing temporal dependency information at different time scales and enhancing the representational ability of the EEG signals. Dynamic sampling is performed from a cue pool including multiple manual cues to construct multiple sets of sample pairs of manual cues and EEG signals, enhancing the model's robustness to cue combinations and solving the problem of poor model generalization ability caused by insufficient data samples. The recognition cue vector and multiple temporal dependency features are processed to obtain global signal features that integrate the recognition cue vector and multiple temporal dependency features, enhancing the overall representational and understanding capabilities of the EEG signals and improving the robustness and discriminative power of the global signal features. Based on the global signal features and the text features of the text data, the multimodal features of the target object are determined. Classification is performed based on the multimodal features, and an epileptic seizure recognition result for the target object is output to assist the user in making a judgment. It enables multimodal recognition of epileptic seizures, improving the model's adaptability and recognition accuracy in complex clinical scenarios.

[0054] In the embodiments of this application, when performing such Figure 2 Before operation S220, the epileptic seizure recognition method based on multimodal data further includes: convolutional kernels of different sizes, including at least a first convolutional kernel and a second convolutional kernel, wherein the kernel length of the second convolutional kernel is greater than the kernel length of the first convolutional kernel; for each EEG signal segment, extracting the temporal dependency features of the EEG signal segment, including: using the first convolutional kernel to extract features from the EEG signal segment to obtain a first temporal dependency sub-feature of the EEG signal segment; using the second convolutional kernel to extract features from the EEG signal segment to obtain a second temporal dependency sub-feature of the EEG signal segment; concatenating the first temporal dependency sub-feature and the second temporal dependency sub-feature in the channel dimension to obtain a temporal dependency concatenated feature; and mapping the temporal dependency concatenated feature to a feature space of a preset dimension to obtain a temporal dependency feature.

[0055] A lightweight multi-scale EEGNet (MS-EEGNet) can be used to capture information at different time scales contained in EEG signals. The network includes at least two parallel branches, that is, it uses at least two convolutional kernels of different sizes to extract features from EEG signal segments. The convolutional kernels of different sizes include at least a first convolutional kernel and a second convolutional kernel, wherein the kernel length of the second convolutional kernel is greater than the kernel length of the first convolutional kernel.

[0056] Specifically, for each EEG signal segment, a first convolutional kernel is used to convolve the segment, extracting temporal-dependent sub-features reflecting rapid local changes, termed the first temporal-dependent sub-feature. Simultaneously, a second convolutional kernel is used to convolve the same segment, extracting temporal-dependent sub-features reflecting longer-term background activity, termed the second temporal-dependent sub-feature. The first and second temporal-dependent sub-features are concatenated along the channel dimension to obtain a concatenated temporal-dependent feature incorporating multi-scale information. This concatenated feature is then mapped through a fully connected layer to a feature space of a predetermined dimension, yielding the final temporal-dependent feature for that EEG signal segment.

[0057] According to embodiments of this application, in addition to using two convolutional kernels of different sizes, three convolutional kernels of different sizes can also be used, for example, the kernel sizes are set to 4, 8, and 16, respectively. The smaller convolutional kernel of size 4 can capture transient high-frequency components such as spikes and sharp waves in the EEG signal, the medium-sized convolutional kernel of size 8 can sense local rhythm changes, and the larger convolutional kernel of size 16 can capture the overall trend of slow waves or background activity.

[0058] According to embodiments of this application, by employing convolutional kernels of different sizes to extract multi-scale temporal features from EEG signals, it is possible to simultaneously capture the changing trends of local transient discharges and overall background activity. After concatenating and fusing the multi-scale sub-features and mapping them to a unified dimension, complementary temporal information is preserved while feature dimensionality reduction and alignment are achieved. This enhances the model's feature representation ability and robustness in low-sample scenarios, providing information-rich and dimensionally unified input for subsequent cross-modal alignment with text features, thereby improving classification accuracy and generalization performance.

[0059] In the embodiments of this application, when performing such Figure 2 Before operation S220, the epileptic seizure recognition method based on multimodal data further includes: the structured text data includes multiple text pairs; the text encoder is determined by: randomly arranging the multiple text pairs in the structured text data to obtain disordered text data; using the text encoder to extract features from the disordered text data to obtain disordered text features; and when the vector similarity between the disordered text features and the global signal features is less than a first similarity threshold, adjusting the parameters of the text encoder until the vector similarity is greater than or equal to the first similarity threshold, thereby determining the text encoder.

[0060] Text pairs can be information related to epileptic seizures of the target object, such as age: 36, gender: male. Based on the one-to-one correspondence between preset cue vectors and text pairs, the preset cue vectors are dynamically sampled to obtain corresponding text pairs. Each text pair consists of multiple key electrophysiological results, and the order in which these results are described usually does not need to be strictly defined. Therefore, the selected text pairs are further randomly arranged to obtain disordered text data.

[0061] The disordered text data is reorganized into coherent and fluent text, and then fed into a text encoder along with a preset cue vector for feature extraction to obtain disordered text features. The vector similarity between the disordered text features and the global signal features is calculated and compared with a preset first similarity threshold. If the similarity is lower than the threshold, the parameters of the text encoder are adjusted until the similarity reaches or exceeds the threshold.

[0062] According to embodiments of this application, encoding by a text encoder enhances the robustness of the model to the order of text descriptions, effectively alleviates the overfitting problem in low-sample scenarios by introducing semantic diversity, and improves the expressive power and generalization performance of the text encoder for clinical texts.

[0063] In the embodiments of this application, when performing such Figure 2 Before the operation S220 shown, the epileptic seizure recognition method based on multimodal data further includes: performing structuring processing on text data based on a preset prompt vector to obtain structured text data; and using a text encoder to extract features from the structured text data to obtain text features of the text data.

[0064] Based on the clinical knowledge contained in the preset prompt vectors, information is extracted and organized from the raw unstructured text data, transforming it into structured text data. For example, a natural language model is used to automatically extract information related to epileptic seizures from the descriptive information of EEG signal data, such as the target subject's age, gender, discharge area, discharge frequency, and background activity characteristics, and organizes them according to fixed fields or formats, transforming them into standardized text descriptions. Subsequently, a text encoder is used to extract features from this structured text data to obtain text features for subsequent fusion.

[0065] According to embodiments of this application, by performing structuring processing and feature extraction on text data, text data can be transformed into a unified and standardized representation that is easy for models to process, effectively solving the problem of information utilization difficulties caused by unstructured text.

[0066] In the embodiments of this application, when performing such Figure 2Prior to operation S230, the epileptic seizure recognition method based on multimodal data further includes: the recognition cue vector includes at least one of the following: a preset cue vector and a learnable cue vector, wherein the learnable cue vector is obtained by fitting through an end-to-end model training process; the preset cue vector is set according to the signal characteristics of the EEG signal data during the epileptic seizure.

[0067] The preset cue vectors can be predefined and encoded with explicit biomarker semantics by clinical experts based on their expertise. For example, for self-limiting centrotemporal spike epilepsy, experts would define a cue for centrotemporal spikes. In the embodiments of this application, a cue pool including multiple preset cuees can be preset. During the training phase, samples can be dynamically collected from the cue pool, and multiple sets of sample pairs of preset cuees and EEG signals can be constructed to enhance the robustness of the model to cue combinations.

[0068] Learnable cue vectors can be used as part of the model parameters for end-to-end training, enabling the automatic capture of subtle or atypical disease pattern variations from data that are difficult to describe with fixed rules.

[0069] The preset prompt vector is concatenated with the learnable prompt sequence to form the final recognition prompt vector.

[0070] According to embodiments of this application, a comprehensive recognition prompt vector is constructed based on a preset prompt vector and a learnable prompt vector. This enables the recognition prompt vector to both represent prior clinical knowledge for guidance and adaptively learn supplementary semantics from the data, thereby improving the representational ability and generalization of the recognition prompt vector.

[0071] In the embodiments of this application, when performing such Figure 2 During operation S230, the identification cue vector and multiple temporal dependency features are processed to obtain global signal features. This includes: using the identification cue vector as a query and the temporal dependency features as keys and values, and determining the cue-driven features in the temporal dependency features through cross-attention calculation; using the temporal dependency features as queries and the identification cue vector as keys and values, and calibrating the semantic bias introduced into the temporal dependency features through the identification cue vector through cross-attention calculation to obtain calibrated signal features; and fusing the cue-driven features and calibrated signal features to obtain global signal features.

[0072] The temporal-dependent features and the recognition cue vector are input into a bidirectional cross-attention structure, which contains two branches that are computed in parallel: the main branch and the auxiliary branch.

[0073] The main branch is also known as the cue-driven branch. In this branch, the identified cue vector is used as the query, and the temporal-dependent features of the EEG are used as the key and value. Through cross-attention calculation, the cue vector actively scans and filters the most relevant temporal components of the EEG features, thereby aggregating the cue-driven features.

[0074] The auxiliary branch, also known as the EEG-driven calibration branch, uses the temporal-dependent features of the EEG as the query and the identification cue vector as the key and value. Through cross-attention calculation, the EEG features actively retrieve diagnostic cues from the cue vector that best match their own neural activity, thereby self-calibrating against potential semantic biases introduced by the cue vector and obtaining calibration signal features.

[0075] A learnable fusion coefficient λ is introduced, and the contributions of cue-driven features and EEG-driven calibration signals are dynamically balanced by weighted summation. The cue-driven features and calibration signal features are then fused to obtain global signal features.

[0076] According to embodiments of this application, by performing cross-attention calculation on the recognition cue vector and temporal-dependent features, collaborative noise reduction can be achieved from two perspectives, improving the model's feature representation ability, interpretability, and ability to capture atypical patterns in low-sample scenarios. The dynamic fusion strategy not only improves the efficiency of semantic guidance but also preserves the rich details in the original EEG data.

[0077] In the embodiments of this application, the dimension of the text features is higher than the preset dimension. When performing... Figure 2 During operation S240, based on global signal features and text features of text data, the multimodal features of the target object are determined, including: mapping text features to a feature space of a preset dimension to obtain dimensionality-reduced text features; and fusing the dimensionality-reduced text features with global signal features to obtain multimodal features.

[0078] The text encoder can be a pre-trained medical language model. Its output text features typically have a higher dimensionality than the preset dimensionality mapped to by EEG features. For example, text features might be 768-dimensional, while EEG features are 256-dimensional. Therefore, a fully connected layer or linear projection layer can be used to map the text features to a feature space of the preset dimensionality, performing a dimensionality transformation to obtain dimensionality-reduced text features that match the preset dimensionality of the EEG features. Subsequently, the dimensionality-reduced text features are fused with the previously obtained global signal features, for example, through vector concatenation, element-wise addition, or weighted summation, to obtain multimodal features that integrate EEG signal information and clinical semantic information.

[0079] According to the embodiments of this application, the dimension alignment operation ensures that the features of the two modalities can effectively interact and merge in the same feature space, avoiding the problem of information dominance or feature imbalance caused by dimensional differences.

[0080] In embodiments of this application, the epileptic seizure recognition method based on multimodal data further includes: adjusting the parameters of the signal encoder and / or text encoder when the vector similarity between global signal features and text features is determined to be less than a second similarity threshold, wherein the signal encoder is used to extract the temporal dependency features of each EEG signal segment, and the text encoder is used to extract features from the text data to obtain the text features of the text data; and adjusting the parameters of the signal encoder and / or text encoder when the vector similarity between the global signal features of the target object and the text features of other objects is determined to be greater than a third similarity threshold, or the vector similarity between the global signal features of other objects and the text features of the target object is determined to be greater than a third similarity threshold.

[0081] By employing a contrastive learning loss function based on cosine similarity, the distance between the global signal features and text features of the same target object can be minimized in high-dimensional space, and the feature distributions of different target objects can be widened.

[0082] Specifically, the cosine similarity between the global signal features and text features of the same target object is calculated and compared with a preset second similarity threshold. If the similarity is less than the second similarity threshold, it indicates that the current model has failed to sufficiently bring the cross-modal features of the same object closer together. In this case, the parameters of the signal encoder and / or text encoder are adjusted to enhance the feature consistency of positive sample pairs.

[0083] Calculate the vector similarity between the global signal features of the target object and the text features of other objects, as well as the vector similarity between the text features of the target object and the global signal features of other objects. If either similarity is greater than the third similarity threshold, it indicates that the model has incorrectly brought the cross-modal features of different objects closer together, resulting in confusion of negative sample pairs. In this case, the parameters of the signal encoder and / or text encoder are adjusted to increase the feature distance of the negative sample pairs.

[0084] According to embodiments of this application, through an optimization mechanism based on positive and negative sample comparison, the semantic correspondence between EEG and text modality of the same case can be learned step by step, while enhancing the ability to distinguish features of different cases, thereby obtaining more discriminative and robust cross-modal representations in scenarios with few samples.

[0085] In the embodiments of this application, when performing such Figure 2During operation S250, classification is performed based on multimodal features, and the output is used to assist the user in judging the epileptic seizure recognition result of the target object. This includes: using a classifier to determine the similarity between the multimodal features and multiple reference features, wherein the classifier is trained using multiple sample data, the sample data includes sample text data and labels corresponding to the sample text data, and the reference features are obtained by feature extraction from the sample text data; and determining the epileptic seizure recognition result based on the label corresponding to the reference feature with the highest similarity.

[0086] To address the semantic discrepancy between the diagnostic reports used during model pre-training and the classification label text required for downstream tasks, such as "epilepsy seizure" and "epilepsy without seizure," a targeted few-shot classification training process is employed. Specifically, a cue ensemble approach is used to generate a set of semantically equivalent text descriptions for each disease category.

[0087] A classifier is used to determine the similarity between the multimodal features and multiple reference features. The classifier is pre-trained using multiple sample data sets, each containing sample text data and its corresponding label, such as "seizure" or "no seizure". Reference features are obtained by feature extraction from the sample text data; specifically, a text encoder can convert the sample text into feature vectors, with each label corresponding to one or more reference features. During the classification phase, the similarity between the multimodal features of the current target object and each reference feature is calculated, such as cosine similarity. The reference feature with the highest similarity is then selected, and the label corresponding to that reference feature is used as the final seizure identification result.

[0088] According to the embodiments of this application, feature matching is performed using a classifier, thereby determining the classification result based on the reference feature with the highest similarity. This enables the construction of a stable category representation using limited labeled samples, enhances the robustness of the model to differences in label descriptions, and further improves the accuracy and stability of classification.

[0089] Figure 3 A data flow diagram of a seizure identification method based on multimodal data according to an embodiment of this application is shown.

[0090] like Figure 3 As shown in the embodiments of this application, the EEG signal of the target object can be obtained, and the EEG signal and text data of each target object correspond one-to-one. The text data can be processed based on the preset prompt vector to obtain structured text data.

[0091] Structured text data can be augmented by randomly arranging text pairs within it. The augmented structured text data is then input into a text encoder. The text encoder extracts features from the structured text to obtain text features.

[0092] EEG signals are segmented according to a preset granularity to obtain multiple EEG signal segments, and their temporal dependency features are extracted through a multi-scale EEG coding network. Using a text encoder, a preset cue vector is concatenated with a learnable cue to obtain a recognition cue vector. This cue vector, along with the temporal dependency features, is input into a bidirectional cross-attention aggregation mechanism. Through bidirectional interaction between cue-driven and signal self-calibration, a fused global signal feature is generated.

[0093] By contrastive learning alignment, the distance between global signal features and text features of the same target object in the semantic space can be narrowed, while the feature distributions of different target objects can be widened, achieving cross-modal semantic alignment. Based on this, the fused multimodal features are input into a classifier to output classification results that characterize epileptic seizures.

[0094] Based on the above-described method for epileptic seizure recognition using multimodal data, this application also provides a device for epileptic seizure recognition using multimodal data. The following will be combined with... Figure 4 The device is described in detail.

[0095] Figure 4 A structural block diagram of a multimodal data-based epileptic seizure recognition device according to an embodiment of this application is shown.

[0096] like Figure 4 As shown, the epileptic seizure recognition device 400 based on multimodal data in this embodiment includes a signal segmentation module 410, a feature extraction module 420, a feature processing module 430, a feature fusion module 440, and a result output module 450.

[0097] The signal segmentation module 410 is used to segment the EEG signal data in the multimodal data according to a preset granularity in response to acquiring the multimodal data of the target object, thereby obtaining multiple EEG signal segments. The multimodal data includes EEG signal data and corresponding text data. In one embodiment, the signal segmentation module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0098] The feature extraction module 420 is used to extract temporal dependency features for each EEG signal segment. These temporal dependency features are obtained by concatenating multiple temporal dependency sub-features, which are extracted from the EEG signal segment using convolutional kernels of different sizes. In one embodiment, the feature extraction module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0099] The feature processing module 430 processes the recognition cue vector and multiple temporally dependent features to obtain global signal features. These global signal features are obtained by fusing multiple global signal sub-features, which are processed through a bidirectional cross-attention aggregation mechanism. In one embodiment, the feature processing module 430 can perform the operation S230 described above, which will not be repeated here.

[0100] The feature fusion module 440 is used to determine the multimodal features of the target object based on global signal features and text features of text data. In one embodiment, the feature fusion module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0101] The result output module 450 is used to perform classification based on multimodal features and output the epileptic seizure recognition results of the target object to assist the user in making a judgment. In one embodiment, the result output module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0102] According to an embodiment of this application, the feature extraction module 420 includes a convolutional submodule.

[0103] The convolutional submodule is used to extract temporal dependency features for each EEG signal segment, including: extracting features from the EEG signal segment using a first convolutional kernel to obtain a first temporal dependency sub-feature of the EEG signal segment; extracting features from the EEG signal segment using a second convolutional kernel to obtain a second temporal dependency sub-feature of the EEG signal segment; concatenating the first and second temporal dependency sub-features along the channel dimension to obtain a temporal dependency concatenated feature; and mapping the temporal dependency concatenated feature to a feature space of a preset dimension to obtain a temporal dependency feature.

[0104] According to an embodiment of this application, the feature processing module 430 includes: a vector construction submodule and a cross-attention submodule.

[0105] The vector construction submodule is used to construct recognition cue vectors, which include at least one of the following: preset cue vectors and learnable cue vectors. The learnable cue vectors are obtained through an end-to-end model training process; the preset cue vectors are set based on the signal characteristics of EEG signal data during an epileptic seizure. The cross-attention submodule is used to process the recognition cue vectors and multiple temporally dependent features to obtain global signal features. This includes: using the recognition cue vectors as queries and the temporally dependent features as keys and values, and using cross-attention calculation to determine the cue-driven features among the temporally dependent features; using the temporally dependent features as queries and the recognition cue vectors as keys and values, and using cross-attention calculation to calibrate the semantic bias introduced by the recognition cue vectors into the temporally dependent features to obtain calibrated signal features; and fusing the cue-driven features and calibrated signal features to obtain global signal features.

[0106] According to an embodiment of this application, the feature fusion module 440 includes: a text feature extraction submodule, a dimension alignment submodule, and a contrast learning submodule.

[0107] The text feature extraction submodule is used to perform structured processing on text data based on preset prompt vectors to obtain structured text data; and to extract features from the structured text data using a text encoder to obtain text features. The dimension alignment submodule is used to determine the multimodal features of the target object based on global signal features and text features of the text data, including: mapping text features to a feature space of preset dimensions to obtain dimensionality-reduced text features; and fusing the dimensionality-reduced text features with global signal features to obtain multimodal features. The contrastive learning submodule is used to adjust the parameters of the signal encoder and / or text encoder when the vector similarity between global signal features and text features is less than a second similarity threshold. Specifically, the signal encoder extracts temporal dependency features for each EEG signal segment, and the text encoder extracts features from the text data to obtain text features; and adjusts the parameters of the signal encoder and / or text encoder when the vector similarity between the global signal features of the target object and the text features of other objects is greater than a third similarity threshold, or when the vector similarity between the global signal features of other objects and the text features of the target object is greater than a third similarity threshold.

[0108] According to an embodiment of this application, the text feature extraction submodule includes a text encoder determination unit.

[0109] The text encoder determination unit is used to randomly arrange multiple text pairs in the structured text data to obtain disordered text data; to extract features from the disordered text data using the text encoder to obtain disordered text features; and to adjust the parameters of the text encoder until the vector similarity between the disordered text features and the global signal features is less than a first similarity threshold, thereby determining the text encoder.

[0110] According to an embodiment of this application, the result output module 450 includes a similarity classification submodule.

[0111] The similarity classification submodule is used for classification based on multimodal features, and outputs the epileptic seizure recognition result of the target object to assist the user in judgment. This includes: using a classifier to determine the similarity between multimodal features and multiple reference features, wherein the classifier is trained using multiple sample data, the sample data includes sample text data and labels corresponding to the sample text data, and the reference features are obtained by feature extraction from the sample text data; and determining the epileptic seizure recognition result based on the label corresponding to the reference feature with the highest similarity.

[0112] According to embodiments of this application, any multiple modules among the signal segmentation module 410, feature extraction module 420, feature processing module 430, feature fusion module 440, and result output module 450 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the signal segmentation module 410, feature extraction module 420, feature processing module 430, feature fusion module 440, and result output module 450 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the signal segmentation module 410, feature extraction module 420, feature processing module 430, feature fusion module 440, and result output module 450 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0113] Figure 5 A block diagram of an electronic device suitable for implementing a multimodal data-based method for epileptic seizure recognition, according to an embodiment of this application, is shown.

[0114] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0115] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0116] According to embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0117] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0118] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0119] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0120] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0121] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0122] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0123] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0126] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for recognizing epileptic seizures based on multimodal data, applied to electronic devices, characterized in that, The method includes: In response to acquiring multimodal data of the target object, the EEG signal data in the multimodal data is segmented according to a preset granularity to obtain multiple EEG signal segments, wherein the multimodal data includes the EEG signal data and text data corresponding to the EEG signal data; For each of the EEG signal segments, temporal dependency features of the EEG signal segments are extracted. The temporal dependency features are obtained by concatenating multiple temporal dependency sub-features. The multiple temporal dependency sub-features are obtained by extracting features from the EEG signal segments using convolutional kernels of different sizes. The recognition cue vector and multiple temporal dependent features are processed to obtain global signal features, wherein the global signal features are obtained by feature fusion of multiple global signal sub-features, and the global signal sub-features are obtained by processing the recognition cue vector and the temporal dependent features through a bidirectional cross-attention aggregation mechanism; Based on the global signal features and the text features of the text data, the multimodal features of the target object are determined; and Based on the multimodal features, classification is performed, and the results of epileptic seizure identification of the target object are output to assist the user in making a judgment.

2. The method for recognizing epileptic seizures according to claim 1, characterized in that, The process of processing the identification cue vector and multiple temporally dependent features to obtain global signal features includes: Using the identification cue vector as the query and the temporal dependency features as the key and value, the cue-driven features in the temporal dependency features are determined through cross-attention calculation; Using the temporal dependency feature as the query and the identification cue vector as the key and value, cross-attention calculation is used to calibrate the semantic bias introduced into the temporal dependency feature by the identification cue vector, resulting in a calibration signal feature; and The global signal features are obtained by fusing the prompt-driven features and the calibration signal features.

3. The method for recognizing epileptic seizures according to claim 1, characterized in that, The recognition prompt vector includes at least one of the following: a preset prompt vector and a learnable prompt vector, wherein the learnable prompt vector is obtained by fitting through an end-to-end model training process; The preset prompt vector is set based on the signal characteristics of EEG signal data during an epileptic seizure.

4. The method for recognizing epileptic seizures according to claim 3, characterized in that, The method further includes: Based on the preset prompt vector, the text data is processed to obtain structured text data; and The structured text data is processed using a text encoder to extract features, thereby obtaining the text features of the text data.

5. The method for recognizing epileptic seizures according to claim 4, characterized in that, The structured text data includes multiple text pairs; The text encoder is determined in the following way: Randomly arrange multiple text pairs in the structured text data to obtain disordered text data; The text encoder is used to extract features from the disordered text data to obtain disordered text features. as well as If the vector similarity between the disordered text features and the global signal features is less than a first similarity threshold, the parameters of the text encoder are adjusted until the vector similarity is greater than or equal to the first similarity threshold, and the text encoder is determined.

6. The method for recognizing epileptic seizures according to claim 1, characterized in that, The classification based on the multimodal features, and the output of epilepsy seizure identification results for the target object to assist user judgment, include: A classifier is used to determine the similarity between the multimodal features and multiple reference features, wherein the classifier is trained using multiple sample data, the sample data including sample text data and labels corresponding to the sample text data, and the reference features are obtained by feature extraction from the sample text data; and The epileptic seizure identification result is determined based on the label corresponding to the reference feature with the highest similarity.

7. The method for recognizing epileptic seizures according to claim 1, characterized in that, The different sizes of convolution kernels include at least a first convolution kernel and a second convolution kernel, wherein the convolution kernel length of the second convolution kernel is greater than the convolution kernel length of the first convolution kernel; The step of extracting the temporal dependency features of each of the EEG signal segments includes: The first convolutional kernel is used to extract features from the EEG signal segment to obtain the first temporal-dependent sub-feature of the EEG signal segment; The second convolutional kernel is used to extract features from the EEG signal segment to obtain the second temporal-dependent sub-features of the EEG signal segment; The first and second temporal dependency sub-features are concatenated along the channel dimension to obtain the concatenated temporal dependency feature; and The temporal dependency concatenation features are mapped onto a feature space of a preset dimension to obtain the temporal dependency features.

8. The method for recognizing epileptic seizures according to claim 7, characterized in that, The dimension of the text feature is higher than the preset dimension; The determination of the multimodal features of the target object based on the global signal features and the text features of the text data includes: The text features are mapped onto the feature space of the preset dimension to obtain dimensionality-reduced text features; and The reduced text features are fused with the global signal features to obtain the multimodal features.

9. The method for recognizing epileptic seizures according to any one of claims 1 to 8, characterized in that, The method further includes: If the vector similarity between the global signal features and the text features is less than a second similarity threshold, the parameters of the signal encoder and / or the text encoder are adjusted. The signal encoder is used to extract the temporal dependency features of each EEG signal segment, and the text encoder is used to extract features from the text data to obtain the text features of the text data. If the vector similarity between the global signal features of the target object and the text features of other objects is greater than a third similarity threshold, or the vector similarity between the global signal features of other objects and the text features of the target object is greater than a third similarity threshold, the parameters of the signal encoder and / or the text encoder are adjusted.

10. A seizure recognition device based on multimodal data, applied to electronic devices, characterized in that, The device includes: The signal segmentation module is used to segment the EEG signal data in the multimodal data according to a preset granularity in response to the acquisition of multimodal data of the target object, so as to obtain multiple EEG signal segments. The multimodal data includes the EEG signal data and text data corresponding to the EEG signal data. The feature extraction module is used to extract the temporal dependency features of each of the EEG signal segments, wherein the temporal dependency features are obtained by concatenating multiple temporal dependency sub-features, and the multiple temporal dependency sub-features are obtained by extracting features from the EEG signal segments using convolution kernels of different sizes; The feature processing module is used to process the recognition cue vector and multiple temporal dependent features to obtain global signal features. The global signal features are obtained by feature fusion of multiple global signal sub-features. The global signal sub-features are obtained by processing the recognition cue vector and the temporal dependent features through a bidirectional cross-attention aggregation mechanism. A feature fusion module is used to determine the multimodal features of the target object based on the global signal features and the text features of the text data; and The result output module is used to classify based on the multimodal features and output the epileptic seizure recognition results of the target object to assist the user in making a judgment.

Images