Electroencephalogram signal classification method, device, equipment and storage medium

Abstract

The embodiment of the application provides a brain electrical signal classification method, device and equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring brain electrical signal data, obtaining a brain electrical time-frequency characteristic signal, performing time-frequency characteristic extraction on the brain electrical time-frequency characteristic signal to obtain a brain electrical time sequence relationship signal, performing spatial characteristic extraction on the brain electrical time-frequency characteristic signal to obtain a brain electrical space relationship signal, performing attention feature fusion on the brain electrical time sequence relationship signal and the brain electrical space relationship signal to obtain brain electrical multi-modal feature information, and then performing signal classification to obtain a brain electrical signal classification result. In the embodiment, the brain electrical time-frequency characteristic signal is subjected to time-frequency characteristic extraction and spatial characteristic extraction respectively, and then multi-modal fusion is performed, and the multi-modal feature information is utilized in classification and recognition, so that deep-level interaction information between brain electrical signal features can be mined, the classification result accuracy of the brain electrical signal is improved, and the application scenario of the brain electrical signal is expanded.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to methods, apparatus, devices and storage media for classifying electroencephalogram (EEG) signals. Background Technology

[0002] Electroencephalogram (EEG) signals are electrical signals generated by neurons in the cerebral cortex. They can be divided into spontaneous EEG and evoked EEG. Information about brain activity can be extracted from EEG signals. Different brain activities are mainly reflected in changes in the frequency of EEG signals, and different activities can be distinguished by different frequency changes. Currently, brain-computer interfaces can capture these electrical signals and correctly "translate" them into the commands the human body needs to execute, making direct interaction between the human brain and external devices a reality.

[0003] In developing this invention, the applicant discovered at least the following problems in the prior art: some EEG signal classification algorithms focus only on extracting features from a single modality, or some EEG signal classification algorithms simply use vector concatenation to fuse the extracted signal features. Both of these methods have limitations and cannot deeply explore the interaction information between EEG signal features, resulting in low accuracy in the classification results of EEG signals. Therefore, how to improve the classification accuracy of EEG signals has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose a method, apparatus, device, and storage medium for classifying electroencephalogram (EEG) signals, thereby improving the accuracy of EEG signal classification.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for classifying electroencephalogram (EEG) signals, comprising:

[0006] Acquire electroencephalogram (EEG) signal data;

[0007] The EEG signal data is subjected to time-frequency feature construction to obtain EEG time-frequency feature signals;

[0008] The EEG time-frequency feature signal is input into a trained time-frequency information extraction model to extract time-frequency features and obtain the EEG time sequence relationship signal.

[0009] The EEG time-frequency feature signal is input into a trained graph embedding coding model for spatial feature extraction to obtain the EEG spatial relationship signal;

[0010] The EEG temporal relationship signal and the EEG spatial relationship signal are fused with attention features to obtain EEG multimodal feature information;

[0011] The EEG multimodal feature information is classified to obtain the EEG signal classification result.

[0012] In some embodiments, constructing time-frequency features from the EEG signal data to obtain EEG time-frequency feature signals includes:

[0013] The EEG signal data is divided into one or more time segments by using a preset sliding window to perform sliding segmentation according to the temporal relationship;

[0014] Perform a discrete Fourier transform on each of the time segment data to obtain more than one power spectral density information;

[0015] By concatenating one or more power spectral density information, time-frequency feature data of the time segment data is formed;

[0016] Each of the time-frequency feature data is spliced ​​together according to the time sequence relationship to obtain the EEG time-frequency feature signal.

[0017] In some embodiments, the time-frequency information extraction model includes: one or more one-dimensional convolutional layers and at least one time-series network. The step of inputting the EEG time-frequency feature signal into the trained time-frequency information extraction model for time-frequency feature extraction to obtain the EEG time-series relationship signal includes:

[0018] By sequentially performing convolution operations on the EEG time-frequency feature signal using one or more of the aforementioned one-dimensional convolutional layers, the deep semantic information of the EEG time-frequency feature signal can be obtained.

[0019] The deep semantic information is extracted using at least one time-series network to obtain the EEG temporal relationship signal.

[0020] In some embodiments, the graph embedding coding model includes a graph convolution module, and the step of inputting the EEG time-frequency feature signal into the trained graph embedding coding model for spatial feature extraction to obtain the EEG spatial relationship signal includes:

[0021] The EEG time-frequency feature signal is transformed into a graph to obtain the graph representation information of the EEG time-frequency feature signal. The channels of the EEG time-frequency feature signal correspond to the nodes of the graph representation information, and the relationships between the channels of the EEG time-frequency feature signal correspond to the edges between the nodes of the graph representation information.

[0022] The graph representation information is input into the graph convolution module to perform graph convolution operation, thereby obtaining the EEG spatial relationship signal.

[0023] In some embodiments, the step of fusing the EEG temporal relationship signal and the EEG spatial relationship signal with attention features to obtain EEG multimodal feature information includes:

[0024] The time-series EEG relationship signal is multiplied by the first weight matrix and the second weight matrix respectively, and then added together to obtain the time-frequency modal feature information;

[0025] The spatial relationship signal of the EEG is multiplied by the third weight matrix to obtain the spatial modality feature information;

[0026] The EEG multimodal feature information is obtained based on the time-frequency modal feature information and the spatial modal feature information.

[0027] In some embodiments, after acquiring the EEG signal data, the method further includes:

[0028] The EEG signal data is preprocessed to obtain the preprocessed EEG signal data. The preprocessing includes at least one of the following: cropping operation, channel selection operation, downsampling operation, filtering operation, or baseline removal operation.

[0029] In some embodiments, classifying the EEG multimodal feature information to obtain EEG signal classification results includes:

[0030] The EEG multimodal feature information is input into at least one fully connected layer to obtain classification feature information;

[0031] The classification feature information is input into the classification layer to obtain the classification probability;

[0032] The classification result of the EEG signal is obtained based on the classification probability.

[0033] To achieve the above objectives, a second aspect of this application provides an electroencephalogram (EEG) signal classification device, comprising:

[0034] Acquisition unit, used to acquire EEG signal data;

[0035] A time-frequency feature construction unit is used to construct time-frequency features on the electroencephalogram (EEG) signal data to obtain an EEG time-frequency feature signal.

[0036] The time-frequency information extraction unit is used to input the EEG time-frequency feature signal into the trained time-frequency information extraction model to extract the time-frequency features and obtain the EEG time sequence relationship signal.

[0037] The spatial feature extraction unit is used to input the EEG time-frequency feature signal into a trained graph embedding coding model to extract spatial features and obtain EEG spatial relationship signals.

[0038] A multimodal feature fusion unit is used to fuse the EEG temporal relationship signal and the EEG spatial relationship signal by attention features to obtain EEG multimodal feature information;

[0039] The signal classification unit is used to classify the EEG multimodal feature information to obtain the EEG signal classification result.

[0040] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0041] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0042] The EEG signal classification method, apparatus, device, and storage medium proposed in this application acquire EEG signal data, construct time-frequency features from the EEG signal data to obtain EEG time-frequency feature signals, then input the EEG time-frequency feature signals into a trained time-frequency information extraction model for time-frequency feature extraction to obtain EEG time-series relationship signals, and input the EEG time-frequency feature signals into a trained graph embedding coding model for spatial feature extraction to obtain EEG spatial relationship signals. Attention feature fusion is then performed on the EEG time-series relationship signals and EEG spatial relationship signals to obtain EEG multimodal feature information. Finally, signal classification is performed based on the EEG multimodal feature information to obtain the EEG signal classification result. This embodiment performs time-frequency feature extraction and spatial feature extraction separately on the EEG time-frequency feature signals after time-frequency feature construction, and then performs multimodal fusion of the EEG time-series relationship signals and EEG spatial relationship signals. Utilizing multimodal feature information during classification and recognition can uncover deep-level interaction information between EEG signal features, improve the accuracy of EEG signal classification results, and expand the application scenarios of EEG signals. Attached Figure Description

[0043] Figure 1 This is a flowchart of the electroencephalogram (EEG) signal classification method provided in an embodiment of the present invention.

[0044] Figure 2 yes Figure 1 The flowchart for step S120.

[0045] Figure 3 This is a schematic diagram of the time-frequency information extraction model structure of the EEG signal classification method provided in another embodiment of the present invention.

[0046] Figure 4 yes Figure 1 The flowchart of step S130.

[0047] Figure 5 yes Figure 1The flowchart for step S140.

[0048] Figure 6 This is a schematic diagram of the graphical representation information of the electroencephalogram (EEG) signal classification method provided in another embodiment of the present invention.

[0049] Figure 7 yes Figure 1 The flowchart for step S150.

[0050] Figure 8 yes Figure 1 The flowchart for step S160 in the process.

[0051] Figure 9 This is a flowchart of an electroencephalogram (EEG) signal classification method provided in another embodiment of the present invention.

[0052] Figure 10 This is a structural block diagram of an electroencephalogram (EEG) signal classification device provided in another embodiment of the present invention.

[0053] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0057] First, let's clarify some of the terms used in this invention:

[0058] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0059] Brain-Computer Interface (BCI) technology is a human-computer interaction technology that enables direct communication between the human brain and computers or other electronic devices. BCI technology is widely used in medical rehabilitation of patients with mobility impairments who have lost control of their limbs, research on intelligent robots, military exoskeletons, video games, and virtual reality (VR) technology. BCI can restore normal brain function by inducing activity-dependent brain plasticity, enabling the diagnosis of epilepsy, direct brain control of robots, and prosthetic limbs. It provides possibilities for disabled people to regain normal functional abilities. The core of BCI is the recognition of electroencephalography (EEG) signals. In EEG recognition, a specific type is typically classified for diagnosis or control, while combinations of multiple EEG types can achieve more diverse control methods.

[0060] Time domain: The independent variable is time, i.e., the horizontal axis represents time, and the vertical axis represents the change of the signal. Its dynamic signal x(t) is a function describing the value of the signal at different times.

[0061] Frequency domain: The independent variable is frequency, that is, the horizontal axis is frequency and the vertical axis is the amplitude of the signal at that frequency, which is usually referred to as the spectrum diagram.

[0062] Attention units: Research on attention units first appeared in psychology, where it was observed that when a person sees a scene, although they are faced with the entire scene, their attention is focused on only a specific point. In 2014, Bahdanau et al. first introduced attention units into machine translation, combining them with neural networks to improve the accuracy of machine translation. Subsequent studies have attempted to combine attention units with neural networks for different tasks, all achieving significant advantages over traditional methods to varying degrees.

[0063] Deep learning involves learning the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies. Deep learning has also yielded significant advancements in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thought, solving many complex pattern recognition problems and significantly advancing artificial intelligence-related technologies.

[0064] Neural Networks: Neural networks are a type of machine learning modeled after the human brain. They are capable of deep learning. The basic component of an artificial neural network is a perceptron, which performs simple signal processing and is then connected to a large mesh network. Neural networks with deep learning capabilities cannot be directly programmed to complete tasks; they need to learn information. There are generally three learning methods: 1) Supervised learning: The computer modifies the model based on a labeled dataset until it can process the dataset to obtain the desired results; 2) Unsupervised learning: Without a labeled dataset to learn from, the neural network analyzes the dataset, then uses a cost function to tell the neural network how far it is from the target, and then adjusts the network to improve the algorithm's accuracy; 3) Reinforcement learning: In this algorithm, positive results from the neural network are reinforced, while negative results are penalized, forcing the neural network to learn over time.

[0065] Feature extraction involves reducing the dimensionality of the original input data or recombining the original features for later use. Simply put, it has two functions: reducing data dimensionality and organizing existing data features.

[0066] Electroencephalogram (EEG) signals are electrical signals generated by neurons in the cerebral cortex. They can be divided into spontaneous EEG and evoked EEG. Information about brain activity can be extracted from EEG signals. Different brain activities are mainly reflected in changes in the frequency of EEG signals, and different activities can be distinguished by different frequency changes. Currently, brain-computer interfaces can capture these electrical signals and correctly "translate" them into the commands the human body needs to execute, making direct interaction between the human brain and external devices a reality.

[0067] In developing this invention, the applicant discovered at least the following problems in the prior art: some EEG signal classification algorithms focus only on extracting features from a single modality, or some EEG signal classification algorithms simply use vector concatenation to fuse the extracted signal features. Both of these methods have limitations and cannot deeply explore the interaction information between EEG signal features, resulting in low accuracy in the classification results of EEG signals. Therefore, how to improve the classification accuracy of EEG signals has become an urgent technical problem to be solved.

[0068] Based on this, embodiments of the present invention provide a method, apparatus, device, and storage medium for classifying electroencephalogram (EEG) signals. The EEG signal classification method acquires EEG signal data, constructs time-frequency features from the EEG signal data to obtain EEG time-frequency feature signals, then inputs these signals into a trained time-frequency information extraction model for time-frequency feature extraction to obtain EEG temporal relationship signals, and simultaneously inputs them into a trained graph embedding coding model for spatial feature extraction to obtain EEG spatial relationship signals. Attention feature fusion is then performed on the EEG temporal relationship signals and EEG spatial relationship signals to obtain multimodal EEG feature information. Finally, signal classification is performed based on the multimodal EEG feature information to obtain the EEG signal classification result. This embodiment performs time-frequency feature extraction and spatial feature extraction on the constructed EEG time-frequency feature signals, and then performs multimodal fusion of the EEG temporal relationship signals and EEG spatial relationship signals. Utilizing multimodal feature information during classification and recognition allows for the discovery of deep-level interaction information between EEG signal features, improving the accuracy of EEG signal classification results and expanding the application scenarios of EEG signals.

[0069] The present invention provides a method, apparatus, device and storage medium for classifying electroencephalogram (EEG) signals, which are specifically described through the following embodiments. First, the EEG signal classification method in the embodiments of the present invention is described.

[0070] This invention can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0071] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0072] The EEG signal classification method provided in this invention relates to the field of artificial intelligence technology, and particularly to the field of data mining technology. This EEG signal classification method can be applied to a terminal, a server, or a computer program running on either a terminal or a server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client that supports EEG signal classification; it can also be a small program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The EEG signal classification method can be executed by the terminal or the server, or by the terminal and the server working together.

[0073] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device uses the EEG signal classification method of this embodiment to provide related services, enhancing the driving experience. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server can have the EEG signal classification system's server-side software installed, allowing interaction with the terminal. For example, the server can have corresponding software installed, which could be an application implementing the EEG signal classification method, but is not limited to the above forms. The terminal and the server can be connected via Bluetooth, USB (Universal Serial Bus), or network communication methods, and this embodiment does not impose any restrictions.

[0074] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] To facilitate understanding of the embodiments of this application, the concept of EEG signal classification will be briefly introduced below with examples of specific application scenarios.

[0076] Because information transmission between neurons in the brain is conducted via electrical signals, the potential changes emitted by a single neuron are extremely weak and insufficient to be detected by external devices. However, when a group of neurons or neural tissue in the brain emit the same potential signal, the corresponding potential changes will superimpose and become detectable by a device. After processing by an external amplifier, this signal can be recorded by electrodes on an electrode cap, forming an electroencephalogram (EEG) signal. Therefore, the user's EEG signal is first collected, then features are extracted from the EEG signal to obtain EEG feature data, which is then classified according to classification requirements. Some common classification requirements include: motor imagery, emotion classification, or mental state classification.

[0077] In one application scenario, it's necessary to classify EEG signals to determine the user's mental state. Mental states include: wakefulness, sleep, anxiety / tension, fatigue / drowsiness, calm / relaxed, emotional fluctuation, and concentration. If the user is a long-distance driver, the classification results of the EEG signals can be used to determine whether the user's current mental state is suitable for driving, allowing for appropriate safety procedures.

[0078] In one application scenario, it's necessary to classify EEG signals to determine the user's emotional state, including emotions such as happiness, sadness, love, hatred, romance, calmness, loathing, and anger. If the user's current entertainment need is music or movies, music or movie recommendations can be made based on the classification results of the EEG signals.

[0079] In one application scenario: it is necessary to classify EEG signals to determine the user's movement tendencies, including head shaking, nodding, forward movement, left turn, right turn, swaying, and other limb movements, and corresponding device operation commands. If the user is physically disabled, requires a wheelchair, and cannot operate it independently (e.g., due to missing fingers), the classification results of the EEG signals can determine the user's wheelchair operation commands. If the user is a VR device user, the classification results of the EEG signals can determine the user's virtual movements, enabling device operation or control of related game characters to perform corresponding actions, thereby improving the user's experience with virtual reality devices.

[0080] Input: Collected user EEG information;

[0081] Output: The analysis yields the classification results for the corresponding requirements.

[0082] The following describes the electroencephalogram (EEG) signal classification method in the embodiments of the present invention.

[0083] Figure 1 This is an optional flowchart of the EEG signal classification method provided in the embodiments of the present invention. Figure 1The method may include, but is not limited to, steps S110 to S160. It is also understood that this embodiment... Figure 1 The order of steps S110 to S160 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0084] Step S110: Acquire EEG signal data.

[0085] In one embodiment, the acquisition of electroencephalogram (EEG) signal data includes invasive and non-invasive methods. This embodiment primarily focuses on non-invasive acquisition, such as using an electrode cap to acquire EEG signal data. An electrode cap is a cap-shaped device that can be worn on the user's head. Microneedle electrodes are positioned at different locations on this device, enabling the acquisition of EEG signal data from corresponding locations. In one embodiment, each electrode serves as a acquisition channel; that is, the number of microneedle electrodes used determines the number of channels of EEG signal data obtained.

[0086] In one embodiment, the number of electrodes can be 16, 32, 64, or 512. There is no limitation on the number of electrodes; it is understood that a larger number of electrodes improves the spatial resolution of the EEG signal data. The appropriate number of electrodes can be selected based on actual needs. For example, using 16 electrodes, placing them in 16 different locations on the brain allows for the acquisition of EEG signal data from 16 channels corresponding to these 16 locations.

[0087] As mentioned above, EEG signal data can be collected through electrodes, but the acquired EEG signal data is generally very weak, which is not conducive to subsequent research. Furthermore, the EEG signal data collected by electrodes also includes electromyographic noise and environmental noise. Therefore, the acquired EEG signal data needs to undergo a series of preprocessing operations to reduce interference information or noise caused by the experimental environment, acquisition equipment, and the physiological activities of the subjects themselves during the signal collection process, thereby improving the accuracy of subsequent data analysis results.

[0088] In one embodiment, the EEG signal data is subjected to data preprocessing operations to obtain preprocessed EEG signal data. The data preprocessing operations include at least one of the following: cropping operation, channel selection operation, downsampling operation, filtering operation, or baseline removal operation.

[0089] In one embodiment, theoretically, the EEG signal data can be infinitely long. For data analysis purposes, the EEG signal data is cropped to extract a portion for subsequent data analysis.

[0090] In one embodiment, the brain's longitudinal fissure divides it into left and right hemispheres. These hemispheres, based on actual grooves and hypothetical abstract boundaries, can be primarily divided into four regions: the frontal lobe, parietal lobe, occipital lobe, and temporal lobe. In the cranial bone structure, the frontal lobe corresponds to the frontal bone, the parietal lobe to the parietal bone, the occipital lobe to the occipital bone, and the temporal lobe to the temporal bone, each controlling different brain functions. Therefore, according to the classification requirements of the EEG signals, channel selection is needed to select the signal channels corresponding to the brain regions. This avoids the negative impact of collecting too many EEG signal data from channels irrelevant to the classification requirements on the classification results, while also reducing the data complexity of subsequent data processing and improving classification efficiency.

[0091] In one embodiment, downsampling the EEG signal data can reduce the amount of EEG signal data, reduce the data complexity of subsequent data processing, and improve classification efficiency.

[0092] In one embodiment, the EEG signal data is filtered to remove irrelevant noise signals, such as various eye-point artifacts, EMG artifacts, ECG artifacts, and other signals. In one embodiment, the filtering operation includes bandpass filtering or wavelet transform.

[0093] In one embodiment, because the EEG signal data contains low-frequency noise, a baseline removal operation is performed on the EEG signal data to eliminate the signal baseline. In one embodiment, wavelet transform is used to remove the baseline; after wavelet packet decomposition, low-frequency data is extracted, retaining the specific frequency band information of interest.

[0094] The preprocessed EEG signal data obtained from the above steps are then subjected to subsequent data analysis.

[0095] Step S120: Construct time-frequency features from the EEG signal data to obtain the EEG time-frequency feature signal.

[0096] In one embodiment, in order to enhance the video features in the EEG signal data and improve the accuracy of subsequent time-frequency information extraction, time-frequency features are constructed on the EEG signal data to obtain EEG time-frequency feature signals.

[0097] In one embodiment, reference is made to Figure 2 This is a flowchart illustrating a specific implementation of step S120 in one embodiment. In this embodiment, step S120, which constructs time-frequency features from EEG signal data to obtain EEG time-frequency feature signals, includes steps S121 to S124:

[0098] Step S121: Use a preset sliding window to perform sliding segmentation according to the temporal relationship, and divide the EEG signal data into more than one time segment data.

[0099] In one embodiment, in order to protect feature information without increasing the risk of overfitting in the subsequent learning process, the EEG signal data is processed by sliding window, that is, the EEG signal data is divided into one or more time segments by using a preset sliding window according to the temporal relationship. One time segment is one frame. This operation is also called frame segmentation.

[0100] In one embodiment, the sliding window size is set according to actual needs to obtain a preset sliding window. Starting from the beginning position of the EEG signal data, the data is slid-segmented backward according to the temporal relationship, resulting in multiple time segment data. Each time segment data contains data from multiple channels. By segmenting the EEG signal data into time frame data through sliding segmentation, the time period is shortened, thereby reducing the amount of information input to subsequent networks. This allows subsequent classification algorithms that process temporal inputs to learn the features contained in the EEG signal data more accurately, thus improving classification accuracy. In addition, during model training, the number of samples can be effectively increased while preserving the original information of the EEG signal data, improving the model training accuracy.

[0101] Step S122: Perform a discrete Fourier transform on each time segment of data to obtain more than one power spectral density information.

[0102] In one embodiment, the time segment data of each time frame obtained above is discretized to obtain multiple power spectral density information. In one embodiment, the discrete Fourier transform is used to achieve spectral discretization. After the discrete Fourier transform (DFT), both the time domain and the frequency domain are discretized, enabling subsequent calculations on the signal in the time and frequency domains.

[0103] In one embodiment, because the rhythmic firing of brain cell groups results in rhythmic characteristics in the electroencephalogram (EEG) signal data, the EEG signal data is classified according to frequency to represent various signal components. In this embodiment, the EEG signal data is divided into five frequency bands: the first band, the second band, the third band, the fourth band, and the fifth band. It is understood that the number of frequency bands can be adjusted according to actual needs and is not specifically limited here.

[0104] In one embodiment, the first frequency band is the delta band, with a frequency range of 0.5Hz-4Hz. The second frequency band is the theta band, with a frequency range of 4Hz-8Hz. The third frequency band is the alpha band, with a frequency range of 8Hz-12Hz. The fourth frequency band is the beta band, with a frequency range of 12Hz-30Hz. The fifth frequency band is the gamma band, with a frequency range of 30Hz-47Hz. It is understood that the frequency ranges of different bands can be adjusted according to actual needs, and no specific limitations are made here.

[0105] Among them, the first frequency band of EEG signal data is a high-amplitude, low-frequency, slow wave that is rarely recorded during EEG collection in normal adults. It mostly occurs when brain neurons are relaxed, such as during deep sleep, deep anesthesia, hypoxia, or when there is organic lesion in the brain. Therefore, the first frequency band only occurs in the cerebral cortex and does not participate in the information processing process.

[0106] The second frequency of EEG signal data will appear when the human body is in a calm and relaxed state and begins to feel sleepy. The appearance of the second frequency of EEG signal data reflects the process of the human body transitioning from a waking state to a sleep state. The amplitude is low and the frequency is slow. This wave can generally be recorded in the hippocampus of the brain. Studies have shown that this wave is related to the brain's thinking, memory and learning processes.

[0107] The third frequency band of EEG signal data is a low-amplitude synchronous wave, which is the most prominent in the acquired EEG signals. It can be detected in the parietal and occipital regions of the skull, and its waveform is generally similar to a sine wave. When a normal adult is in a quiet, closed-eye state, this wave is usually the dominant waveform in the acquired EEG, and at this time, the third frequency band of EEG signal data appears most frequently and has the largest amplitude.

[0108] The fourth frequency band of EEG signals consists of low-amplitude, high-frequency fast waves within the brain's rhythmic waves. It is primarily distributed in the temporal lobe and anterior hemisphere of the brain, but can be detected throughout the entire brain. It commonly occurs when the central nervous system is engaged in intense mental activity or when the body is under stress. Therefore, fourth frequency band EEG signals reflect the brain's cognitive activity. When a person's attention is focused, the fourth frequency band of EEG signals increases, indicating that the brain's central nervous system is in an excited state.

[0109] The fifth frequency band of EEG signals has a higher frequency and is associated with anxiety.

[0110] In one embodiment, based on the frequency range of the five frequency bands, step S122 performs a discrete Fourier transform on the data of each time segment and uses the short-time Fourier algorithm to extract the power spectral density information of the five frequency bands.

[0111] Step S123: Concatenate one or more power spectral density information to form time-frequency feature data of time segment data.

[0112] In one embodiment, the power spectral density information of each frequency band obtained above is spliced ​​together to obtain the time-frequency feature data corresponding to the time segment data of the time frame.

[0113] Understandably, since EEG signal data contains data from multiple channels, performing a Discrete Fourier Transform (DFT) on the time segment data of each time frame refers to performing a DFT on the data of each channel of the time segment data of each time frame to obtain the power spectral density information corresponding to each channel. The time-frequency feature data obtained after splicing using steps S123 contains power spectral density information corresponding to multiple channels.

[0114] Step S124: Concatenate each time-frequency feature data according to the time sequence relationship to obtain the EEG time-frequency feature signal.

[0115] In one embodiment, since the time-frequency feature data is the data of each time frame obtained according to the time sequence relationship, each time-frequency feature data is spliced ​​together according to the time sequence relationship to obtain the EEG time-frequency feature signal.

[0116] As can be seen from the above, each time frame of the obtained EEG time-frequency feature signal contains power spectral density information of multiple channels and multiple frequency bands. The EEG time-frequency feature signal is still a time series data, which retains the temporal relationship. The purpose of constructing the above time-frequency features is to improve the computational efficiency and accuracy of subsequent classification processing.

[0117] Step S130: Input the EEG time-frequency feature signal into the trained time-frequency information extraction model to extract time-frequency features and obtain the EEG time sequence relationship signal.

[0118] In one embodiment, a trained time-frequency information extraction model is used to extract time-frequency features. The time-frequency information extraction model can be trained using training samples to obtain better model parameters suitable for the scenario. This embodiment does not specifically limit the training process.

[0119] In one embodiment, reference is made to Figure 3 This is a schematic diagram of the time-frequency information extraction model structure. The time-frequency information extraction model 100 includes: three sequentially connected one-dimensional convolutional layers 110 and two sequentially connected time series networks 120. Here, the time series network 120 can be a Long Short-Term Memory (LSTM) artificial neural network or a Bi-directional Long Short-Term Memory (Bi-LSTM) neural network.

[0120] In one embodiment, reference is made to Figure 4 This is a flowchart illustrating a specific implementation of step S130 in one embodiment. In this embodiment, step S130, which involves inputting the EEG time-frequency feature signal into a trained time-frequency information extraction model for time-frequency feature extraction to obtain the EEG time-series relationship signal, includes steps S131 to S132:

[0121] Step S131: Convolve the EEG time-frequency feature signal sequentially using one or more one-dimensional convolutional layers to obtain the deep semantic information of the EEG time-frequency feature signal.

[0122] In one embodiment, combined with Figure 3 The EEG time-frequency feature signal first passes through three one-dimensional convolutional layers to extract deep semantic information layer by layer.

[0123] Step S132: Use at least one time series network to extract contextual information from deep semantic information to obtain EEG temporal relationship signals.

[0124] In one embodiment, combined with Figure 3 The deep semantic information is sequentially input into two time-series networks, and contextual information is extracted from the deep semantic information obtained after three one-dimensional convolutional layers. In one embodiment, the first time-series network is a Bi-LSTM bidirectional long short-term memory neural network, and the second time-series network is an LSTM long short-term memory artificial neural network.

[0125] In one embodiment, the time-series network controls the cell state C of the neural network. t-1 The system determines whether to delete or add feature information, and it consists of three gates: a forget gate, an input gate, and an output gate. The forget gate contains a sigmoid unit, which processes the output h from the previous time step. t-1 and the input x at the current time t Calculate a vector between 0 and 1, and combine it with the cell state C transmitted from the previous time step. t-1 Perform a dot product, and determine which information about the cell state to discard based on the result. The input gate takes the output h from the previous time step. t-1 and the input x at the current time t The tanh layer is used to calculate the information of candidate states to be added to the cell state. Then, a forgetting gate-like method is used to forget some of the information, and the remaining information is added to the cell state C. t-1 A new cell state C is obtained in the process. t The output gate passes the output from the previous time step and the input from the current time step through a sigmoid layer to obtain the decision condition, and then sets the cell state C. t The tanh layer produces a vector, which is then multiplied by the decision condition obtained from the output gate to obtain the final output h. t Based on the temporal relationship, the final result is an EEG temporal relationship signal containing contextual information.

[0126] As described above, by extracting time-frequency features in step S130, the EEG time-frequency feature signal can be obtained, which contains the time-frequency feature information of the EEG signal data. In order to uncover the deeper interaction information between EEG signal features, it is necessary to combine multimodal feature information during classification and recognition. Therefore, the following steps are used to extract the spatial features of the EEG time-frequency feature signal.

[0127] Step S140: Input the EEG time-frequency feature signal into the trained graph embedding coding model to extract spatial features and obtain the EEG spatial relationship signal.

[0128] In one embodiment, the graph embedding coding model includes a graph convolution module. Spatial feature extraction is performed using the trained graph embedding coding model. The model can be trained using training samples to obtain optimal model parameters suitable for the scenario. This embodiment does not specifically limit the training process.

[0129] In one embodiment, reference is made to Figure 5 This is a flowchart illustrating a specific implementation of step S140 in one embodiment. In this embodiment, step S140, which involves inputting the EEG time-frequency feature signal into a trained graph embedding coding model for spatial feature extraction to obtain the EEG spatial relationship signal, includes steps S141 to S142:

[0130] Step S141: Perform graph transformation on the EEG time-frequency feature signal to obtain the graph representation information of the EEG time-frequency feature signal.

[0131] In one embodiment, the EEG time-frequency feature signal comprises two parts. The first part is the data itself from multiple channels, each with a consistent data length, the length of which is set according to actual needs. The second part is the correlation coefficients between different channels. If the number of channels is N, the correlation coefficients between channels are an N*N matrix, where each element represents the correlation between two channels. In this embodiment, the graph embedding coding model first represents the EEG time-frequency feature signal graphically to obtain graph representation information. The graph representation process uses specific mapping relationships to show the hidden spatial relationships between the channels of the EEG time-frequency feature signal in the form of a graph.

[0132] In one embodiment, the channels of the EEG time-frequency feature signal correspond to nodes in the graph representation information, and the relationships between the EEG time-frequency feature signal channels correspond to the edges between the nodes in the graph representation information. In one embodiment, the graph representation information is typically stored in the form of an adjacency matrix, where nodes represent channels, carry the signal data of each channel itself, and the edges between nodes represent the correlation between channels; for example, this correlation is the Pearson correlation coefficient between different channels. The K-nearest neighbor algorithm can also be used to obtain the correlation between channels, and no specific limitation is made here.

[0133] In one embodiment, reference is made to Figure 6 This is a schematic diagram of graph representation information. In the diagram, each channel of the EEG time-frequency characteristic signal is mapped as a node of the graph, and the correlation coefficients of different channels are mapped as edges. Together, they form the graph data. It can be seen that the edge between two nodes represents the correlation between the channels corresponding to those two nodes. It can be understood that the actual mapped graph representation information is a three-dimensional structure diagram. Theoretically, there is a correlation between each node in the graph, only the value of the correlation value varies. The diagram does not show all the edges between nodes, but this does not mean that only the edges shown in the diagram are represented.

[0134] Step S142: Input the graph representation information into the graph convolution module to perform graph convolution operation to obtain the EEG spatial relationship signal.

[0135] In one embodiment, the graph representation information (adjacency matrix) obtained through the graph representation process is input into the graph convolution module for graph convolution operation. Specifically, the graph convolution operation utilizes graph Fourier transform to perform spectral filtering, obtaining several feature vectors that form a set of orthogonal bases in the space. Then, graph convolution is performed on the obtained feature vectors to obtain the EEG spatial relationship signal. In one embodiment, to shorten the training time of the graph convolution module and improve the accuracy of spatial feature extraction, a Chebyshev polynomial is used as the graph convolution kernel. The graph convolution process is represented as follows:

[0136]

[0137]

[0138]

[0139] T k+1 (X)=2XT k (X)-T k-1 (X)

[0140] Where T0(X) = 1, T1(X) = X, X represents the feature vector of the input graph convolution module, k represents the highest order of the Chebyshev polynomial, θ represents the vector of Chebyshev coefficients, Λ represents the diagonal matrix of eigenvalues, and λ max K represents the largest eigenvalue of the regularized matrix, I represents the identity matrix, and K represents the largest eigenvalue of the regularized matrix. e (Λ) represents a convolution kernel based on Chebyshev polynomials.

[0141] As described above, the graph embedding coding model can be used to extract spatial features from the EEG time-frequency feature signals to obtain the EEG spatial relationship signals. Thus, combining the time-frequency characteristic information (EEG time-series relationship signals) and spatial characteristic information (EEG spatial relationship signals) corresponding to the EEG signal data obtained in steps S130 and S140, the subsequent steps will classify and recognize the obtained multimodal feature information.

[0142] Step S150: The EEG temporal relationship signal and the EEG spatial relationship signal are fused by attention feature to obtain EEG multimodal feature information.

[0143] In one embodiment, attention feature fusion is performed using a trained cross-modal attention feature fusion model. The cross-modal attention feature fusion model can be trained using training samples to obtain better model parameters suitable for the scenario. This embodiment does not specifically limit the training process.

[0144] In one embodiment, reference is made to Figure 7 This is a flowchart illustrating a specific implementation of step S150 in one embodiment. In this embodiment, step S150, which fuses the EEG temporal relationship signal and the EEG spatial relationship signal to obtain EEG multimodal feature information, includes steps S151 to S153:

[0145] Step S151: Multiply the EEG temporal relationship signal with the first weight matrix and the second weight matrix respectively, and then add them together to obtain the time-frequency modal feature information.

[0146] Step S152: Multiply the EEG spatial relationship signal with the third weight matrix to obtain spatial modality feature information.

[0147] Step S153: Obtain EEG multimodal feature information based on time-frequency modal feature information and spatial modal feature information.

[0148] In one embodiment, the cross-modal attention feature fusion model uses the principle of similarity of self-attention mechanisms to fuse attention features. First, three attention weight matrices are trained to obtain the first weight matrix, the second weight matrix and the third weight matrix mentioned above. Then, the attention weight matrices are used to weight the EEG temporal relationship signal and the EEG spatial relationship signal respectively to obtain EEG multimodal feature information.

[0149] In one embodiment, the attention weight matrix includes: query information Q, key K, and value V, wherein query information Q is the third weight matrix, key K is the first weight matrix, and value V is the second weight matrix. In this embodiment, the query information Q and the set of key K and value V are used to weight the EEG temporal relationship signal and the EEG spatial relationship signal respectively.

[0150] In one embodiment, the time-series EEG signal is multiplied by a first weight matrix K and a second weight matrix V, and then summed to obtain time-frequency modal feature information. Then, the spatial EEG signal is multiplied by a third weight matrix Q to obtain spatial modal feature information. Finally, the time-frequency and spatial modal feature information are superimposed to obtain multimodal EEG feature information.

[0151] As can be seen from the above, in the embodiments of this application, one modality (e.g., time-frequency) can receive information from another modality (e.g., space), which helps to combine the feature information of different modalities. By combining multiple modal features such as time-frequency and space, the long-term contextual correlation between different modal information in EEG signal data can be obtained, and deep fusion can be performed to extract deeper modal interaction information, thereby obtaining richer feature representations and improving the accuracy of subsequent classification.

[0152] Step S160: Classify the EEG multimodal feature information to obtain the EEG signal classification result.

[0153] In one embodiment, the EEG multimodal feature information is classified according to classification requirements. In one embodiment, the classification requirements include: motor imagery, emotion classification, or mental state classification, that is, the EEG multimodal feature information is classified into motor imagery, emotion classification, or mental state classification to obtain the corresponding EEG signal classification results.

[0154] In one embodiment, reference is made to Figure 8 This is a flowchart illustrating a specific implementation of step S160 in one embodiment. In this embodiment, step S160, which involves classifying EEG multimodal feature information to obtain EEG signal classification results, includes steps S161 to S163:

[0155] Step S161: Input the EEG multimodal feature information into at least one fully connected layer to obtain classification feature information.

[0156] Step S162: Input the classification feature information into the classification layer to obtain the classification probability.

[0157] Step S163: Obtain the classification result of the EEG signal based on the classification probability.

[0158] In one embodiment, a fully connected layer classifier is used for EEG signal classification. Specifically, a fully connected layer is used to perform dimensionality reduction and feature extraction on the multimodal EEG features to obtain classification feature information. Then, a softmax layer is used as the classification layer to calculate the classification probability value corresponding to the classification feature information. The classification probability value is compared with the probability value corresponding to the classification category defined according to the classification requirements to determine which classification category the probability value belongs to, thereby obtaining the EEG signal classification result.

[0159] In one embodiment, the EEG signal classification method is encapsulated into an EEG signal classification model. This model is trained using a large number of samples to obtain optimal model weight parameters adapted to the scenario. Because it incorporates multimodal information for model training, the accuracy of the EEG signal classification results can be improved. Similarly, it can be designed as a program and ported to relevant devices for EEG signal classification and recognition. For example, wheelchair controllers, VR devices, or mobile terminals. In actual use, only EEG signal data needs to be acquired; the trained EEG signal classification model can then be used to classify the EEG signals. After obtaining the corresponding EEG signal classification result, further guided operations can be performed using this result. This embodiment does not impose specific limitations on this.

[0160] In one embodiment, reference is made to Figure 9 The diagram shows a flowchart of the EEG signal classification method according to an embodiment of this application. First, unprocessed EEG signal data is acquired. Then, preprocessing is performed on the EEG signal data to obtain preprocessed EEG signal data. Time-frequency features are constructed from the acquired EEG signal data to obtain EEG time-frequency feature signals. Next, the EEG time-frequency feature signals are input into a trained time-frequency information extraction model for time-frequency feature extraction, resulting in EEG time-series relationship signals represented by time-frequency features. Simultaneously, the EEG time-frequency feature signals are input into a trained graph embedding coding model for spatial feature extraction, resulting in EEG spatial relationship signals represented by spatial features. Then, cross-modal attention feature fusion is performed on the EEG time-series relationship signals and the EEG spatial relationship signals to obtain multimodal EEG feature information. Finally, a fully connected layer classifier is used to classify the multimodal EEG feature information to obtain the EEG signal classification result.

[0161] In this application embodiment, time-frequency feature extraction and spatial feature extraction are performed on the EEG time-frequency feature signal after the time-frequency feature is constructed. Then, the EEG time-series relationship signal and the EEG spatial relationship signal are fused in a multimodal manner. When classifying and recognizing, the multimodal feature information can be used to explore the deep-level interaction information between EEG signal features, improve the accuracy of EEG signal classification results, and expand the application scenarios of EEG signals.

[0162] This invention also provides an electroencephalogram (EEG) signal classification device, which can implement the above-described EEG signal classification method, as described above. Figure 10 The device includes:

[0163] Acquisition unit 1010 is used to acquire electroencephalogram (EEG) signal data.

[0164] The time-frequency feature construction unit 1020 is used to construct time-frequency features from EEG signal data to obtain EEG time-frequency feature signals.

[0165] The time-frequency information extraction unit 1030 is used to input the EEG time-frequency feature signal into the trained time-frequency information extraction model to extract the time-frequency features and obtain the EEG time sequence relationship signal.

[0166] The spatial feature extraction unit 1040 is used to input the EEG time-frequency feature signal into the trained graph embedding coding model for spatial feature extraction to obtain the EEG spatial relationship signal.

[0167] The multimodal feature fusion unit 1050 is used to fuse the temporal relationship signal and the spatial relationship signal of EEG to obtain EEG multimodal feature information.

[0168] The signal classification unit 1060 is used to classify the multimodal features of EEG to obtain the EEG signal classification results.

[0169] The specific implementation of the EEG signal classification device in this embodiment is basically the same as the specific implementation of the EEG signal classification method described above, and will not be repeated here.

[0170] This invention also provides an electronic device, comprising:

[0171] At least one memory;

[0172] At least one processor;

[0173] At least one program;

[0174] The program is stored in a memory, and the processor executes the at least one program to implement the electroencephalogram (EEG) signal classification method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0175] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0176] The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0177] The memory 1102 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1102 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and called and executed by the processor 1101 to execute the EEG signal classification method of the embodiments of this invention.

[0178] Input / output interface 1103 is used to implement information input and output;

[0179] Communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0180] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0181] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0182] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described EEG signal classification method.

[0183] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0184] The present invention provides an EEG signal classification method, EEG signal classification device, electronic device, and storage medium. In this embodiment, the time-frequency feature signal of the EEG signal after the time-frequency feature is constructed is extracted for time-frequency feature extraction and spatial feature extraction respectively. Then, the EEG temporal relationship signal and the EEG spatial relationship signal are fused in a multimodal manner. By utilizing the multimodal feature information during classification and recognition, the deep-level interaction information between EEG signal features can be mined, thereby improving the accuracy of EEG signal classification results and expanding the application scenarios of EEG signals.

[0185] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0186] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0187] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0189] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0190] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0192] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for classifying electroencephalogram (EEG) signals, characterized in that, include: Acquire electroencephalogram (EEG) signal data; The EEG signal data is subjected to time-frequency feature construction to obtain EEG time-frequency feature signals; The EEG time-frequency feature signal is convolved sequentially by three one-dimensional convolutional layers in the time-frequency information extraction model to obtain the deep semantic information of the EEG time-frequency feature signal; the deep semantic information is extracted by two cascaded time series networks in the time-frequency information extraction model to obtain the EEG time-series relationship signal, wherein the time series networks are Bi-LSTM network and LSTM network, respectively. The EEG time-frequency feature signal is transformed into a graph to obtain the graph representation information of the EEG time-frequency feature signal. The channels of the EEG time-frequency feature signal correspond to the nodes of the graph representation information, and the relationships between the channels of the EEG time-frequency feature signal correspond to the edges between the nodes of the graph representation information. The graph representation information is then input into the graph convolution module in the graph embedding coding model to perform graph convolution operation to obtain the EEG spatial relationship signal. The EEG temporal relationship signal and the EEG spatial relationship signal are fused with attention features to obtain EEG multimodal feature information; The EEG multimodal feature information is classified to obtain the EEG signal classification result.

2. The EEG signal classification method according to claim 1, characterized in that, The step of constructing time-frequency features from the EEG signal data to obtain EEG time-frequency feature signals includes: The EEG signal data is divided into one or more time segments by using a preset sliding window to perform sliding segmentation according to the temporal relationship; Perform a discrete Fourier transform on each of the time segment data to obtain more than one power spectral density information; By concatenating one or more power spectral density information, time-frequency feature data of the time segment data is formed; Each of the time-frequency feature data is spliced ​​together according to the time sequence relationship to obtain the EEG time-frequency feature signal.

3. The EEG signal classification method according to claim 1, characterized in that, The step of fusing the temporal and spatial EEG signals to obtain multimodal EEG feature information includes: The time-series EEG relationship signal is multiplied by the first weight matrix and the second weight matrix respectively, and then added together to obtain the time-frequency modal feature information; The spatial relationship signal of the EEG is multiplied by the third weight matrix to obtain the spatial modality feature information; The EEG multimodal feature information is obtained based on the time-frequency modal feature information and the spatial modal feature information.

4. The EEG signal classification method according to any one of claims 1 to 3, characterized in that, After acquiring the EEG signal data, the process also includes: The EEG signal data is preprocessed to obtain the preprocessed EEG signal data. The preprocessing includes at least one of the following: cropping operation, channel selection operation, downsampling operation, filtering operation, or baseline removal operation.

5. The EEG signal classification method according to any one of claims 1 to 3, characterized in that, The step of classifying the EEG multimodal feature information to obtain EEG signal classification results includes: The EEG multimodal feature information is input into at least one fully connected layer to obtain classification feature information; The classification feature information is input into the classification layer to obtain the classification probability; The classification result of the EEG signal is obtained based on the classification probability.

6. A brainwave signal classification device, characterized in that, include: Acquisition unit, used to acquire EEG signal data; A time-frequency feature construction unit is used to construct time-frequency features on the electroencephalogram (EEG) signal data to obtain an EEG time-frequency feature signal. The time-frequency information extraction unit is used to sequentially perform convolution operations on the EEG time-frequency feature signal using three one-dimensional convolutional layers in the time-frequency information extraction model to obtain the deep semantic information of the EEG time-frequency feature signal; and to extract contextual information from the deep semantic information using two cascaded time-series networks in the time-frequency information extraction model to obtain the EEG time-series relationship signal, wherein the time-series networks are a Bi-LSTM network and an LSTM network. A spatial feature extraction unit is used to perform graph transformation on the EEG time-frequency feature signal to obtain graph representation information of the EEG time-frequency feature signal. The channels of the EEG time-frequency feature signal correspond to the nodes of the graph representation information, and the relationships between the channels of the EEG time-frequency feature signal correspond to the edges between the nodes of the graph representation information. The graph representation information is input into the graph convolution module in the graph embedding coding model to perform graph convolution operation to obtain the EEG spatial relationship signal. A multimodal feature fusion unit is used to fuse the EEG temporal relationship signal and the EEG spatial relationship signal by attention features to obtain EEG multimodal feature information; The signal classification unit is used to classify the EEG multimodal feature information to obtain the EEG signal classification result.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the EEG signal classification method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the electroencephalogram (EEG) signal classification method according to any one of claims 1 to 5.

Images