Abnormal discharge detection method, device, medium and equipment between epileptic seizures
By combining feature extraction of EEG, ECG, EMG signals and video data with neural network model, the misjudgment problem in abnormal discharge detection during epilepsy is solved, the detection accuracy is improved, and the sleep artifacts are achieved Effectively distinguish.
Patent Information
- Application Number
- CN202311630571.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-11-30
AI Technical Summary
In the detection of abnormal discharge during epilepsy seizures, sleep artifacts are similar to the actual abnormal discharge waveform, resulting in frequent misjudgment and making it difficult to accurately identify abnormal discharge during epilepsy seizures.
A method is adopted to obtain the target user's EEG signal, ECG signal and EMG signal, and combine video data, and use the trained sleep prediction model and neural network model to perform signal preprocessing and feature extraction to distinguish sleep states. and abnormal discharge to improve detection accuracy.
Through the learning of video data and sleep staging, misjudgment of suspected abnormal discharges in sleep state is reduced, the accuracy of positive identification is improved, and the ability to detect abnormal discharges during epilepsy is enhanced.
Smart Images

Figure CN117598712B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of biomedical detection technology. More specifically, the embodiments of the present disclosure relate to a method for detecting abnormal discharges during epileptic seizures, a device for detecting abnormal discharges during epileptic seizures, a computer-readable storage medium, and an electronic device. Background Art
[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.
[0003] Epilepsy is a chronic neurological disease characterized by sudden abnormal brain reactions and recurrences. It is caused by abnormal activity of brain neurons. This epileptic seizure caused by excessive discharge of neurons is usually accompanied by disorders of movement, sensation, emotion or mental function. Once this sudden neurological disease occurs, the patient's brain will stop working normally, resulting in abnormal reactions such as fainting, body imbalance, convulsions, muscle contractions, loss of consciousness and intuition. For any epilepsy patient, epileptic seizures have a great impact on all aspects of the patient's and his family's life, and may even endanger the patient's life.
[0004] Electroencephalogram (EEG) is a non-invasive signal acquisition method used to clinically record brain-related activities. It has a high temporal resolution and plays an important role in the detection, diagnosis, treatment, efficacy evaluation and pathological research of epilepsy. At present, the diagnosis of epilepsy based on EEG is mainly the detection of epileptic seizures, however, epileptic seizures are not common. In contrast, nearly 80% of epilepsy patients have abnormal neuronal discharges between seizures. These abnormal discharges observable in EEG are called interictal epileptiform discharges (IED). IED is a strong support for epilepsy diagnosis or epileptic seizure risk assessment. In the evaluation of abnormal interictal discharges, artifact identification is a very important task. Sleep artifact is one of them. Summary of the invention
[0005] However, the EEG produced by the patient in the sleep state is very similar to the waveform of abnormal discharges between epilepsy periods, which leads to some misjudgments in the estimation.
[0006] Therefore, there is a great need for a method for detecting abnormal discharges between epileptic seizures to identify the patient's sleep state, so as to improve the accuracy of positive identification to a certain extent.
[0007] In this context, embodiments of the present disclosure are intended to provide a method for detecting abnormal discharges during epileptic seizures, an apparatus for detecting abnormal discharges during epileptic seizures, a computer-readable storage medium, and an electronic device.
[0008] According to a first aspect of the present disclosure, a method for detecting abnormal discharges between epileptic seizures is provided, the method comprising: obtaining an electrical signal to be tested and video data of a target user, and preprocessing the electrical signal to be tested to obtain a primary signal and a secondary signal of the electrical signal to be tested; the electrical signal to be tested comprises an electroencephalogram signal, an electrocardiogram signal and an electromyogram signal; the video data comprises a body image and a facial image; the electroencephalogram signal of a preset channel in the electrical signal to be tested is input into a trained sleep prediction model for processing, and a sleep prediction result is output; the trained sleep prediction model is obtained by training with a first training set, the first training set comprises a plurality of first training samples; the first training samples comprise electroencephalogram signal samples of preset channels in a sleep state and corresponding labels corrected based on video data, The preset channels at least include FZ channel and CZ channel; the primary signal, secondary signal, video data and sleep prediction result of the electric signal to be tested are input into the trained neural network model for processing, and the predicted value of abnormal discharge of the electric signal to be tested is output; the trained neural network model is trained by a second training set, and the second training set includes a plurality of second training samples; the second training samples include electric signals with abnormal discharge during the interictal period of epileptic patients and corresponding labels, electric signals without abnormal discharge during the interictal period of epileptic patients and corresponding labels, electric signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from the video data.
[0009] In one embodiment, the preprocessing of the electrical signal to be measured to obtain the primary signal and the secondary signal of the electrical signal to be measured includes: for each channel in the EEG signal, calculating the potential difference between the channel and the reference electrode corresponding to the channel to obtain the primary signal of the EEG signal; preprocessing the electrocardiogram signal and the electromyogram signal to obtain the primary signal of the electrocardiogram signal and the primary signal of the electromyogram signal, respectively; connecting the primary signal of the EEG signal, the primary signal of the electrocardiogram signal and the primary signal of the electromyogram signal to obtain the primary signal of the electrical signal to be measured; and determining the secondary signal of the electrical signal to be measured based on the primary signal of the electrical signal to be measured.
[0010] In one embodiment, the secondary signal of the electrical signal to be measured includes a time-frequency spectrum of the electrical signal to be measured; determining the secondary signal of the electrical signal to be measured based on the primary signal of the electrical signal to be measured includes: performing a short-time Fourier transform on the primary signal of the electrical signal to be measured to obtain a time-frequency spectrum of the electrical signal to be measured.
[0011] In one embodiment, the inputting the primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested into a trained neural network model for processing, and outputting a prediction value of the presence of abnormal discharge in the electrical signal to be tested, includes: acquiring the primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested; performing feature extraction on the primary signal and the secondary signal of the electrical signal to be tested to obtain a first intermediate feature; the first intermediate feature includes the original signal feature and the frequency spectrum feature of the electrical signal to be tested; performing feature extraction on the video data to obtain a second intermediate feature; the second intermediate feature includes the time-series body features, time-series facial features, body key point motion features, and facial key point motion features of the target user; performing feature extraction on the sleep prediction result to obtain a third intermediate feature; fusing the first intermediate feature, the second intermediate feature and the third intermediate feature to obtain a fused feature; performing dimensionality reduction on the fused feature to obtain an embedded feature; and determining the prediction value of the presence of abnormal discharge in the electrical signal to be tested according to the embedded feature.
[0012] In one embodiment, the feature extraction of the feature of the electrical signal to be measured to obtain the first intermediate feature includes: extracting the feature of the primary signal of the electrical signal to be measured to obtain the original signal feature of the electrical signal to be measured; extracting the feature of the secondary signal of the electrical signal to be measured to obtain the frequency spectrum feature of the electrical signal to be measured; and using the original signal feature and the frequency spectrum feature of the electrical signal to be measured as the first intermediate feature.
[0013] In one embodiment, the feature extraction of the video data to obtain the second intermediate feature includes: performing body feature extraction and facial feature extraction on the video data to obtain body region of interest, body information, facial region of interest, and facial information; performing feature extraction on the body region of interest and the facial region of interest, respectively, to obtain time-series body features and time-series facial features; performing frame difference calculation on the body information and the facial information to obtain body key point motion features and facial key point motion features.
[0014] In one embodiment, the preprocessing of the electrical signal to be measured further includes: performing at least one of resampling, filtering, denoising, segmentation, and numerical standardization on the electrical signal to be measured.
[0015] In one embodiment, the training process of the trained neural network model includes: obtaining a second training set; the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from video data; the second training set is input into the neural network model to obtain a predicted classification of the second training set; the predicted classification includes positive or negative; the positive indicates that the training sample has abnormal discharges, and the negative indicates that the training sample does not have abnormal discharges; according to the predicted classification of the second training set and the label of the second training set, the loss value of the neural network model is determined; according to the loss value, the neural network model is iteratively trained until a preset convergence condition is reached to obtain the trained neural network model.
[0016] In one embodiment, inputting the training set into a neural network model to obtain a predicted classification of the training set includes: inputting the second training set into a neural network model to obtain a predicted value for the presence of abnormal discharge in the second training sample; and determining a predicted classification of the second training sample based on the predicted value for the presence of abnormal discharge in the second training sample.
[0017] In one embodiment, the neural network model is iteratively trained according to the loss value until a preset convergence condition is reached to obtain a trained neural network model, including: updating model parameters of the neural network model according to the loss value; the model parameters include weights and biases; when the loss value meets the preset convergence condition, the training is terminated to obtain the trained neural network model.
[0018] According to a second aspect of the present disclosure, there is provided an abnormal discharge detection device during epileptic seizures, the device comprising: a data acquisition module, configured to acquire a target user's electrical signal to be tested and video data, and pre-process the electrical signal to be tested to obtain a primary signal and a secondary signal of the electrical signal to be tested; the electrical signal to be tested comprises an electroencephalogram signal, an electrocardiogram signal and an electromyogram signal; the video data comprises a body image and a facial image; a sleep prediction module, configured to input the electroencephalogram signal of a preset channel in the electrical signal to be tested into a trained sleep prediction model for processing, and output a sleep prediction result; the trained sleep prediction model is obtained by training with a first training set, the first training set comprises a plurality of first training samples; the first training samples comprise electroencephalogram signal samples of preset channels in a sleep state corrected based on video data and corresponding labels, The preset channels include at least an FZ channel and a CZ channel; an abnormal discharge prediction module is configured to input the primary signal, the secondary signal, the video data and the sleep prediction result of the electric signal to be tested into a trained neural network model for processing, and output a prediction value of the existence of abnormal discharge in the electric signal to be tested; the trained neural network model is trained by a second training set, and the second training set includes a plurality of second training samples; the second training samples include electric signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electric signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electric signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from the video data.
[0019] In one embodiment, the data acquisition module is configured to: for each channel in the EEG signal, calculate the potential difference between the channel and the potential of the reference electrode corresponding to the channel to obtain a primary signal of the EEG signal; preprocess the ECG signal and the EMG signal to obtain a primary signal of the ECG signal and a primary signal of the EMG signal, respectively; connect the primary signal of the EEG signal, the primary signal of the ECG signal, and the primary signal of the EMG signal to obtain the primary signal of the electrical signal to be measured; and determine the secondary signal of the electrical signal to be measured based on the primary signal of the electrical signal to be measured.
[0020] In one embodiment, the data acquisition module is configured to: perform short-time Fourier transform on the primary signal of the electrical signal to be measured to obtain a time-frequency spectrum diagram of the electrical signal to be measured.
[0021] In one embodiment, the abnormal discharge prediction module is configured to: obtain the primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested; perform feature extraction on the primary signal and the secondary signal of the electrical signal to be tested to obtain a first intermediate feature; the first intermediate feature includes the original signal feature and the frequency spectrum feature of the electrical signal to be tested; perform feature extraction on the video data to obtain a second intermediate feature; the second intermediate feature includes the temporal body features, temporal facial features, body key point motion features, and facial key point motion features of the target user; perform feature extraction on the sleep prediction result to obtain a third intermediate feature; fuse the first intermediate feature, the second intermediate feature and the third intermediate feature to obtain a fused feature; perform dimensionality reduction on the fused feature to obtain an embedded feature; and determine the prediction value of abnormal discharge of the electrical signal to be tested based on the embedded feature.
[0022] In one embodiment, the abnormal discharge prediction module is configured to: perform feature extraction on the primary signal of the electrical signal to be measured to obtain the original signal feature of the electrical signal to be measured; perform feature extraction on the secondary signal of the electrical signal to be measured to obtain the frequency spectrum feature of the electrical signal to be measured; and use the original signal feature and the frequency spectrum feature of the electrical signal to be measured as the first intermediate feature.
[0023] In one embodiment, the abnormal discharge prediction module is configured to: extract body features and facial features from the video data to obtain body regions of interest, body information, facial regions of interest, and facial information; perform feature extraction on the body regions of interest and the facial regions of interest, respectively, to obtain time-series body features and time-series facial features; perform frame difference calculation on the body information and the facial information to obtain body key point motion features and facial key point motion features.
[0024] In one embodiment, the data acquisition module is configured to perform at least one of resampling, filtering, denoising, segmentation, and numerical standardization on the electrical signal to be measured.
[0025] In one embodiment, the device also includes a model training module, and the model training module is configured to: obtain a second training set; the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from video data; the second training set is input into the neural network model to obtain a predicted classification of the second training set; the predicted classification includes positive or negative; the positive indicates that the training sample has abnormal discharges, and the negative indicates that the training sample does not have abnormal discharges; according to the predicted classification of the second training set and the label of the second training set, determine the loss value of the neural network model; according to the loss value, iteratively train the neural network model until a preset convergence condition is reached to obtain the trained neural network model.
[0026] In one embodiment, the model training module is configured to: input the second training set into the neural network model to obtain a prediction value of the presence of abnormal discharge in the second training sample; and determine a predicted classification of the second training sample based on the prediction value of the presence of abnormal discharge in the second training sample.
[0027] In one embodiment, the model training module is configured to: update the model parameters of the neural network model according to the loss value; the model parameters include weights and biases; when the loss value meets a preset convergence condition, end the training to obtain the trained neural network model.
[0028] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above methods is implemented.
[0029] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above methods by executing the executable instructions.
[0030] According to the abnormal discharge detection method during epileptic seizures, the abnormal discharge detection device during epileptic seizures, the computer-readable storage medium and the electronic device of the embodiments of the present disclosure, the target user's electrical signal to be tested and the video data are obtained, and the electrical signal to be tested is preprocessed to obtain the primary signal and the secondary signal of the electrical signal to be tested; the electrical signal to be tested includes an electroencephalogram signal, an electrocardiogram signal and an electromyogram signal; the video data includes a body image and a facial image; the electroencephalogram signal of a preset channel in the electrical signal to be tested is input into a trained sleep prediction model for processing, and a sleep prediction result is output; the trained sleep prediction model is obtained by training with a first training set, and the first training set includes a plurality of first training samples; the first training sample includes an electroencephalogram signal sample of a preset channel in a sleep state corrected based on the video data and a corresponding standard The invention relates to a method for preparing a neural network model for detecting abnormal discharge of an electric signal under a sleep state, and a method for detecting abnormal discharge of an electric signal under a sleep state. The method comprises: obtaining a sample of an electroencephalogram signal of a preset channel in a non-sleep state corrected based on video data and a corresponding label; the preset channel comprises at least an FZ channel and a CZ channel; the primary signal, the secondary signal, the video data and the sleep prediction result of the electric signal to be detected are input into a trained neural network model for processing, and outputting a predicted value of abnormal discharge of the electric signal to be detected; the trained neural network model is obtained by training a second training set, and the second training set comprises a plurality of second training samples; the second training samples comprise an electric signal of abnormal discharge between seizures of an epileptic patient and a corresponding label, an electric signal of an epileptic patient without abnormal discharge between seizures and a corresponding label, an electric signal of a non-epileptic patient and a corresponding label, sleep features, and video features; the video features are obtained by extracting features from the video data. In this way, by learning the video data and sleep stages, the model can reduce the situation where the electric signal of suspected abnormal discharge generated in the sleep state is judged as positive; thereby, the accuracy of positive judgment is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:
[0032] Figure 1 A schematic diagram of an abnormal discharge detection process architecture during an epileptic seizure period in an embodiment of the present disclosure is shown;
[0033] Figure 2 A flow chart showing a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0034] Figure 3 A schematic diagram showing the structure of a trained neural network model in a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0035] Figure 4 A flow chart showing the determination of primary signals and secondary signals in a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0036] Figure 5 A flowchart showing data processing of a trained neural network model in a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0037] Figure 6 A flowchart showing a first intermediate feature in a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0038] Figure 7 A flowchart showing a second intermediate feature in a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0039] Figure 8 A flowchart showing a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure for training a neural network model to obtain a trained neural network model is shown;
[0040] Fig. 9 A sub-flow chart of training a neural network model in a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0041] Fig.10 A sub-flow chart of training a neural network model in a method for detecting abnormal discharges between epileptic seizures in an embodiment of the present disclosure is shown;
[0042] Fig.11 A schematic diagram showing the structure of an abnormal discharge detection device during epileptic seizures in an embodiment of the present disclosure is shown;
[0043] Fig.12 A schematic structural diagram of an electronic device in an embodiment of the present disclosure is shown.
[0044] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION
[0045] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0046] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0047] According to an embodiment of the present disclosure, a method for detecting abnormal discharges during epileptic seizures, a device for detecting abnormal discharges during epileptic seizures, a computer-readable storage medium, and an electronic device are provided.
[0048] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.
[0049] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION
[0051] Epilepsy is a chronic neurological disease characterized by sudden abnormal brain reactions and recurrences. It is caused by abnormal activity of brain neurons. This epileptic seizure caused by excessive discharge of neurons is usually accompanied by disorders of movement, sensation, emotion or mental function. Once this sudden neurological disease occurs, the patient's brain will stop working normally, resulting in abnormal reactions such as fainting, body imbalance, convulsions, muscle contractions, loss of consciousness and intuition. For any epilepsy patient, epileptic seizures have a great impact on all aspects of the patient's and his family's life, and may even endanger the patient's life.
[0052] Electroencephalogram (EEG) is a non-invasive signal acquisition method used to clinically record brain-related activities. It has a high temporal resolution and plays an important role in the detection, diagnosis, treatment, efficacy evaluation and pathological research of epilepsy. At present, the diagnosis of epilepsy based on EEG is mainly the detection of epileptic seizures, however, epileptic seizures are not common. In contrast, nearly 80% of epilepsy patients have abnormal neuronal discharges between seizures. These abnormal discharges observable in EEG are called interictal epileptiform discharges (IEDs). IEDs are a strong support for epilepsy diagnosis or epileptic seizure risk assessment. Artifact identification is a very important task in the assessment of abnormal discharges between epilepsy periods. Sleep artifacts are one of them. However, the EEG generated by patients in the sleep state is very close to the waveform of abnormal discharges between epilepsy periods, which makes it expected that some misjudgments will occur.
[0053] In view of the above, the present disclosure provides a method for detecting abnormal discharges during epileptic seizures, an abnormal discharge detection device during epileptic seizures, a computer-readable storage medium, and an electronic device, which obtain the target user's electrical signal and video data to be tested, and pre-process the electrical signal to be tested to obtain the primary signal and secondary signal of the electrical signal to be tested; input the EEG signal of the preset channel in the electrical signal to be tested into the trained sleep prediction model for processing, and output the sleep prediction result; input the primary signal, secondary signal, video data, and sleep prediction result of the electrical signal to be tested into the trained neural network model for processing, and output the prediction value of the presence of abnormal discharge in the electrical signal to be tested. In this way, by learning the video data and sleep stages, the model can reduce the situation where the electrical signal suspected of abnormal discharge generated in the sleep state is judged as positive; thereby, the accuracy of positive judgments is improved.
[0054] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.
[0055] Application Scenario Overview
[0056] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.
[0057] The present disclosure can be applied to any scenario where it is necessary to identify abnormal discharges between seizures of epileptic patients. First, the target user's electrical signal and video data to be tested are obtained, and the electrical signal to be tested is preprocessed to obtain the primary signal and secondary signal of the electrical signal to be tested; secondly, the EEG signal of the preset channel in the electrical signal to be tested is input into the trained sleep prediction model for processing, and the sleep prediction result is output; finally, the primary signal, secondary signal, video data and sleep prediction result of the electrical signal to be tested are input into the trained neural network model for processing, and the prediction value of abnormal discharge in the electrical signal to be tested is output. In this way, by learning video data and sleep stages, the model can reduce the situation where the electrical signal suspected of abnormal discharge generated in the sleep state is judged as positive; thereby, the accuracy of positive judgment is improved.
[0058] Exemplary Methods
[0059] Combine the following Figure 1 The system architecture and application scenarios of the operating environment of this exemplary embodiment are exemplarily described.
[0060] Figure 1A schematic diagram of the system architecture is shown. The system architecture 100 may include an epileptic interictal abnormal discharge detection system 110. The epileptic interictal abnormal discharge detection system 110 may be a server or a cluster formed by multiple servers. The epileptic interictal abnormal discharge detection system 110 first obtains the target user's electrical signal and video data to be tested, and pre-processes the electrical signal to be tested to obtain the primary signal and secondary signal of the electrical signal to be tested. Secondly, the EEG signal of the preset channel in the electrical signal to be tested is input into the trained sleep prediction model for processing, and the sleep prediction result is output. Finally, the primary signal, secondary signal, video data and sleep prediction result of the electrical signal to be tested are input into the trained neural network model for processing, and the prediction value of abnormal discharge of the electrical signal to be tested is output. In this way, by learning the video data and sleep stages, the model can reduce the situation where the electrical signal suspected of abnormal discharge generated in the sleep state is judged as positive; thereby, the accuracy of positive judgment is improved.
[0061] The exemplary embodiment of the present disclosure first provides a method for detecting abnormal discharges between epileptic seizures, which may include:
[0062] Acquire the target user's electrical signals and video data to be tested, and pre-process the electrical signals to be tested to obtain the primary and secondary signals of the electrical signals to be tested; the electrical signals to be tested include brain electroencephalogram signals, electrocardiogram signals, and electromyography signals; the video data include body images and facial images;
[0063] Inputting the EEG signal of the preset channel in the electrical signal to be tested into the trained sleep prediction model for processing, and outputting the sleep prediction result; the trained sleep prediction model is obtained by training the first training set, and the first training set includes a plurality of first training samples; the first training samples include EEG signal samples of the preset channel in the sleep state corrected based on the video data and the corresponding labels, and EEG signal samples of the preset channel in the non-sleep state corrected based on the video data and the corresponding labels; the preset channels include at least the FZ channel and the CZ channel;
[0064] The primary signal, secondary signal, video data and sleep prediction result of the electrical signal to be tested are input into the trained neural network model for processing, and a prediction value of abnormal discharge of the electrical signal to be tested is output; the trained neural network model is trained by a second training set, and the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from video data.
[0065] Figure 2 The exemplary process of the abnormal discharge detection method between epileptic seizures is shown below. Figure 2Each step in the procedure is described in detail.
[0066] refer to Figure 2 In step S210, the electrical signal to be tested and the video data of the target user are obtained, and the electrical signal to be tested is preprocessed to obtain the primary signal and the secondary signal of the electrical signal to be tested.
[0067] The electrical signals to be tested include electroencephalogram signals, electrocardiogram signals and electromyography signals; and the video data include body images and facial images.
[0068] The electrical signal to be tested is an electrical signal collected from the object to be tested. The electrical signal is an active signal spontaneously generated by the physiological process of the living body, which can more accurately reflect the physiological state or physical state of the person, so that the electrical signal can be used to determine whether the living body is abnormal. Exemplarily, the above-mentioned electrical signal may include any one or more of the following: EEG signal, ECG signal, and EMG signal.
[0069] EEG signals can be electroencephalograms (EEGs) collected by multi-channel brain machines. EEGs are graphs obtained by amplifying and recording the spontaneous biopotentials of the cerebral cortex from the scalp using sophisticated instruments. They are spontaneous, rhythmic electrical activities of brain cell groups recorded by electrodes. This type of electrical activity is a plane diagram of the relationship between the potential and time, with the potential as the vertical axis and time as the horizontal axis. The frequency (period), amplitude, and phase of brain waves constitute the basic characteristics of EEGs.
[0070] The ECG signal can be an electrocardiogram collected by a multi-lead electrocardiogram machine. An electrocardiogram (ECG) refers to a graph of various potential changes drawn from the body surface by an electrocardiograph, with the pacemaker, atrium, and ventricle being excited successively in each cardiac cycle, accompanied by changes in the electrocardiogram bioelectricity. The electrocardiogram is an objective indicator of the occurrence, propagation, and recovery process of cardiac excitement.
[0071] Electromyography (EMG) is the superposition of action potentials of motor units in numerous muscle fibers in time and control, which can be obtained by sticking electromyography sensors on the skin.
[0072] The purpose of preprocessing the electrical signal to be tested is to extract representative features and facilitate subsequent data analysis. The primary signal of the electrical signal to be tested can be a signal obtained by preprocessing the electrical signal to be tested; the secondary signal of the electrical signal to be tested can be a signal obtained by processing the primary signal of the electrical signal to be tested; of course, the primary signal and the secondary signal of the electrical signal to be tested can also be electrical signals obtained by processing the electrical signal to be tested differently, which is not limited here.
[0073] Continue to refer Figure 2 In step S220, the EEG signal of the preset channel in the electrical signal to be tested is input into the trained sleep prediction model for processing, and the sleep prediction result is output.
[0074] Among them, the trained sleep prediction model is obtained by training the first training set, and the first training set includes multiple first training samples; the first training samples include preset channel EEG signal samples and corresponding labels in the sleep state corrected based on video data, and preset channel EEG signal samples and corresponding labels in the non-sleep state corrected based on video data; the preset channels include at least FZ channel and CZ channel (EEG's FZ and CZ channel).
[0075] The FZ channel and the CZ channel are two electrode locations in the EEG. The FZ channel is located in the center of the scalp, associated with the prefrontal lobe, and is usually used to monitor prefrontal activity. The CZ channel is located at the top of the scalp, associated with the parietal lobe, and is usually used to monitor activity in the central region. These two channels are one of the standard electrode locations commonly used in EEG.
[0076] The trained sleep prediction model can be obtained by training the sleep prediction model in the prior art, for example, by training SeqSleepNet, a sequence-to-sequence model based on multi-channel input, and DeepSleepNet, a model based on single-channel input, to obtain the trained sleep prediction model, which is not limited here.
[0077] The first training sample can be obtained by directly labeling the preset channel EEG signal samples in the sleep state and the preset channel EEG signal samples in the non-sleep state in combination with the video data and the time data; it can also be obtained by first inputting the preset channel EEG signal samples in the sleep state and the preset channel EEG signal samples in the non-sleep state into the sleep prediction model for sleep prediction, and then correcting the sleep prediction result in combination with the video data and the time data; for example: the video data and the time data show that the user's face and body have basically no changes from 22:00 to 5:00, then it is determined that the user is in a sleep state; for another example: the preset channel EEG signal sample is input into the sleep prediction model, and the output sleep prediction result is a sleep state, while the label of the preset channel EEG signal sample is a non-sleep state, then, in combination with the video data and the time data at this time, the video data and the time data show that the user's face and body have basically no changes from 22:00 to 5:00, then the label of the preset channel EEG signal sample is corrected from "non-sleep state" to "sleep state"; in this way, labeling or correcting the sample data in combination with the video data and the time data can improve the quality of the sample data.
[0078] Continue to refer Figure 2 In step S230, the primary signal, secondary signal, video data and sleep prediction result of the electrical signal to be tested are input into the trained neural network model for processing, and a prediction value of abnormal discharge of the electrical signal to be tested is output.
[0079] Among them, the trained neural network model is trained by the second training set, and the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; video features are obtained by extracting features from video data.
[0080] The neural network model may include an input layer, a feature extraction layer, a feature fusion layer, an embedding layer, and an output layer; wherein the input layer is used to obtain the features of the electrical signal to be tested, the video data, and the sleep prediction results in this step; the feature extraction layer is used to extract features from the data obtained by the input layer; the feature fusion layer is used to fuse the features output by the feature extraction layer; the embedding layer is used to map the fused features output by the feature fusion layer to map the fused features from the original space to the new space; the output layer is used to convert the output of the embedding layer into the final prediction value. Exemplarily, the neural network model can be trained to obtain the above-mentioned trained neural network model. Exemplarily, refer to Figure 3 , the structure of the trained neural network model is consistent with that of the neural network model.
[0081] Exemplarily, the electrical signals of abnormal discharges between seizures of epileptic patients and the corresponding labels, sleep characteristics, and video characteristics can be regarded as positive samples, and the electrical signals of epileptic patients without abnormal discharges between seizures and the corresponding labels, and the electrical signals of non-epileptic patients and the corresponding labels can be regarded as negative samples. Exemplarily, the electrical signals of abnormal discharges between seizures of epileptic patients and the corresponding labels are obtained by professional staff through identification and labeling of EEG signals, ECG signals, and EMG signals collected within the same period of time based on professional knowledge; in this way, firstly, the electrical signals suspected of abnormal discharges in EEG signals caused by ECG signals and / or EMG signals can be excluded, and then the electrical signals of abnormal discharges between seizures of epileptic patients and the corresponding labels are obtained; secondly, by learning video data and sleep stages, the model can reduce the situation where the electrical signals suspected of abnormal discharges generated in the sleep state are judged as positive; thereby, the accuracy of positive recognition is improved.
[0082] In one embodiment, in order to prevent interference from other bioelectric signals, the average lead of the electrical signal to be measured can be used as the feature of the electrical signal to be measured; specifically, refer to Figure 4The above step S210 of “preprocessing the electrical signal to be measured to obtain the primary signal and the secondary signal of the electrical signal to be measured” may further include the following steps S410 to S440:
[0083] Step S410: For each channel in the EEG signal, calculate the potential difference between the channel potential and the reference electrode corresponding to the channel to obtain a primary signal of the EEG signal.
[0084] Among them, the electroencephalogram is a waveform that records the potential difference between two electrodes on the scalp. Therefore, each lead must have two electrodes, one of which is connected to one input terminal of the EEG amplifier, and the other electrode is connected to the other input terminal of the amplifier. If there is a zero potential point on the human body, the potential difference between the electrode placed at this point and the other electrode placed on the scalp is the absolute value of the potential change at the latter electrode. We call the electrode placed at the zero potential point the reference electrode or the irrelevant electrode, and the electrode placed at a non-zero potential the active electrode or the active electrode. Therefore, the lead methods of EEG signal acquisition equipment are divided into two categories: unipolar lead method (one pole is the reference electrode and the other is the active electrode) and bipolar lead method (both poles are used as active electrodes).
[0085] The average lead is actually a type of unipolar lead. The reference electrode in the unipolar lead cannot maintain zero potential and is easily mixed with interference from other bioelectric signals. In order to overcome this shortcoming, the point where multiple active electrodes on the scalp are connected together through a certain resistance is used as the reference electrode, which is called the average reference electrode. The connection method between the active electrode and the average reference electrode is called the average lead.
[0086] In actual operation, different channels can use the same reference electrode, such as: unipolar lead, average lead, etc., which is not limited here.
[0087] Step S420: pre-process the ECG signal and the EMG signal to obtain a primary ECG signal and a primary EMG signal, respectively.
[0088] The preprocessing of the ECG signal and the EMG signal is mainly for data cleaning, which may include at least one of resampling, filtering, denoising, segmentation, and numerical standardization.
[0089] Step S430: connect the primary signal of the EEG signal, the primary signal of the ECG signal and the primary signal of the EMG signal to obtain the primary signal of the electrical signal to be measured.
[0090] The splicing in this step can be understood as feature splicing, that is, connecting two or more features in a certain order to form a longer feature, which can be spliced horizontally or vertically. For example, there are two features A = [1, 2, 3] and B = [4, 5, 6]. If they are spliced horizontally, the result is C = [1, 2, 3, 4, 5, 6]; if they are spliced vertically, the result is
[0091]
[0092] Step S440: Determine the secondary signal of the electrical signal to be measured based on the primary signal of the electrical signal to be measured.
[0093] Among them, the secondary signal of the electrical signal to be measured can be a more representative and richer feature mined on the basis of the primary signal of the electrical signal to be measured; for example: time domain spectrogram, frequency domain spectrogram, time-frequency spectrum diagram, etc.
[0094] In one embodiment, in order to improve the accuracy of the prediction value of abnormal discharge of the electrical signal to be tested, more abundant features can be obtained, such as: time-frequency spectrum; specifically, the secondary signal of the electrical signal to be tested includes the time-frequency spectrum of the electrical signal to be tested; the above step S330 can further include the following steps:
[0095] Perform short-time Fourier transform on the primary signal of the electrical signal to be measured to obtain a time-frequency spectrum diagram of the electrical signal to be measured.
[0096] Among them, the short-time Fourier transform (STFT) is a time-frequency analysis method that decomposes the signal in time and frequency and can analyze the short-time changes of the signal. Compared with the Fourier transform, the short-time Fourier transform can capture the local characteristics of the signal in time and frequency. Therefore, the short-time Fourier transform is more suitable for analyzing non-stationary signals.
[0097] When performing short-time Fourier transform, you first need to select a suitable window function to segment the signal. Commonly used window functions include Hanning window, rectangular window, Gaussian window, etc. The choice of window function should be determined according to the actual application scenario and characteristics to maximize the retention of the signal's spectral information.
[0098] Then, the window function is applied to each time period of the signal, and the signal within each window is Fourier transformed. In this way, the spectrum information of each time period can be obtained. Usually, in order to reduce the amount of calculation, you can choose to use the fast Fourier transform algorithm for calculation.
[0099] The obtained spectrum information can be represented as a two-dimensional matrix, where the horizontal axis represents time and the vertical axis represents frequency. Each element in the matrix represents the signal energy at the corresponding time and frequency. By visualizing this matrix, the changes of the signal in time and frequency can be observed more intuitively.
[0100] In one embodiment, the trained neural network model is used to predict the probability of abnormal discharge in the electrical signal to be tested. Therefore, the input data needs to be processed layer by layer; specifically, refer to Figure 5 , the above step S230 may further include the following steps S510 to S570:
[0101] Step S510: Acquire the primary signal, secondary signal, video data and sleep prediction result of the electrical signal to be tested.
[0102] This step can be performed in the input layer of a trained neural network model.
[0103] The input layer is the first layer of a trained neural network model that receives input data and passes it to the next layer but does not perform any operations on the input data; therefore, the input layer has no weights and bias values.
[0104] Step S520: extract features from the primary signal and the secondary signal of the electrical signal to be measured to obtain a first intermediate feature.
[0105] The first intermediate feature includes the original signal feature and the frequency spectrum feature of the electrical signal to be measured.
[0106] This step can be performed in the feature extraction layer of the trained neural network model. Figure 3 , the features of the electrical signals to be measured (EEG+ECG+EMG) can be extracted through the Transformer network and the EfficientNetV2 network; for details, refer to Figure 6 , step S420 may further include the following steps S610 to S630:
[0107] Step S610: extracting features from the primary signal of the electrical signal to be tested to obtain original signal features of the electrical signal to be tested.
[0108] Among them, this step is to obtain richer features and improve the accuracy of the prediction value of abnormal discharge in the electrical signal to be tested; the characteristics of the electrical signal to be tested on the time axis can be analyzed; specifically, the primary signal of the electrical signal to be tested can be converted into a time domain spectrum, and then sampling and analysis can be performed on the time domain spectrum to obtain the original signal characteristics (feature_raw_wave) of the electrical signal to be tested.
[0109] In actual operation, the feature extraction of the feature of the electrical signal to be measured can be performed at a fixed frequency, or it can be performed randomly, which is not limited here. Figure 3 , this step can be achieved through a pre-trained Transformer network.
[0110] Step S620: extract features from the secondary signal of the electrical signal to be measured to obtain frequency spectrum features of the electrical signal to be measured.
[0111] Among them, this step is to obtain richer features and improve the accuracy of the prediction value of abnormal discharge of the measured electrical signal; the frequency characteristics of the measured electrical signal can be analyzed; specifically, the frequency spectrum characteristics (feature_spectrum) of the measured electrical signal can be directly extracted from the secondary signal (time spectrum diagram) of the measured electrical signal. Figure 3 , this step can be achieved through the pre-trained EfficientNetV2 network.
[0112] Step S630: taking the original signal characteristics and the frequency spectrum characteristics of the electrical signal to be measured as the first intermediate characteristics.
[0113] In actual operation, only the original signal features or the frequency spectrum features may be used; if one wants to further improve the accuracy of the prediction value of the presence of abnormal discharge in the measured electrical signal, one may use both the original signal features and the frequency spectrum features.
[0114] Step S530: extract features from the video data to obtain second intermediate features.
[0115] Among them, the second intermediate features include the target user's temporal body features (feature_body), temporal facial features (feature_face), body key point motion features (feature_body_move_info), and facial key point motion features (feature_face_move_info).
[0116] The video data records the user's state over a period of time, and the user may have changes in action; therefore, the video data (Video) can be firstly extracted with the trained body model (Body models) and the trained face model (Face models); then the extracted features are analyzed to obtain the above-mentioned second intermediate features; for details, refer to Figure 7 , the above step S530 may further include the following steps S710 to S730:
[0117] Step S710: extract body features and facial features from the video data to obtain body regions of interest, body information, facial regions of interest, and facial information.
[0118] Among them, the Region of Interest (ROI) refers to a rectangular area or mask defined in the image, which is used to specify the part of the image we are interested in. The extraction of the region of interest is very important for image processing and computer vision tasks. It can be used in various applications such as object detection, image segmentation, feature extraction, etc.
[0119] By extracting the region of interest, we can limit the scope of attention, reduce the amount of image processing calculations, and improve processing efficiency. At the same time, the region of interest can also help us extract the target from the complex background, which is convenient for subsequent analysis and processing.
[0120] You can use color or texture features for segmentation, for example, use image segmentation algorithms (such as K-Means, GrabCut, etc.) to divide the image into different regions, and then use edge detection algorithms (such as the Canny algorithm) to identify the contours of the image and finally determine the region of interest. In addition, you can also use deep learning techniques, such as convolutional neural networks (CNN) to train a model to identify regions of interest in an image.
[0121] Body information (Body info) may include body key points, body key point location information, etc.; similarly, face information (Face info) may include facial key points, facial key point location information, etc.
[0122] Step S720: extract features from the body region of interest and the facial region of interest respectively to obtain time-series body features and time-series facial features.
[0123] This step can be implemented by a neural network, such as a convolutional neural network (CNN).
[0124] Step S730: Calculate the frame difference of the body information and the facial information to obtain the motion features of the key points of the body and the facial key points.
[0125] Among them, the frame difference calculation uses the image cache module to obtain the interval between two frames of images. According to the Y component input from the two frames, during the period of effective image signal, the Y component of the latter frame is subtracted from the Y component of the previous frame to obtain the difference. The difference is compared with the frame difference threshold to obtain a frame difference flag, which is determined to be a sign of motion. According to the frame difference flag, the pixels that are considered to have changed are assigned white pixel values, and the rest are assigned black pixel values. In this way, the obtained image is an image with only black and white pixels, among which the points with signs of motion are white, and the rest are black, thus paving the way for the subsequent picture frames.
[0126] Step S540: extract features from the sleep prediction results to obtain a third intermediate feature.
[0127] Among them, the sleep model in the prior art can be used for feature extraction, for example: SeqSleepNet, a sequence-to-sequence model based on multi-channel input, and DeepSleepNet, a model based on single-channel input, are used for feature extraction, which is not limited here. Figure 3 As shown, the third intermediate feature is sleep info.
[0128] Step S550: fuse the first intermediate feature, the second intermediate feature and the third intermediate feature to obtain a fused feature.
[0129] Among them, fusing features of different scales is an important means to improve segmentation performance. Low-level features have higher resolution and contain more position and detail information, but because they have undergone fewer convolutions, they have lower semantics and more noise. High-level features have stronger semantic information, but their resolution is very low and their perception of details is poor. At present, the performance of detection and segmentation is generally improved by fusing multiple layers.
[0130] In actual operation, the network in the prior art can be used for feature fusion; for example, refer to Figure 3 , a multi-layer perceptron (MLP) can be used to align the dimensions of the first intermediate feature, the second intermediate feature, and the third intermediate feature, and then concatenate them to obtain the above-mentioned fusion feature.
[0131] Step S560: reduce the dimension of the fused features to obtain embedded features.
[0132] In actual operation, this step can be implemented through the embedding layer. The embedding layer is actually a mapping that can map features from the original space to the new space; for example: from high-dimensional space to low-dimensional space, or from low-dimensional space to high-dimensional space; because the fusion features are relatively complex, this step can map the fusion features from high-dimensional space to low-dimensional space to retain more important features and discard less important features; specifically, the attention mechanism can be used to select more important features from the fusion features to obtain the above-mentioned low-dimensional embedding features.
[0133] Step S570: Determine, based on the embedded features, a prediction value of abnormal discharge in the electrical signal to be measured.
[0134] In actual operation, this step can be implemented through the output layer; the output layer usually uses a fully connected layer with an output dimension of 1; at the same time, the Softmax activation function is used to transform the output of all vectors under the same target supply node and convert them into values between 0 and 1 (i.e., predicted values).
[0135] Exemplary, reference Figure 3 Feature extraction can be performed using a trained body model and a trained facial model; specifically, feature extraction is performed on the trained body model to obtain body regions of interest and body information, and feature extraction is performed on the trained facial model to obtain facial regions of interest and facial information; feature extraction is then performed on the body regions of interest to obtain temporal body features; feature extraction is performed on the facial regions of interest to obtain temporal facial features; motion features of body key points are obtained by performing frame difference calculation on body information; motion features of facial key points are obtained by performing frame difference calculation on facial information.
[0136] In one implementation, in order to improve the accuracy of data analysis, the data may be preprocessed; specifically, the “preprocessing the electrical signal to be measured” in the above step S210 may further include the following steps:
[0137] At least one of resampling, filtering, denoising, segmenting, and numerical standardization is performed on the electrical signal to be measured.
[0138] Resampling refers to repeatedly extracting samples from the training set; these samples are then used to refit a specific model to retrieve more information about the fitted model. The purpose is to collect more information about the samples, improve accuracy, and estimate uncertainty.
[0139] Filtering is to filter out a specific frequency point or frequencies outside the frequency point to obtain a power signal with a specific frequency, or to eliminate a power signal with a specific frequency.
[0140] Denoising refers to removing noise from a signal; it can be done by moving average, median filtering, wavelet transform, and adaptive filtering.
[0141] Numerical standardization refers to the conversion of original data into dimensionless indicator evaluation values through certain mathematical transformations. That is, all indicator values are at the same quantitative level, which is conducive to comprehensive analysis and comparison. Numerical standardization can be carried out by standardization, normalization, centralization and other methods.
[0142] Segmentation refers to segmenting the electrical signals collected within a period of time according to certain time intervals to obtain more sample data or electrical signals to be tested.
[0143] In one implementation, the neural network model is trained to obtain the trained neural network model. Figure 8 The training process of the trained neural network model may include the following steps S810 to S840:
[0144] Step S810: Obtain a second training set.
[0145] Among them, the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; video features are obtained by extracting features from video data.
[0146] A training set can be obtained from an open source dataset. When there is no corresponding open source dataset for the electrical signal to be tested, manual labeling can be performed. Specifically, a certain number of electrical signals can be randomly selected from the obtained electrical signals as sample signals, and then the category corresponding to each sample signal can be labeled to generate sample signals and corresponding labels, thereby obtaining a training set.
[0147] Generally speaking, the more sample signals there are, the more accurate the model training results will be; but at the same time, the training time will be longer. Therefore, users can determine the appropriate number of sample signals according to their needs.
[0148] Step S820: input the second training set into the neural network model to obtain the predicted classification of the second training set.
[0149] Among them, the prediction classification includes positive or negative; positive means that the training sample has abnormal discharge, and negative means that the training sample does not have abnormal discharge.
[0150] The neural network model may include an input layer, a feature extraction layer, a feature fusion layer, an embedding layer, and an output layer. For example, the neural network model may be trained to obtain the trained neural network model. For example, refer to Figure 3 , the structure of the trained neural network model is consistent with that of the neural network model.
[0151] The classifier in the neural network model can be a binary classifier, and then output a positive prediction result or a negative prediction result. For details, refer to Fig. 9 , the above step S820 may further include the following steps S910 and S920:
[0152] Step S910: input the second training set into the neural network model to obtain a prediction value of the presence of abnormal discharge in the second training sample.
[0153] Among them, the neural network model may include an input layer, a feature extraction layer, a feature fusion layer, an embedding layer, and an output layer.
[0154] The output layer usually uses a fully connected layer with an output dimension of 1. At the same time, the Softmax activation function is used to transform the output of all vectors under the same target supply node and convert them into values between 0 and 1 (i.e., predicted values).
[0155] Step S920: Determine the predicted classification of the second training sample according to the predicted value of the second training sample having abnormal discharge.
[0156] The output layer can select the category with the highest probability as the predicted category of the input training sample.
[0157] Step S830: Determine the loss value of the neural network model according to the predicted classification of the second training set and the label of the second training set.
[0158] Among them, the prediction accuracy of the training set can be determined according to the prediction results of the training set and the label of the training set, and then the prediction accuracy of the training set can be subtracted from 1 to obtain the loss value of the neural network model; the loss value of the neural network model can also be determined based on the model output results and the label of the training set by using a loss function. For example, the loss of each iterative training can be calculated according to the cross entropy loss function.
[0159] Step S840: Iteratively train the neural network model according to the loss value until a preset convergence condition is reached to obtain a trained neural network model.
[0160] Among them, the preset convergence condition is an iteration number threshold or a loss value threshold set according to historical data or experience; then, when the iteration number threshold or the loss value threshold is reached, the iterative training is ended to obtain a trained neural network model.
[0161] In one embodiment, the trained neural network model is obtained by continuously adjusting the parameters of the model; specifically, refer to Fig.10 , the above step S840 may further include the following steps S1010 and S1020:
[0162] Step S1010: Update the model parameters of the neural network model according to the loss value.
[0163] Among them, model parameters include weights and biases.
[0164] Specifically, the weights and biases of the neural network can be updated using a back-propagation algorithm based on the loss value.
[0165] Step S1020: When the loss value meets the preset convergence condition, the training is terminated to obtain a trained neural network model.
[0166] Among them, the preset convergence condition is an iteration number threshold or a loss value threshold set according to historical data or experience; then, when the iteration number threshold or the loss value threshold is reached, the iterative training is ended to obtain a trained neural network model.
[0167] In actual operation, the test data can also be used to further test the neural network model trained by the training set. After the test passes, the neural network model that passes the test is used as a trained neural network model.
[0168] Exemplary Devices
[0169] After introducing the article rearrangement method according to the exemplary embodiment of the present disclosure, next, referring to Fig.11 An apparatus for detecting abnormal discharges between epileptic seizures according to an exemplary embodiment of the present disclosure will be described.
[0170] refer to Fig.11As shown, an abnormal discharge detection device 1100 during epileptic seizures includes: a data acquisition module 1110, configured to acquire the target user's electrical signal to be tested and video data, and pre-process the electrical signal to be tested to obtain the primary signal and secondary signal of the electrical signal to be tested; the electrical signal to be tested includes an electroencephalogram signal, an electrocardiogram signal and an electromyogram signal; the video data includes a body image and a facial image; a sleep prediction module 1120, configured to input the electroencephalogram signal of a preset channel in the electrical signal to be tested into a trained sleep prediction model for processing, and output a sleep prediction result; the trained sleep prediction model is obtained by training a first training set, and the first training set includes a plurality of first training samples; the first training samples include electroencephalogram signal samples of preset channels in a sleep state corrected based on the video data and corresponding labels, and Corrected preset channel EEG signal samples in a non-sleep state and corresponding labels; the preset channels include at least an FZ channel and a CZ channel; an abnormal discharge prediction module 1130 is configured to input the primary signal, secondary signal, video data and sleep prediction result of the electrical signal to be tested into a trained neural network model for processing, and output a prediction value of the presence of abnormal discharge in the electrical signal to be tested; the trained neural network model is trained by a second training set, and the second training set includes multiple second training samples; the second training samples include electrical signals of epileptic patients with abnormal discharges during the interictal period and corresponding labels, electrical signals of epileptic patients without abnormal discharges during the interictal period and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from video data.
[0171] In one embodiment, the data acquisition module 1110 is configured to: for each channel in the EEG signal, calculate the potential difference between the channel's potential and the reference electrode corresponding to the channel to obtain a primary signal of the EEG signal; preprocess the ECG signal and the EMG signal to obtain a primary signal of the ECG signal and a primary signal of the EMG signal, respectively; connect the primary signal of the EEG signal, the primary signal of the ECG signal, and the primary signal of the EMG signal to obtain the primary signal of the electrical signal to be measured; and determine the secondary signal of the electrical signal to be measured based on the primary signal of the electrical signal to be measured.
[0172] In one implementation, the data acquisition module 1110 is configured to: perform short-time Fourier transform on the primary signal of the electrical signal to be measured to obtain a time-frequency spectrum diagram of the electrical signal to be measured.
[0173] In one embodiment, the abnormal discharge prediction module 1130 is configured to: obtain the primary signal, secondary signal, video data and sleep prediction result of the electrical signal to be tested; perform feature extraction on the primary signal and secondary signal of the electrical signal to be tested to obtain a first intermediate feature; the first intermediate feature includes the original signal feature and frequency spectrum feature of the electrical signal to be tested; perform feature extraction on the video data to obtain a second intermediate feature; the second intermediate feature includes the temporal body features, temporal facial features, body key point motion features, and facial key point motion features of the target user; perform feature extraction on the sleep prediction result to obtain a third intermediate feature; fuse the first intermediate feature, the second intermediate feature and the third intermediate feature to obtain a fused feature; perform dimensionality reduction on the fused feature to obtain an embedded feature; and determine the prediction value of abnormal discharge of the electrical signal to be tested based on the embedded feature.
[0174] In one embodiment, the abnormal discharge prediction module 1130 is configured to: perform feature extraction on the primary signal of the electrical signal to be measured to obtain the original signal feature of the electrical signal to be measured; perform feature extraction on the secondary signal of the electrical signal to be measured to obtain the frequency spectrum feature of the electrical signal to be measured; and use the original signal feature and the frequency spectrum feature of the electrical signal to be measured as the first intermediate feature.
[0175] In one embodiment, the abnormal discharge prediction module 1130 is configured to: extract body features and facial features from video data to obtain body regions of interest, body information, facial regions of interest, and facial information; extract features from body regions of interest and facial regions of interest, respectively, to obtain time-series body features and time-series facial features; perform frame difference calculation on body information and facial information to obtain body key point motion features and facial key point motion features.
[0176] In one embodiment, the data acquisition module 1110 is configured to perform at least one of resampling, filtering, denoising, segmenting, and numerical standardization on the electrical signal to be measured.
[0177] In one embodiment, the device also includes a model training module, which is configured to: obtain a second training set; the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from video data; the second training set is input into the neural network model to obtain a predicted classification of the second training set; the predicted classification includes positive or negative; positive indicates that the training sample has abnormal discharges, and negative indicates that the training sample does not have abnormal discharges; according to the predicted classification of the second training set and the label of the second training set, determine the loss value of the neural network model; according to the loss value, iteratively train the neural network model until a preset convergence condition is reached to obtain a trained neural network model.
[0178] In one embodiment, the model training module is configured to: input the second training set into the neural network model to obtain a prediction value of abnormal discharge in the second training sample; and determine the predicted classification of the second training sample based on the prediction value of abnormal discharge in the second training sample.
[0179] In one embodiment, the model training module is configured to: update the model parameters of the neural network model according to the loss value; the model parameters include weights and biases; when the loss value meets the preset convergence conditions, end the training and obtain a trained neural network model.
[0180] Exemplary Storage Media
[0181] The storage medium according to the exemplary embodiment of the present disclosure is described below.
[0182] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus, or a device.
[0183] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0184] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0185] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.
[0186] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0187] Exemplary Electronic Devices
[0188] refer to Fig.12 An electronic device according to an exemplary embodiment of the present disclosure is described.
[0189] Fig.12 The electronic device 1200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0190] like Fig.12 As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one storage unit 1220, a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210), and a display unit 1240.
[0191] The storage unit stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1210 can perform the following steps: Figure 1 The method steps shown, etc.
[0192] The storage unit 1220 may include a volatile storage unit, such as a random access storage unit (RAM) 1221 and / or a cache storage unit 1222 , and may further include a read-only storage unit (ROM) 1223 .
[0193] The storage unit 1220 may also include a program / utility 1224 having a set (at least one) of program modules 1225, such program modules 1225 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0194] The bus 1230 may include a data bus, an address bus, and a control bus.
[0195] The electronic device 1200 may also communicate with one or more external devices 2000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed through an input / output (I / O) interface 1250. The electronic device 1200 also includes a display unit 1240, which is connected to the input / output (I / O) interface 1250 for display. In addition, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 through a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0196] It should be noted that, although several modules or submodules of the device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.
[0197] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0198] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.
Claims
1. A method for detecting abnormal discharges between epileptic seizures, characterized in that: The method comprises: Acquire the target user's electrical signal and video data to be tested, and pre-process the electrical signal to be tested to obtain the primary signal and secondary signal of the electrical signal to be tested; the electrical signal to be tested includes an electroencephalogram signal, an electrocardiogram signal, and an electromyography signal; the video data includes a body image and a facial image; Input the EEG signal of the preset channel in the electrical signal to be tested into a trained sleep prediction model for processing, and output a sleep prediction result; the trained sleep prediction model is obtained by training a first training set, and the first training set includes a plurality of first training samples; the first training samples include EEG signal samples of preset channels in a sleep state corrected based on video data and corresponding labels, and EEG signal samples of preset channels in a non-sleep state corrected based on video data and corresponding labels; the preset channels include at least an FZ channel and a CZ channel; The primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested are input into a trained neural network model for processing, and a prediction value of the presence of abnormal discharge in the electrical signal to be tested is output; the trained neural network model is trained by a second training set, and the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from the video data, and the sleep features are obtained by extracting features from the sleep prediction results.
2. The method according to claim 1, characterized in that The preprocessing of the electrical signal to be tested to obtain a primary signal and a secondary signal of the electrical signal to be tested includes: For each channel of the EEG signal, calculating the potential difference between the channel and the reference electrode corresponding to the channel to obtain a primary signal of the EEG signal; Preprocessing the electrocardiogram signal and the electromyogram signal to obtain a primary signal of the electrocardiogram signal and a primary signal of the electromyogram signal respectively; Connecting the primary signal of the EEG signal, the primary signal of the ECG signal and the primary signal of the EMG signal to obtain the primary signal of the electrical signal to be tested; Based on the primary signal of the electrical signal to be measured, a secondary signal of the electrical signal to be measured is determined.
3. The method according to claim 2, characterized in that The secondary signal of the electrical signal to be measured includes a time-frequency spectrum of the electrical signal to be measured; and determining the secondary signal of the electrical signal to be measured based on the primary signal of the electrical signal to be measured includes: Performing short-time Fourier transform on the primary signal of the electrical signal to be measured to obtain a time-frequency spectrum diagram of the electrical signal to be measured.
4. The method according to claim 1, characterized in that: The step of inputting the primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested into a trained neural network model for processing, and outputting a prediction value of abnormal discharge of the electrical signal to be tested, comprises: Acquire the primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested; Extracting features of the primary signal and the secondary signal of the electrical signal to be measured to obtain a first intermediate feature; the first intermediate feature includes original signal features and frequency spectrum features of the electrical signal to be measured; Extracting features from the video data to obtain second intermediate features; the second intermediate features include temporal body features, temporal facial features, body key point motion features, and facial key point motion features of the target user; Performing feature extraction on the sleep prediction result to obtain a third intermediate feature; Fusing the first intermediate feature, the second intermediate feature, and the third intermediate feature to obtain a fused feature; Performing dimension reduction on the fused features to obtain embedded features; A prediction value of abnormal discharge in the electrical signal to be measured is determined according to the embedded feature.
5. The method according to claim 4, characterized in that The extracting the feature of the electrical signal to be measured to obtain the first intermediate feature includes: Extracting features of the primary signal of the electrical signal to be measured to obtain original signal features of the electrical signal to be measured; Extracting features of the secondary signal of the electrical signal to be measured to obtain frequency spectrum features of the electrical signal to be measured; The original signal characteristics and frequency spectrum characteristics of the electrical signal to be measured are used as the first intermediate characteristics.
6. The method according to claim 4, characterized in that The extracting features from the video data to obtain the second intermediate features includes: Performing body feature extraction and facial feature extraction on the video data to obtain a body region of interest, body information, a facial region of interest, and facial information; Extracting features from the body region of interest and the facial region of interest respectively to obtain time-series body features and time-series facial features; Frame difference calculation is performed on the body information and the facial information to obtain body key point motion features and facial key point motion features.
7. The method according to claim 3, characterized in that The preprocessing of the electrical signal to be tested further includes: At least one of resampling, filtering, denoising, segmentation, and numerical standardization is performed on the electrical signal to be measured.
8. The method according to claim 1, characterized in that: The training process of the trained neural network model includes: Acquire a second training set; the second training set includes a plurality of second training samples; the second training samples include electrical signals of epileptic patients with abnormal discharges during interictal periods and corresponding labels, electrical signals of epileptic patients without abnormal discharges during interictal periods and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from video data; Inputting the second training set into the neural network model to obtain a predicted classification of the second training set; the predicted classification includes positive or negative; the positive indicates that the training sample has abnormal discharge, and the negative indicates that the training sample does not have abnormal discharge; Determining a loss value of the neural network model according to the predicted classification of the second training set and the label of the second training set; According to the loss value, the neural network model is iteratively trained until a preset convergence condition is reached to obtain the trained neural network model.
9. The method according to claim 8, characterized in that The step of inputting the second training set into the neural network model to obtain a predicted classification of the second training set includes: Inputting the second training set into the neural network model to obtain a prediction value of the presence of abnormal discharge in the second training sample; The predicted classification of the second training sample is determined according to the predicted value that the second training sample has abnormal discharge.
10. The method according to claim 8, characterized in that The neural network model is iteratively trained according to the loss value until a preset convergence condition is reached to obtain a trained neural network model, including: According to the loss value, updating the model parameters of the neural network model; the model parameters include weights and biases; When the loss value meets the preset convergence condition, the training is terminated to obtain the trained neural network model.
11. A device for detecting abnormal discharges between epileptic seizures, characterized in that: The device comprises: A data acquisition module is configured to acquire the target user's electrical signal to be tested and video data, and pre-process the electrical signal to be tested to obtain the primary signal and secondary signal of the electrical signal to be tested; the electrical signal to be tested includes an electroencephalogram signal, an electrocardiogram signal, and an electromyography signal; the video data includes a body image and a facial image; The sleep prediction module is configured to input the EEG signal of the preset channel in the electrical signal to be tested into a trained sleep prediction model for processing, and output a sleep prediction result; the trained sleep prediction model is obtained by training a first training set, and the first training set includes a plurality of first training samples; the first training samples include EEG signal samples of the preset channel in the sleep state corrected based on the video data and corresponding labels, and EEG signal samples of the preset channel in the non-sleep state corrected based on the video data and corresponding labels; the preset channels include at least an FZ channel and a CZ channel; The abnormal discharge prediction module is configured to input the primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested into a trained neural network model for processing, and output a prediction value of the presence of abnormal discharge in the electrical signal to be tested; the trained neural network model is trained by a second training set, and the second training set includes multiple second training samples; the second training samples include electrical signals with abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals without abnormal discharges during the interictal period of epileptic patients and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; the video features are obtained by extracting features from the video data, and the sleep features are obtained by extracting features from the sleep prediction results.
12. The device according to claim 11, characterized in that The data acquisition module is configured as follows: For each channel of the EEG signal, calculating the potential difference between the channel and the reference electrode corresponding to the channel to obtain a primary signal of the EEG signal; Preprocessing the electrocardiogram signal and the electromyogram signal to obtain a primary signal of the electrocardiogram signal and a primary signal of the electromyogram signal respectively; Connecting the primary signal of the EEG signal, the primary signal of the ECG signal and the primary signal of the EMG signal to obtain the primary signal of the electrical signal to be tested; Based on the primary signal of the electrical signal to be measured, a secondary signal of the electrical signal to be measured is determined.
13. The device according to claim 12, characterized in that The data acquisition module is configured as follows: Performing short-time Fourier transform on the primary signal of the electrical signal to be measured to obtain a time-frequency spectrum diagram of the electrical signal to be measured.
14. The device according to claim 11, characterized in that The abnormal discharge prediction module is configured as follows: Acquire the primary signal, the secondary signal, the video data and the sleep prediction result of the electrical signal to be tested; Extracting features of the primary signal and the secondary signal of the electrical signal to be measured to obtain a first intermediate feature; the first intermediate feature includes original signal features and frequency spectrum features of the electrical signal to be measured; Extracting features from the video data to obtain second intermediate features; the second intermediate features include temporal body features, temporal facial features, body key point motion features, and facial key point motion features of the target user; Performing feature extraction on the sleep prediction result to obtain a third intermediate feature; Fusing the first intermediate feature, the second intermediate feature, and the third intermediate feature to obtain a fused feature; Performing dimension reduction on the fused features to obtain embedded features; A prediction value of abnormal discharge in the electrical signal to be measured is determined according to the embedded feature.
15. The device according to claim 14, characterized in that The abnormal discharge prediction module is configured as follows: Extracting features of the primary signal of the electrical signal to be measured to obtain original signal features of the electrical signal to be measured; Extracting features of the secondary signal of the electrical signal to be measured to obtain frequency spectrum features of the electrical signal to be measured; The original signal characteristics and frequency spectrum characteristics of the electrical signal to be measured are used as the first intermediate characteristics.
16. The device according to claim 14, characterized in that The abnormal discharge prediction module is configured as follows: Performing body feature extraction and facial feature extraction on the video data to obtain a body region of interest, body information, a facial region of interest, and facial information; Extracting features from the body region of interest and the facial region of interest respectively to obtain time-series body features and time-series facial features; Frame difference calculation is performed on the body information and the facial information to obtain body key point motion features and facial key point motion features.
17. The device according to claim 13, characterized in that The data acquisition module is configured as follows: At least one of resampling, filtering, denoising, segmentation, and numerical standardization is performed on the electrical signal to be measured.
18. The device according to claim 11, characterized in that The device also includes a model training module, which is configured to: Acquire a second training set; the second training set includes a plurality of second training samples; the second training samples include electrical signals of epileptic patients with abnormal discharges during interictal periods and corresponding labels, electrical signals of epileptic patients without abnormal discharges during interictal periods and corresponding labels, electrical signals of non-epileptic patients and corresponding labels, sleep features, and video features; The video features are obtained by extracting features from video data; Inputting the second training set into the neural network model to obtain a predicted classification of the second training set; the predicted classification includes positive or negative; the positive indicates that the training sample has abnormal discharge, and the negative indicates that the training sample does not have abnormal discharge; Determining a loss value of the neural network model according to the predicted classification of the second training set and the label of the second training set; According to the loss value, the neural network model is iteratively trained until a preset convergence condition is reached to obtain the trained neural network model.
19. The device according to claim 18, characterized in that The model training module is configured as follows: Inputting the second training set into the neural network model to obtain a prediction value of the presence of abnormal discharge in the second training sample; The predicted classification of the second training sample is determined according to the predicted value that the second training sample has abnormal discharge.
20. The device according to claim 18, characterized in that The model training module is configured as follows: According to the loss value, updating the model parameters of the neural network model; the model parameters include weights and biases; When the loss value meets the preset convergence condition, the training is terminated to obtain the trained neural network model.
21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.
22. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 10 by executing the executable instructions.
