Method, device, electronic device and computer storage medium for determining epileptic discharge

Abstract

The embodiments of the present application provide a method, device, electronic device and computer storage medium for determining epileptic discharges, and relate to the field of computer technology. The method includes: obtaining EEG characteristic data of the subject; obtaining a monitoring video of the subject; inputting the EEG characteristic data and the monitoring video into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model. It can be seen that the implementation of the embodiments of the present application can improve the accuracy and efficiency of determining whether the subject is currently in an abnormal EEG discharge state by inputting the EEG characteristic data and the monitoring video into a trained epileptic discharge model to determine whether the subject is currently in an abnormal EEG discharge state through the epileptic discharge model.

Description

Background Art

[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Currently, epilepsy is a common disease of the nervous system. Epileptic discharges are an important marker of epilepsy, and their clinical significance has been widely confirmed. However, the judgment of epileptic discharges still relies on medical experts, which leads to low consistency in the judgment of epileptic discharges, and thus low accuracy in judging abnormal EEG discharges.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] Based on the above problems, the inventors have conducted corresponding thinking and made targeted improvements, providing an epileptic discharge determination method, an epileptic discharge determination device, an electronic device and a computer-readable storage medium, which can input EEG feature data and monitoring video into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model, thereby improving the accuracy of determining the abnormal EEG discharge state.

[0006] According to a first aspect of an embodiment of the present application, a method for determining epileptic discharges is disclosed, comprising:

[0007] Acquiring EEG characteristic data of the subject;

[0008] Acquiring a monitoring video of the subject;

[0009] The EEG characteristic data and the monitoring video are input into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0010] In one embodiment, based on the above scheme, the EEG characteristic data is multi-channel EEG monitoring data collected from multiple parts of the subject's head, and the EEG characteristic data includes EEG signal waveform characteristics;

[0011] The step of obtaining the EEG characteristic data of the subject includes:

[0012] Calculating the potential difference between each channel and the reference electrode according to the multi-channel EEG monitoring data to obtain initial characteristic data of the EEG signal;

[0013] Generate a corresponding EEG signal waveform diagram according to the initial EEG signal characteristic data;

[0014] The EEG signal waveform features are extracted according to the EEG signal waveform diagram.

[0015] In one embodiment, based on the above solution, after acquiring the monitoring video of the object under test, the method further includes:

[0016] Obtaining human body feature information and motion feature information of the subject from the monitoring video;

[0017] The EEG signal waveform characteristics, the human body characteristic information and the motion characteristic information are input into a trained epileptic discharge model, so as to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0018] In one embodiment, based on the above solution, obtaining the human body feature information and motion feature information of the subject from the monitoring video includes:

[0019] Extracting a plurality of frame images corresponding to the monitoring video respectively;

[0020] Detecting object key points corresponding to the detected object and key point spatial coordinates corresponding to the object key points from the multiple frame images respectively; the object key points include object facial key points and body key points;

[0021] Constructing human body feature information according to the spatial coordinates of all key points of the tested object for the key points of the target object;

[0022] The human body feature information is subjected to human body motion analysis to obtain the motion feature information of the subject.

[0023] In one embodiment, based on the above solution, the step of inputting the EEG signal waveform features, the human body feature information, and the motion feature information into the trained epileptic discharge model includes:

[0024] Taking a preset time point as a starting time, and based on a preset time interval, dividing the EEG signal waveform feature, the human body feature information, and the motion feature information respectively to obtain a plurality of divided segments;

[0025] The multiple segmented segments are input into the trained epileptic discharge model.

[0026] In one embodiment, based on the above scheme, the epileptic discharge model is a three-dimensional convolutional neural network, a multi-layer perceptron and a fully connected layer;

[0027] The step of inputting the EEG characteristic data and the monitoring video into a trained epileptiform discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptiform discharge model includes:

[0028] Inputting the EEG signal waveform feature into the three-dimensional convolutional neural network to obtain EEG feature information; wherein the EEG feature information includes spatial position information of each channel;

[0029] Inputting the human body feature information and the action feature information into the multi-layer perceptron to obtain motion feature information, and concatenating the EEG feature information and the motion feature information to obtain comprehensive features;

[0030] The comprehensive features are input into the fully connected layer to obtain a predicted probability value; wherein the predicted probability value is used to characterize whether the subject is in an abnormal EEG discharge state.

[0031] In one embodiment, based on the above scheme, the three-dimensional convolutional neural network includes a three-dimensional convolutional layer, a pooling layer and a fully connected layer;

[0032] The step of inputting the EEG signal waveform feature into the three-dimensional convolutional neural network to obtain EEG feature information includes:

[0033] Using the three-dimensional convolutional layer to extract the features of the EEG signal waveform to obtain a feature map;

[0034] Using the pooling layer to perform feature compression on the feature map to obtain a target feature map;

[0035] The target feature map is processed by using the fully connected layer to reduce the dimension, so as to obtain EEG feature information.

[0036] In one embodiment, based on the foregoing solution, the three-dimensional convolutional layer includes multiple feature channels;

[0037] The step of extracting the features of the EEG signal waveform using the three-dimensional convolutional layer to obtain a feature map includes:

[0038] Using the multiple feature channels in the three-dimensional convolutional layer to extract features of the EEG signal waveform, respectively, to obtain channel feature maps corresponding to the multiple feature channels;

[0039] Determine preset weights set for the multiple feature channels respectively, and obtain a feature map based on the preset weights and a channel feature map corresponding to the preset weights.

[0040] In one embodiment, based on the above solution, the method further includes:

[0041] Acquire training samples, wherein the training samples include a plurality of patient data corresponding to a plurality of epilepsy patients; the plurality of patient data respectively include EEG feature data, patient monitoring videos, and EEG abnormal discharge state labels; the EEG abnormal discharge state labels are used to indicate whether the epilepsy patient is in an EEG abnormal discharge state;

[0042] Inputting the training sample into the initial epileptic discharge model to be trained to obtain a model output result; the model output result is used to characterize whether the epilepsy patient is in an abnormal EEG discharge state;

[0043] Determining the loss value of the initial epileptic discharge model to be trained according to the model output result and the abnormal EEG discharge state label;

[0044] According to the loss value, the initial epileptic discharge model to be trained is iteratively trained until a trained epileptic discharge model is obtained.

[0045] In one embodiment, based on the aforementioned scheme, the training samples include positive samples and negative samples, the positive samples include multiple patient data corresponding to multiple epilepsy patients with abnormal EEG discharges, and the negative samples include multiple patient data corresponding to multiple epilepsy patients with normal EEG discharges.

[0046] In one embodiment, based on the above scheme, the step of inputting the training sample into the initial epileptic discharge model to be trained to obtain the model output result includes:

[0047] Inputting the training sample into the initial epileptic discharge model to be trained to obtain a prediction label; the prediction label indicates whether the epilepsy patient is in an abnormal EEG discharge state;

[0048] Determine the model output result corresponding to the training sample according to the predicted label.

[0049] In one embodiment, based on the above scheme, the iterative training of the initial epileptic discharge model to be trained according to the loss value until a trained epileptic discharge model is obtained includes:

[0050] According to the loss value, updating the model parameters of the initial epileptic discharge model to be trained;

[0051] When the loss value satisfies a preset convergence condition, a trained epileptic discharge model is obtained.

[0052] According to a second aspect of an embodiment of the present application, a device for determining epileptic discharge is disclosed, the device comprising:

[0053] An EEG characteristic data acquisition unit is used to acquire EEG characteristic data of a subject;

[0054] A monitoring video acquisition unit is used to acquire the monitoring video of the object under test;

[0055] The abnormal discharge determination unit is used to input the EEG characteristic data and the monitoring video into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0056] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, comprising: a processor; and a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the method for determining epileptic discharges disclosed in the first aspect is implemented.

[0057] According to the fourth aspect of the embodiments of the present application, a computer program medium is disclosed, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the epileptic discharge determination method disclosed in the first aspect of the present application.

[0058] The embodiment of the present application can input EEG characteristic data and monitoring video into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model. Compared with the prior art, the implementation of the embodiment of the present application avoids determining whether the subject is in an abnormal EEG discharge state through consultation with an expert doctor, and combines EEG characteristic data and monitoring video to improve the accuracy and efficiency of determining whether the subject is in an abnormal EEG discharge state.

[0059] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0060] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0062] Figure 1 Shown is a flow chart of a method for determining epileptiform discharge according to an exemplary embodiment of the present application;

[0063] Figure 2What is shown is an EEG signal waveform diagram generated according to the initial feature data of the EEG signal in an example implementation mode of the present application;

[0064] Figure 3 Shown is a head-mounted device for collecting monitoring videos according to an example embodiment of the present application;

[0065] Figure 4 Shown is a sensor for determining motion feature information according to an example implementation of the present application;

[0066] Figure 5 Shown is a schematic diagram of key points of a target object according to an example implementation of the present application;

[0067] Figure 6 Shown is a hand joint capturing glove for determining hand motion feature information according to an example implementation of the present application;

[0068] Figure 7 Shown is a schematic diagram of a three-dimensional convolutional neural network according to an example implementation of the present application;

[0069] Figure 8 Shown is a schematic diagram of an epileptic discharge model according to an exemplary embodiment of the present application;

[0070] Fig. 9 What is shown is a schematic diagram of the result of a three-dimensional convolutional neural network according to an example implementation of the present application;

[0071] Fig.10 What is shown is a flow chart of training an initial epileptic discharge model to be trained to obtain a trained epileptic discharge model according to an optional exemplary implementation of the present application;

[0072] Fig.11 What is shown is a schematic diagram of a flow chart of determining whether the subject is in an abnormal EEG discharge state at this time according to an optional implementation mode of the present application;

[0073] Fig.12 The figure shows a schematic diagram of a scenario for determining whether a subject is in an abnormal EEG discharge state before surgery or during hospitalization according to an exemplary embodiment of the present application;

[0074] Fig.13 Shown is a structural block diagram of an epileptic discharge determination device according to another optional exemplary embodiment of the present application;

[0075] Fig.14 Shown is an electronic device according to another alternative exemplary embodiment of the present application.

[0076] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0077] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0078] Those skilled in the art know that the embodiments of the present application can be implemented as a device, apparatus, method or computer program product. Therefore, the present application can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0079] According to the embodiments of the present application, a method for determining an epileptic discharge, an apparatus for determining an epileptic discharge, an electronic device, and a computer-readable storage medium are proposed.

[0080] Any number of elements in the drawings is for illustration and not limitation, and any naming is for distinction only and does not have any limiting meaning.

[0081] The principles and spirit of the present application are explained in detail below with reference to several representative implementations of the present application. SUMMARY OF THE INVENTION

[0083] At present, the existing technology for determining the abnormal EEG discharge state usually relies on experienced specialists, neurosurgeons, and radiologists to assist in determining whether the patient is currently in an abnormal EEG discharge state. However, in the process of determination, there will be inconsistencies in the conclusions given by doctors. Therefore, the above method reduces the accuracy and efficiency of determining the abnormal EEG discharge state.

[0084] Based on the above problems, the applicant has thought of avoiding the phenomenon of multiple doctors assisting in determining the abnormal EEG discharge state. By inputting EEG feature data and monitoring video into a trained epileptic discharge model, the epileptic discharge model can be used to determine whether the subject is currently in an abnormal EEG discharge state. It can be seen that the epileptic discharge model can be used to determine whether the subject is currently in an abnormal EEG discharge state, avoiding the phenomenon of multiple doctors assisting in determining in the prior art, and improving the accuracy and efficiency of determining whether the subject is in an abnormal EEG discharge state.

[0085] Application Scenario Overview

[0086] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0087] When applied to the scenario of judging abnormal EEG discharge of the subject, it can avoid the phenomenon of multiple doctors assisting in the determination, thereby improving the accuracy and efficiency of determining whether the subject is in a state of abnormal EEG discharge.

[0088] Exemplary Methods

[0089] In combination with the above application scenarios, refer to Figure 1 and Fig.14 A method for determining epileptiform discharge according to an exemplary embodiment of the present application is described below.

[0090] See also Figure 1 , Figure 1 FIG. 2 is a flow chart of a method for determining epileptiform discharge according to an exemplary embodiment of the present application. Figure 1 As shown, the epileptic discharge determination method may include:

[0091] Step S110: Acquire the EEG characteristic data of the subject.

[0092] Step S120: Acquire monitoring video of the object under test.

[0093] Step S130: input the EEG characteristic data and the monitoring video into the trained epileptiform discharge model, so as to determine whether the subject is in an abnormal EEG discharge state through the epileptiform discharge model.

[0094] Implementation Figure 1 The epileptic discharge determination method shown can input the EEG characteristic data and the monitoring video into the trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model. Compared with the prior art, the implementation of the embodiment of the present application avoids determining whether the subject is in an abnormal EEG discharge state through consultation with an expert doctor, and combines the EEG characteristic data and the monitoring video to improve the accuracy and efficiency of determining whether the subject is in an abnormal EEG discharge state.

[0095] These steps are described in detail below.

[0096] In step S110, the EEG characteristic data of the subject is obtained.

[0097] Specifically, the test subject refers to a patient whose EEG discharge state needs to be determined at this time, for example, a patient suspected of having an epileptic seizure. The EEG characteristic data can reflect the EEG fluctuation of the test subject at this time. It is worth noting that the EEG characteristic data is obtained from the scalp position of the test subject's brain, or can be obtained from the skull of the test subject using a probe, and this exemplary embodiment does not specifically limit this.

[0098] As an optional embodiment, the EEG characteristic data is multi-channel EEG monitoring data collected from multiple parts of the head of the subject, and the EEG characteristic data includes EEG signal waveform characteristics; obtaining the EEG characteristic data of the subject includes: calculating the potential difference between each channel and the reference electrode based on the multi-channel EEG monitoring data to obtain the initial EEG signal characteristic data; generating a corresponding EEG signal waveform graph based on the initial EEG signal characteristic data; and extracting the EEG signal waveform characteristics based on the EEG signal waveform graph.

[0099] Specifically, multi-channel EEG monitoring data refers to EEG monitoring data collected from multiple locations of the subject's brain. For example, detection electrodes are placed at multiple locations on the subject's scalp, and each detection electrode at the scalp location can collect one channel of scalp EEG monitoring data. Furthermore, multi-channel EEG monitoring data can be collected by placing detection electrodes at multiple locations on the scalp. In addition, invasive detection electrodes can also be placed at multiple locations in the subject's skull to collect multi-channel EEG monitoring data.

[0100] The reference electrode is used to measure whether the collected multi-channel EEG monitoring data is in an abnormal discharge state at this time. Specifically, the potential difference between the EEG monitoring data under each channel and the reference electrode is calculated. According to the potential difference, the initial characteristic data of the EEG signal reflecting whether the EEG monitoring data of the channel is in an abnormal discharge at this time can be obtained.

[0101] Since the multi-channel EEG monitoring data specifically includes EEG data, ECG data and EMG data, after obtaining the initial feature data of the EEG signal, it is necessary to extract the EEG feature data that reflects the brain discharge state of the subject at this time from the initial feature data of the EEG signal.

[0102] The EEG characteristic data is specifically displayed as an EEG signal waveform. By performing Fourier transformation on the initial characteristics of the EEG signal, an EEG signal waveform diagram can be obtained. Figure 2 What is shown is the EEG signal waveform diagram generated based on the initial feature data of the EEG signal, such as Figure 2 As shown, each row of waveform corresponds to a channel. The fluctuation of the EEG signal waveform can reflect the EEG signal waveform characteristics of the subject in real time.

[0103] It can be seen that by implementing this optional embodiment, the potential difference between each channel and the reference electrode can be calculated to finally obtain EEG characteristic data that can reflect whether the brain of the subject is in abnormal discharge at this time. The EEG signal waveform corresponding to the initial characteristic data of the EEG signal can be obtained, and the EEG fluctuation of the subject can be visually reflected in real time.

[0104] In step S120, a monitoring video of the object under test is obtained.

[0105] Specifically, the monitoring video refers to the video collected during the monitoring of the subject. For example, a monitoring device may be carried on the head of the subject, and a camera may be included in the device to collect dynamic video of the subject in real time. Figure 3 A head mounted device for collecting monitoring video is shown, such as Figure 3 As shown, a mobile phone is placed at point b and connected to the head-mounted device at point a through a connecting axis c, so that the monitoring video of the object wearing the head-mounted device can be collected through the recording function of the mobile phone.

[0106] As an optional embodiment, after obtaining the monitoring video of the subject, the method also includes: obtaining the human body feature information and motion feature information of the subject from the monitoring video; inputting the EEG signal waveform features, the human body feature information and the motion feature information into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0107] Specifically, human body features refer to features that describe a certain part of the human body of the subject. Action features refer to features that describe the actions performed by the subject. Human body feature information can be feature information of the eyes of the subject, feature information of the nose of the subject, feature information of the hands of the subject, feature information of the neck of the subject, or feature information of the feet of the subject, and this exemplary embodiment does not specifically limit this.

[0108] The motion feature information may be the motion feature information corresponding to the lifting motion when the subject raises his arm, or the motion feature information corresponding to the twitching motion when the subject twitches his leg, which is not specifically limited in this exemplary embodiment. Figure 4 A sensor for determining motion feature information is shown, such as Figure 4 The sensors are distributed at various key parts of the object to be tested to collect motion feature information of different parts.

[0109] On this basis, the previously obtained EEG signal waveform features, human feature information obtained through monitoring video, and motion feature information are input into the trained epileptiform discharge model to determine whether the current subject is in an abnormal discharge state. For example, it is determined that the subject is in an abnormal discharge state at this time.

[0110] It can be seen that by implementing this optional embodiment, the data input into the trained epileptic discharge model includes, in addition to the EEG signal waveform characteristics, human body characteristic information and motion characteristic information, which increases the accuracy of determining whether the subject is in an abnormal EEG discharge state.

[0111] As an optional embodiment, human feature information and motion feature information of the test object are obtained from the monitoring video, including: extracting multiple frame images corresponding to the monitoring video respectively; detecting object key points corresponding to the test object and key point spatial coordinates corresponding to the object key points from the multiple frame images; the object key points include object facial key points and body key points; constructing human feature information based on all key point spatial coordinates of the test object for the target object key points; performing human motion analysis on the human feature information to obtain the motion feature information of the test object.

[0112] Specifically, the monitoring video is actually composed of multiple frames of images. According to the characteristics of the human body, it is possible to detect which part of the multi-frame graphics is the object key point of the object under test, and it is also possible to detect the position of the key point in the multi-frame graphics (i.e. the key point spatial coordinates).

[0113] It is worth noting that the object key points include the object facial key points of the face of the subject and the body key points of the body of the subject. Figure 5 FIG. 1 is a schematic diagram of key points of a target object according to an example implementation of the present application. Figure 5 As shown, each black dot represents a target object key point. For example, target object key point No. 6 represents the target object key point corresponding to the outside of the right eye, and target object key point No. 23 represents the target object key point corresponding to the left hip. Figure 6 A hand joint capture glove for determining hand motion feature information is shown. Figure 6 As shown, each glove carries 6 sensors, which are located at the ends of the five fingers and the center of the palm of the subject to collect the spatial coordinates of the key points of each part.

[0114] The human feature information used to characterize the human features of the subject can be obtained through the key point spatial coordinates corresponding to the key points of the same target object in multiple frame graphics. For example, the human feature information of the left hip can be obtained through the key point spatial coordinates of the key points of the target object corresponding to the left hip, and then the human motion analysis of the human feature information can be performed to determine that the left hip of the subject is shaking at this time (the shaking of the left hip is a motion feature information).

[0115] It can be seen that by implementing this optional embodiment, human feature information reflecting key points of the human object and motion feature information reflecting human movements can be obtained by monitoring the video, and then the human feature information and motion feature information can be input into the trained epileptic discharge model to improve the accuracy of determining whether the subject is in an abnormal EEG discharge state.

[0116] As an optional embodiment, the EEG signal waveform characteristics, human body characteristic information and motion characteristic information are input into a trained epileptic discharge model, including: taking a preset time point as the starting time, and based on a preset time interval, dividing the EEG signal waveform characteristics, human body characteristic information and motion characteristic information respectively to obtain multiple divided segments; and inputting the multiple divided segments into the trained epileptic discharge model.

[0117] Specifically, after obtaining the EEG signal waveform characteristics, human body characteristic information and motion characteristic information, the EEG signal waveform characteristics, human body characteristic information and motion characteristic information are not directly input into the trained epileptic discharge model, but the EEG signal waveform characteristics, human body characteristic information and motion characteristic information are divided and processed.

[0118] Specifically, a preset time point is used as the starting time (for example, the starting time of collecting the EEG characteristic data and monitoring video of the subject), and the EEG signal waveform characteristics, human body characteristic information, and motion characteristic information are divided and processed according to a preset time interval (for example, 2 seconds), so as to obtain a plurality of corresponding divided segments. After obtaining the plurality of divided segments, the plurality of divided segments are input into the trained epileptic discharge model.

[0119] It can be seen that by implementing this optional embodiment and inputting multiple divided segments into the epileptic-like discharge model, the obtained abnormal EEG discharge state corresponds to the EEG signal waveform characteristics of a specific segment, the human body characteristic information of a segment, and the motion characteristic information of a segment, which can better understand the characteristic changes corresponding to the abnormal EEG discharge of the subject.

[0120] In step S130, the EEG characteristic data and the monitoring video are input into the trained epileptiform discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptiform discharge model.

[0121] Specifically, the trained epileptiform discharge model refers to a model used to determine whether the subject is in an abnormal EEG discharge state at this time.

[0122] Based on this, after obtaining the EEG feature data and monitoring video, the EEG feature data and monitoring video are input into the trained epileptic discharge model, which can determine whether the subject is in an abnormal EEG discharge state at this time.

[0123] As an optional embodiment, the epileptic discharge model includes a three-dimensional convolutional neural network, a multi-layer perceptron and a fully connected layer; the EEG feature data and the monitoring video are input into the trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model, including: inputting the EEG signal waveform characteristics into the three-dimensional convolutional neural network to obtain EEG feature information; wherein the EEG feature information includes the spatial position information of each channel; inputting the human body feature information and the action feature information into the multi-layer perceptron to obtain motion feature information, and splicing the EEG feature information and the motion feature information to obtain a comprehensive feature; inputting the comprehensive feature into the fully connected layer to obtain a predicted probability value; wherein the predicted probability value is used to characterize whether the subject is in an abnormal EEG discharge state.

[0124] Specifically, the epileptic discharge model includes a three-dimensional convolutional neural network, a multilayer perceptron, and a fully connected layer. A convolutional neural network is a type of feedforward neural network with a deep structure that includes convolution calculations, and is one of the representative algorithms of deep learning. A three-dimensional convolutional neural network adds a dimension for processing input features on the basis of a two-dimensional convolutional neural network. A multilayer perceptron is a feedforward artificial neural network model, and a fully connected layer is used to map comprehensive features to specific predicted probability values.

[0125] Figure 7 A schematic diagram of a three-dimensional convolutional neural network is shown schematically, as Figure 7 As shown in the figure, the convolution kernel of the three-dimensional convolutional neural network moves in three directions: width, height, and channel. Figure 8 A schematic diagram of an epileptiform discharge model is shown schematically, as Figure 8 As shown in Figure 7 In the three-dimensional convolutional neural network shown, EEG feature information can be obtained, which includes the spatial position information of each channel. According to the spatial position information, the distance relationship between each channel having EEG signal waveform characteristics can be known.

[0126] Inputting human feature information and motion feature information into the multi-layer perceptron can obtain different motion feature information of the subject, for example, it can be known that the subject is waving at this time, and for another example, it can be known that the subject is kicking at this time. After obtaining the EEG feature information and motion feature information, the splicing result obtained by splicing the EEG feature information and the motion feature information is the comprehensive feature, and then the comprehensive feature is input into the fully connected layer to obtain the predicted probability value representing whether the subject is in an abnormal EEG discharge state at this time. Specifically, the predicted probability value can be 0 or 1.

[0127] It can be seen that when implementing this optional embodiment, the epileptic discharge model includes a three-dimensional convolutional neural network, a multi-layer perceptron and a fully connected layer, which allows the convolution kernel to slide in three dimensions, thereby improving the accuracy of the obtained EEG feature information. At the same time, the predicted probability value is obtained based on the EEG feature information and combined with the action feature information, which effectively distinguishes between the abnormal discharge caused by epileptic discharge and the EEG signal similar to the abnormal discharge generated by the subject performing a certain action, thereby avoiding misjudgment of abnormal EEG discharge of the subject.

[0128] As an optional embodiment, a three-dimensional convolutional neural network includes a three-dimensional convolutional layer, a pooling layer and a fully connected layer; the EEG signal waveform features are input into the three-dimensional convolutional neural network to obtain EEG feature information, including: using the three-dimensional convolutional layer to extract the EEG signal waveform features to obtain a feature map; using the pooling layer to compress the feature map to obtain a target feature map; using the fully connected layer to perform dimensionality reduction processing on the target feature map to obtain EEG feature information.

[0129] Specifically, the three-dimensional convolutional neural network specifically includes a three-dimensional convolutional layer, a pooling layer and a fully connected layer. After the EEG signal waveform features are input into the three-dimensional convolutional neural network, the EEG signal waveform features are passed through the three-dimensional convolutional layer to obtain the corresponding feature map. The feature map is used to represent the spatial features and depth features of the EEG signal waveform features extracted by the three-dimensional convolutional layer. Then, after the feature map passes through the pooling layer, a target feature map after feature compression is obtained. The pooling layer can reduce the amount of network calculation and improve the generalization ability of the network. Finally, the target feature map is processed by reducing the dimension through the fully connected layer to obtain EEG feature information.

[0130] Fig. 9 The schematic diagram of the structure of the three-dimensional convolutional neural network is shown schematically. Fig. 9As shown, the three-dimensional convolutional neural network includes a three-dimensional convolutional layer, a pooling layer and a fully connected layer, and the three-dimensional convolutional layer includes convolution layer 910, convolution layer 920, convolution layer 930, convolution layer 940, convolution layer 950, convolution layer 960, convolution layer 970, and convolution layer 980. Specifically, convolution layer 910 is Conv3×3×3, indicating that convolution layer 910 uses a 3×3×3 convolution kernel for convolution operation; convolution layer 920 is 2×F-MBConv1k3×3×3, indicating that convolution layer 920 is a fusion branch convolution layer with two 3×3×3 convolution kernels; convolution layer 930 and convolution layer 940 are 4×F-MBConv4 k3×3×3, indicating that convolution layer 930 and convolution layer 940 are four fusion branch convolution layers with 3×3×3 convolution kernels; convolution layer 950 is 6×MBConv4 k3×3×3, indicating that convolution layer 950 is a multi-branch convolution layer with six 3×3×3 convolution kernels; convolution layer 960 is 9×MBConv6 k3×3×3 indicates that convolution layer 960 is a multi-branch convolution layer with 9 convolution kernels of size 3×3×3; convolution layer 970 is 15×MBConv6 k3×3×3, indicating that convolution layer 970 is a multi-branch convolution layer with 15 convolution kernels of size 3×3×3; convolution layer 980 is Conv1×1×1, indicating that convolution layer 980 uses a convolution kernel of size 1×1×1 for convolution operation.

[0131] It can be seen that by implementing this optional embodiment, the EEG feature information is obtained through a three-dimensional convolutional neural network, which allows the convolution kernel to slide in three dimensions, thereby improving the accuracy of the subsequently obtained EEG feature information.

[0132] As an optional embodiment, the three-dimensional convolution layer includes multiple feature channels; the three-dimensional convolution layer is used to extract the features of the EEG signal waveform to obtain a feature map, including: using multiple channels in the three-dimensional convolution layer to extract the features of the EEG signal waveform respectively, and obtaining channel feature maps corresponding to the multiple channels respectively; determining preset weights set for the multiple channels respectively, and obtaining a feature map based on the preset weights and the channel feature map corresponding to the preset weights.

[0133] Specifically, the three-dimensional convolution layer may have multiple feature channels for extracting features. Multiple feature channels will be calculated to obtain three channel feature maps. Each feature channel has a corresponding preset weight. By calculating the three channel feature maps with the preset weights of the corresponding feature channels, a calculation result can be obtained, that is, the final feature map.

[0134] It can be seen that implementing this optional embodiment and setting a corresponding preset weight for each feature channel can increase the ability of the three-dimensional convolutional neural network to fit training samples and improve the accuracy of subsequent determination of whether the subject is in an abnormal EEG discharge state.

[0135] As an optional embodiment, the method also includes: obtaining training samples, the training samples include multiple patient data corresponding to multiple epilepsy patients; the multiple patient data respectively include EEG feature data, patient monitoring videos and EEG abnormal discharge state labels; the EEG abnormal discharge state labels are used to characterize whether the epilepsy patient is in an EEG abnormal discharge state; inputting the training samples into the initial epilepsy-like discharge model to be trained to obtain the model output result; the model output result is used to characterize whether the epilepsy patient is in an EEG abnormal discharge state; determining the loss value of the initial epilepsy-like discharge model to be trained based on the model output result and the EEG abnormal discharge state label; and iteratively training the initial epilepsy-like discharge model to be trained based on the loss value until a trained epilepsy-like discharge model is obtained.

[0136] Specifically, before inputting EEG feature data and monitoring videos into the trained epileptiform discharge model, the initial epileptiform discharge model needs to be trained.

[0137] The training sample refers to the patient data used to train the initial epileptic discharge model. Specifically, the patient data includes the segmented segments of the patient's EEG feature data, the segmented segments of the patient's monitoring video, and the EEG abnormal discharge state labels corresponding to the segmented segments of the EEG feature data and the segmented segments of the patient's monitoring video. The EEG abnormal discharge state label is used to indicate whether the patient corresponding to the patient data is in an abnormal EEG discharge state.

[0138] The training samples are input into the initial epileptiform discharge model that has not been trained, and the model output results can be obtained. The model output results include output results for indicating whether the epileptic patient is in an abnormal EEG discharge state at this time.

[0139] The model output result is the prediction result obtained by the initial epileptic discharge model based on the training sample. Since the prediction result is not necessarily accurate, it is necessary to determine the loss value of the initial epileptic discharge model based on the model output result and the abnormal EEG discharge state label. Then, the initial epileptic discharge model is iteratively trained based on the loss value (i.e., the difference between the predicted result and the true result) so that the loss value meets the preset convergence condition, thereby obtaining a trained epileptic discharge model.

[0140] Fig.10FIG. 1 is a flow chart of training an initial epileptic discharge model to be trained to obtain a trained epileptic discharge model according to an optional exemplary embodiment of the present application, such as Fig.10 As shown, data 1010 is the divided segments of EEG feature data and the divided segments of patient monitoring videos, model 1020 is the initial epileptic discharge model to be trained, result 1030 is the model output result, data 1040 is the EEG abnormal discharge state label corresponding to the divided segments of EEG feature data and the divided segments of patient monitoring videos, and data 1050 is the loss value determined according to the model output result and the EEG abnormal discharge state label.

[0141] It can be seen that, by implementing this optional embodiment, the trained epileptiform discharge model is obtained by training the initial epileptiform discharge model with multiple training samples, which increases the prediction accuracy of the trained epileptiform discharge model.

[0142] As an optional embodiment, the training samples include positive samples and negative samples, the positive samples include multiple patient data corresponding to multiple epilepsy patients with abnormal EEG discharges, and the negative samples include multiple patient data corresponding to multiple epilepsy patients with normal EEG discharges.

[0143] Specifically, the training samples include positive samples and negative samples. The so-called positive samples refer to patient data corresponding to multiple epilepsy patients in a state of abnormal EEG discharge, that is, positive samples are patient data corresponding to patients in a state of abnormal EEG discharge; negative samples are patient data corresponding to multiple epilepsy patients who are not in a state of abnormal EEG discharge.

[0144] It can be seen that by implementing this optional embodiment, the training samples include positive samples and negative samples. By inputting the positive samples and negative samples into the initial epileptic discharge model, the initial epileptic discharge model can more accurately predict whether the subject is in an abnormal EEG discharge state at this time.

[0145] As an optional embodiment, the training sample is input into the initial epileptic discharge model to be trained to obtain the model output result, including: inputting the training sample into the initial epileptic discharge model to be trained to obtain a prediction label; the prediction label represents whether the epilepsy patient is in an abnormal EEG discharge state; and determining the model output result corresponding to the training sample according to the prediction label.

[0146] Specifically, the prediction label refers to a label used to characterize whether the epilepsy patient determined by the epilepsy-like discharge model is in an abnormal EEG discharge state. For example, if the obtained prediction label is 1, it proves that the model output result corresponding to the training sample shows that the patient corresponding to the training sample is in an abnormal EEG discharge state; if the obtained prediction label is 0, it proves that the model output result corresponding to the training sample shows that the patient corresponding to the training sample is in a normal EEG discharge state.

[0147] It can be seen that by implementing this optional embodiment, the prediction results of the initial epileptic discharge model to be trained for the training samples can be more intuitively known through the prediction labels.

[0148] As an optional embodiment, the initial epileptic discharge model to be trained is iteratively trained according to the loss value until a trained epileptic discharge model is obtained, including: updating the model parameters of the initial epileptic discharge model to be trained according to the loss value; until the loss value meets the preset convergence condition, a trained epileptic discharge model is obtained.

[0149] Specifically, the loss value refers to the difference between the model output result and the abnormal EEG discharge state label, that is, the loss value is the difference between the prediction result of the initial epileptic discharge model and the patient's actual EEG discharge state (corresponding to the training sample). If the loss value does not meet the preset convergence condition (for example, the condition that the loss value is less than the preset loss threshold), the model parameters in the initial epileptic discharge model need to be continuously adjusted until the loss value meets the preset convergence condition, then the model parameters in the initial epileptic discharge model are stopped from being adjusted to obtain a trained epileptic discharge model.

[0150] It can be seen that, by implementing this optional embodiment, the adjustment of the model parameters of the initial epileptic discharge model is stopped only when the loss value meets the preset convergence condition, so as to improve the accuracy of the trained epileptic discharge model.

[0151] Fig.11 FIG. 1 is a flow chart of determining whether the subject is in an abnormal EEG discharge state according to an optional implementation manner of the present application. Fig.11 As shown, data 1110 is the EEG characteristic data of the subject (specifically, the EEG characteristic data under multiple channels). Data 1120 is the monitoring video of the subject. Data 1140 (i.e., the human body characteristic information and motion characteristic information of the subject) can be obtained through the key point detection model of model 1130. Data 1140 and data 1010 are input into the trained epileptic discharge model 1150 (specifically, the epileptic discharge model is a three-dimensional convolutional neural network), and a prediction result 1160 (i.e., predicting whether the subject is in an abnormal EEG discharge state) can be obtained.

[0152] Fig.12 FIG. 1 is a schematic diagram of a scenario for determining whether a subject is in an abnormal EEG discharge state before surgery or during hospitalization according to an exemplary embodiment of the present application. Fig.12 As shown, object 1210 is the subject. The acquisition device 1220 is used to acquire the original EEG monitoring data of the subject 1210. During the acquisition process, electrodes 1221, 1222, and 1223 can be fixed at various positions on the head of the subject 1210, and connected to electrodes 1221, 1222, and 1223 through the interface of the acquisition device 1220 to acquire multi-channel original EEG monitoring data. Electrodes 1221, 1222, and 1223 can be installed in a helmet, a hat, or any head-mounted device, and this exemplary embodiment does not specifically limit this.

[0153] At the same time, the camera 1230 collects monitoring videos of the face and body of the subject. It is worth noting that the acquisition device 1220 can be coupled with the abnormal EEG discharge state detection device 1240. Fig.12 The coupling method is only an exemplary one. The acquisition device 1220 and the abnormal EEG discharge state detection device 1240 can be integrated into the same device, which can be a camera 1230 or a head-mounted device.

[0154] After receiving the EEG characteristic data and the monitoring video, the EEG abnormal discharge state detection device 1240 can directly use the trained epileptiform discharge model to determine whether the subject is in the EEG abnormal discharge state, or can import the EEG characteristic data and the monitoring video into the server 1250. After receiving the EEG characteristic data and the monitoring video, the server 1250 can use the trained epileptiform discharge model to determine whether the subject is in the EEG abnormal discharge state. It is worth noting that the server 1250 can be any form of data processing server such as a cloud server and a distributed server, and this exemplary embodiment does not specifically limit this.

[0155] After determining whether the subject is in an abnormal EEG discharge state, the determined EEG discharge state (i.e., abnormal EEG discharge state or normal EEG discharge state) can be displayed on the display 1260 to facilitate the doctor or the subject to understand the specific EEG discharge situation.

[0156] By implementing the embodiments of the present application, the EEG characteristic data and the monitoring video can be input into the trained epileptiform discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptiform discharge model. Compared with the prior art, by implementing the embodiments of the present application, it is avoided to determine whether the subject is in an abnormal EEG discharge state through consultation with an expert doctor, and the accuracy and efficiency of determining whether the subject is in an abnormal EEG discharge state is improved by combining the EEG characteristic data and the monitoring video.

[0157] Exemplary Media

[0158] After introducing the method according to the exemplary embodiment of the present application, next, the medium according to the exemplary embodiment of the present application will be described.

[0159] In some possible embodiments, various aspects of the present application may also be implemented as a medium on which program code is stored, which, when executed by a processor of a device, is used to implement the steps of the method for determining epileptic discharges according to various exemplary embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0160] Specifically, when the processor of the device executes the program code, it is used to implement the following steps:

[0161] Obtaining EEG characteristic data of the subject; obtaining a monitoring video of the subject; inputting the EEG characteristic data and the monitoring video into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0162] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: calculating the potential difference between each channel and the reference electrode based on the multi-channel EEG monitoring data to obtain initial characteristic data of the EEG signal; generating a corresponding EEG signal waveform graph based on the initial characteristic data of the EEG signal; and extracting EEG signal waveform features based on the EEG signal waveform graph.

[0163] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: obtaining the human feature information and motion feature information of the subject from the monitoring video; inputting the EEG signal waveform characteristics, human feature information and motion feature information into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0164] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: respectively extracting multiple frame images corresponding to the monitoring video; respectively detecting object key points corresponding to the test object and key point spatial coordinates corresponding to the object key points from the multiple frame images; the object key points include object facial key points and body key points; constructing human body feature information based on all key point spatial coordinates of the test object for the target object key points; performing human motion analysis on the human body feature information to obtain motion feature information of the test object.

[0165] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: taking a preset time point as the starting time, based on a preset time interval, dividing the EEG signal waveform characteristics, human body characteristic information, and motion characteristic information respectively to obtain multiple divided segments; and inputting the multiple divided segments into a trained epileptic discharge model.

[0166] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: using a three-dimensional convolution layer to extract features of EEG waveform features, human body feature information, and motion feature information to obtain a feature map; using a pooling layer to compress the feature map to obtain a target feature map; using a fully connected layer to map the target feature map to a label vector, and determining whether the subject is in an abnormal EEG discharge state through the label vector.

[0167] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: using multiple feature channels in the three-dimensional convolutional layer to respectively extract EEG waveform features, human feature information, and motion feature information, and obtain channel feature maps corresponding to the multiple feature channels; determining preset weights set for the multiple feature channels, and obtaining a feature map based on the preset weights and the channel feature map corresponding to the preset weights.

[0168] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: obtaining training samples, the training samples include multiple patient data corresponding to multiple epilepsy patients; the multiple patient data respectively include EEG feature data, patient monitoring videos and EEG abnormal discharge state labels; the EEG abnormal discharge state label is used to characterize whether the epilepsy patient is in an abnormal EEG discharge state; the training samples are input into the initial epilepsy-like discharge model to be trained to obtain the model output result; the model output result is used to characterize whether the epilepsy patient is in an abnormal EEG discharge state; according to the model output result and the EEG abnormal discharge state label, the loss value of the initial epilepsy-like discharge model to be trained is determined; according to the loss value, the initial epilepsy-like discharge model to be trained is iteratively trained until a trained epilepsy-like discharge model is obtained.

[0169] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: the training samples include positive samples and negative samples, the positive samples include multiple patient data corresponding to multiple epilepsy patients with abnormal EEG discharges, and the negative samples include multiple patient data corresponding to multiple epilepsy patients with normal EEG discharges.

[0170] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: inputting the training sample into the initial epileptic-like discharge model to be trained to obtain a prediction label; the prediction label characterizes whether the epilepsy patient is in an abnormal EEG discharge state; and determining the model output result corresponding to the training sample based on the prediction label.

[0171] In some embodiments of the present application, the processor of the device is also used to implement the following steps when executing the program code: updating the model parameters of the initial epileptic discharge model to be trained according to the loss value; until the loss value meets the preset convergence condition, a trained epileptic discharge model is obtained.

[0172] It should be noted that the above-mentioned medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0173] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0174] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0175] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0176] Exemplary Devices

[0177] After introducing the medium of the exemplary embodiment of the present application, next, reference is made to Fig.13 An epileptic discharge determination device according to an exemplary embodiment of the present application is described.

[0178] See also Fig.13 , Fig.13 FIG. 2 is a block diagram of a device for determining epilepsy-like discharge according to an exemplary embodiment of the present application. Fig.13 As shown, an epileptic discharge determination device 1300 according to an exemplary embodiment of the present application includes: an EEG feature data acquisition unit 1310, a monitoring video acquisition unit 1320, and an abnormal discharge determination unit 1330, wherein:

[0179] The EEG characteristic data acquisition unit 1310 is used to acquire the EEG characteristic data of the subject;

[0180] The monitoring video acquisition unit 1320 is used to acquire the monitoring video of the object under test;

[0181] The abnormal discharge determination unit 1330 is used to input the EEG characteristic data and the monitoring video into the trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0182] It can be seen that the implementation Fig.13 The device shown can input EEG characteristic data and monitoring video into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model. Compared with the prior art, the implementation of the embodiment of the present application avoids determining whether the subject is in an abnormal EEG discharge state through consultation with an expert doctor, and combines EEG characteristic data and monitoring video to improve the accuracy and efficiency of determining whether the subject is in an abnormal EEG discharge state.

[0183] In one embodiment, based on the above scheme, the EEG characteristic data acquisition unit 1310, the EEG characteristic data is multi-channel EEG monitoring data collected from multiple parts of the head of the subject, and the EEG characteristic data includes EEG signal waveform characteristics; acquiring the EEG characteristic data of the subject includes:

[0184] The potential difference between each channel and the reference electrode is calculated according to the multi-channel EEG monitoring data to obtain the initial characteristic data of the EEG signal; the corresponding EEG signal waveform diagram is generated according to the initial characteristic data of the EEG signal; and the EEG signal waveform characteristics are extracted according to the EEG signal waveform diagram.

[0185] It can be seen that by implementing this optional embodiment, the potential difference between each channel and the reference electrode can be calculated to finally obtain EEG characteristic data that can reflect whether the brain of the subject is in abnormal discharge at this time. The EEG signal waveform corresponding to the initial characteristic data of the EEG signal can be obtained, and the EEG fluctuation of the subject can be visually reflected in real time.

[0186] In one embodiment, based on the above solution, after the monitoring video acquisition unit 1310 acquires the monitoring video of the object under test, the method further includes:

[0187] The human body characteristic information and motion characteristic information of the subject are obtained from the monitoring video; the EEG signal waveform characteristics, human body characteristic information and motion characteristic information are input into the trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

[0188] It can be seen that by implementing this optional embodiment, the data input into the trained epileptic discharge model includes, in addition to the EEG signal waveform characteristics, human body characteristic information and motion characteristic information, which increases the accuracy of determining whether the subject is in an abnormal EEG discharge state.

[0189] In one embodiment, based on the aforementioned scheme, human feature information and motion feature information of the subject are obtained from the monitoring video, including: extracting multiple frame images corresponding to the monitoring video respectively; detecting object key points corresponding to the subject and key point spatial coordinates corresponding to the object key points from the multiple frame images; the object key points include object facial key points and body key points; constructing human feature information based on all key point spatial coordinates of the subject for the target object key points; performing human motion analysis on the human feature information to obtain the motion feature information of the subject.

[0190] It can be seen that the human feature information and motion feature information of the test object are obtained from the monitoring video, including: extracting multiple frame images corresponding to the monitoring video respectively; detecting the object key points corresponding to the test object and the key point spatial coordinates corresponding to the object key points from the multiple frame images; the object key points include the object facial key points and the body key points; constructing the human feature information according to the spatial coordinates of all key points of the test object for the target object key points; performing human motion analysis on the human feature information to obtain the motion feature information of the test object.

[0191] In one embodiment, based on the aforementioned scheme, the EEG signal waveform characteristics, human body characteristic information and motion characteristic information are input into a trained epileptic discharge model, including: taking a preset time point as the starting time, and based on a preset time interval, dividing the EEG signal waveform characteristics, human body characteristic information and motion characteristic information respectively to obtain multiple divided segments; and inputting the multiple divided segments into the trained epileptic discharge model.

[0192] It can be seen that by implementing this optional embodiment and inputting multiple divided segments into the epileptic-like discharge model, the obtained abnormal EEG discharge state corresponds to the EEG signal waveform characteristics of a specific segment, the human body characteristic information of a segment, and the motion characteristic information of a segment, which can better understand the characteristic changes corresponding to the abnormal EEG discharge of the subject.

[0193] In one embodiment, based on the aforementioned scheme, the epileptic discharge model includes a three-dimensional convolutional neural network, a multi-layer perceptron and a fully connected layer; the abnormal discharge determination unit 1330 inputs the EEG feature data and the monitoring video into the trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model, including: inputting the EEG signal waveform features into the three-dimensional convolutional neural network to obtain EEG feature information; wherein the EEG feature information includes the spatial position information of each channel; inputting the human body feature information and the motion feature information into the multi-layer perceptron to obtain motion feature information, and splicing the EEG feature information and the motion feature information to obtain a comprehensive feature; inputting the comprehensive feature into the fully connected layer to obtain a predicted probability value; wherein the predicted probability value is used to characterize whether the subject is in an abnormal EEG discharge state.

[0194] It can be seen that when implementing this optional embodiment, the epileptic discharge model includes a three-dimensional convolutional neural network, a multi-layer perceptron and a fully connected layer, which allows the convolution kernel to slide in three dimensions, thereby improving the accuracy of the obtained EEG feature information. At the same time, the predicted probability value is obtained based on the EEG feature information and combined with the action feature information, which effectively distinguishes between the abnormal discharge caused by epileptic discharge and the EEG signal similar to the abnormal discharge generated by the subject performing a certain action, thereby avoiding misjudgment of abnormal EEG discharge of the subject.

[0195] In one embodiment, based on the aforementioned scheme, a three-dimensional convolutional neural network includes a three-dimensional convolutional layer, a pooling layer, and a fully connected layer; the EEG signal waveform features are input into the three-dimensional convolutional neural network to obtain EEG feature information, including: using the three-dimensional convolutional layer to extract the EEG signal waveform features to obtain a feature map; using the pooling layer to compress the feature map to obtain a target feature map; using the fully connected layer to perform dimensionality reduction processing on the target feature map to obtain EEG feature information.

[0196] It can be seen that by implementing this optional embodiment, the EEG feature information is obtained through a three-dimensional convolutional neural network, which allows the convolution kernel to slide in three dimensions, thereby improving the accuracy of the subsequently obtained EEG feature information.

[0197] In one embodiment, based on the aforementioned scheme, the three-dimensional convolution layer includes multiple feature channels; the three-dimensional convolution layer is used to extract features of the EEG signal waveform to obtain a feature map, including: using the multiple feature channels in the three-dimensional convolution layer to extract features of the EEG signal waveform respectively, to obtain channel feature maps corresponding to the multiple feature channels respectively; determining preset weights set for the multiple feature channels respectively, and obtaining a feature map based on the preset weights and the channel feature map corresponding to the preset weights.

[0198] It can be seen that implementing this optional embodiment and setting a corresponding preset weight for each feature channel can increase the ability of the three-dimensional convolutional neural network to fit the training set and improve the accuracy of subsequent determination of whether the subject is in an abnormal EEG discharge state.

[0199] In one embodiment, based on the aforementioned scheme, it also includes: obtaining training samples, the training samples include multiple patient data corresponding to multiple epilepsy patients; the multiple patient data respectively include EEG feature data, patient monitoring videos and EEG abnormal discharge state labels; the EEG abnormal discharge state labels are used to characterize whether the epilepsy patient is in an abnormal EEG discharge state; the training samples are input into the initial epilepsy-like discharge model to be trained to obtain the model output result; the model output result is used to characterize whether the epilepsy patient is in an abnormal EEG discharge state; according to the model output result and the EEG abnormal discharge state label, the loss value of the initial epilepsy-like discharge model to be trained is determined; according to the loss value, the initial epilepsy-like discharge model to be trained is iteratively trained until a trained epilepsy-like discharge model is obtained.

[0200] It can be seen that, by implementing this optional embodiment, the trained epileptiform discharge model is obtained by training the initial epileptiform discharge model with multiple training samples, which increases the prediction accuracy of the trained epileptiform discharge model.

[0201] In one embodiment, based on the aforementioned scheme, the training samples include positive samples and negative samples. The positive samples include multiple patient data corresponding to multiple epilepsy patients with abnormal EEG discharges, and the negative samples include multiple patient data corresponding to multiple epilepsy patients without abnormal EEG discharges.

[0202] It can be seen that by implementing this optional embodiment, the training samples include positive samples and negative samples. By inputting the positive samples and negative samples into the initial epileptic discharge model, the initial epileptic discharge model can more accurately predict whether the subject is in an abnormal EEG discharge state at this time.

[0203] In one embodiment, based on the aforementioned scheme, the training samples are input into the initial epileptic discharge model to be trained to obtain the model output results, including: inputting the training samples into the initial epileptic discharge model to be trained to obtain the predicted labels; the predicted labels characterize whether the epileptic patient is in an abnormal EEG discharge state; and determining the model output results corresponding to the training samples according to the predicted labels.

[0204] It can be seen that by implementing this optional embodiment, the prediction results of the initial epileptic discharge model to be trained for the training samples can be more intuitively known through the prediction labels.

[0205] In one embodiment, based on the aforementioned scheme, the initial epileptic discharge model to be trained is iteratively trained according to the loss value until a trained epileptic discharge model is obtained, including: updating the model parameters of the initial epileptic discharge model to be trained according to the loss value; until the loss value satisfies a preset convergence condition, a trained epileptic discharge model is obtained.

[0206] It can be seen that, by implementing this optional embodiment, the adjustment of the model parameters of the initial epileptic discharge model is stopped only when the loss value meets the preset convergence condition, so as to improve the accuracy of the trained epileptic discharge model.

[0207] It should be noted that, although several modules or units of the epileptiform discharge determination device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0208] Exemplary Electronic Devices

[0209] After introducing the method, medium, and apparatus according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0210] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0211] Refer to the following Fig.14 1400 according to this embodiment of the present invention is described. Fig.14 The electronic device 1400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0212] like Fig.14 As shown, the electronic device 1400 is in the form of a general computing device. The components of the electronic device 1400 may include but are not limited to: the at least one processing unit 1410, the at least one storage unit 1420, a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410), and a display unit 1440.

[0213] The storage unit stores program codes, which can be executed by the processing unit 1410, so that the processing unit 1410 executes the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0214] The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1421 and / or a cache memory unit 1422 , and may further include a read-only memory unit (ROM) 1423 .

[0215] The storage unit 1420 may also include a program / utility 1424 having a set (at least one) of program modules 1425, such program modules 1425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the reality of a network environment.

[0216] The bus 1430 may include a data bus, an address bus, and a control bus.

[0217] The electronic device 1400 may also communicate with one or more external devices 1470 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed via an input / output (I / O) interface 1450. In addition, the electronic device 1400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1460. As shown, the network adapter 1460 communicates with other modules of the electronic device 1400 via a bus 1430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0218] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0219] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0220] Although the spirit and principle of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the specific embodiments of the invention, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for determining epileptic discharge, characterized in that: The method comprises: Acquiring EEG characteristic data of the tested subject; Acquiring a monitoring video of the subject; Inputting the EEG characteristic data and the monitoring video into a trained epileptic discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model; wherein the epileptic discharge model includes a three-dimensional convolutional neural network, a multi-layer perceptron, and a fully connected layer; the three-dimensional convolutional neural network includes a three-dimensional convolutional layer, a pooling layer, and a fully connected layer; The step of inputting the EEG characteristic data and the monitoring video into a trained epileptiform discharge model to determine whether the subject is in an abnormal EEG discharge state through the epileptiform discharge model includes: Using the three-dimensional convolutional layer to extract the EEG signal waveform features included in the EEG feature data to obtain a feature graph; Using the pooling layer to perform feature compression on the feature map to obtain a target feature map; Using the fully connected layer to perform dimensionality reduction processing on the target feature map to obtain EEG feature information; Inputting the human feature information and the motion feature information of the subject obtained from the monitoring video into the multi-layer perceptron to obtain motion feature information, and splicing the EEG feature information and the motion feature information to obtain a comprehensive feature; Inputting the comprehensive features into the fully connected layer to obtain a predicted probability value; wherein the predicted probability value is used to characterize whether the subject is in an abnormal EEG discharge state; Wherein, obtaining the human body feature information and motion feature information of the subject from the monitoring video includes: Extracting a plurality of frame images corresponding to the monitoring video respectively; Detecting object key points corresponding to the detected object and key point spatial coordinates corresponding to the object key points from the multiple frame images respectively; the object key points include object facial key points and body key points; Constructing human body feature information according to the spatial coordinates of all key points of the tested object for the key points of the target object; The human body feature information is subjected to human body motion analysis to obtain the motion feature information of the subject.

2. The method according to claim 1, characterized in that: The EEG characteristic data is multi-channel EEG monitoring data collected from multiple parts of the subject's head; The step of obtaining the EEG characteristic data of the subject includes: Calculating the potential difference between each channel and the reference electrode according to the multi-channel EEG monitoring data to obtain initial characteristic data of the EEG signal; Generate a corresponding EEG signal waveform diagram according to the initial EEG signal characteristic data; The EEG signal waveform features are extracted according to the EEG signal waveform diagram.

3. The method according to claim 2, characterized in that After obtaining the monitoring video of the object under test, the method further includes: Obtaining human body feature information and motion feature information of the subject from the monitoring video; The EEG signal waveform characteristics, the human body characteristic information and the motion characteristic information are input into a trained epileptic discharge model, so as to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

4. The method according to claim 3, characterized in that The step of inputting the EEG signal waveform feature, the human body feature information, and the motion feature information into a trained epileptic discharge model includes: Taking a preset time point as a starting time, and based on a preset time interval, dividing the EEG signal waveform feature, the human body feature information, and the motion feature information respectively to obtain a plurality of divided segments; The multiple segmented segments are input into the trained epileptic discharge model.

5. The method according to claim 1, characterized in that The three-dimensional convolutional layer includes multiple feature channels; The step of extracting features from the EEG feature data using the three-dimensional convolutional layer to obtain a feature map includes: Using the multiple feature channels in the three-dimensional convolutional layer to perform feature extraction on the EEG feature data respectively, to obtain channel feature maps corresponding to the multiple feature channels respectively; Determine preset weights set for the multiple feature channels respectively, and obtain a feature map based on the preset weights and a channel feature map corresponding to the preset weights.

6. The method according to claim 1, characterized in that The method further comprises: Acquire training samples, wherein the training samples include a plurality of patient data corresponding to a plurality of epilepsy patients; the plurality of patient data respectively include EEG feature data, patient monitoring videos, and EEG abnormal discharge state labels; the EEG abnormal discharge state labels are used to indicate whether the epilepsy patient is in an EEG abnormal discharge state; Inputting the training sample into the initial epileptic discharge model to be trained to obtain a model output result; the model output result is used to characterize whether the epilepsy patient is in an abnormal EEG discharge state; Determining the loss value of the initial epileptic discharge model to be trained according to the model output result and the abnormal EEG discharge state label; According to the loss value, the initial epileptic discharge model to be trained is iteratively trained until a trained epileptic discharge model is obtained.

7. The method according to claim 6, characterized in that The training samples include positive samples and negative samples. The positive samples include multiple patient data corresponding to multiple epilepsy patients with abnormal EEG discharges, and the negative samples include multiple patient data corresponding to multiple epilepsy patients with normal EEG discharges.

8. The method according to claim 6, characterized in that The step of inputting the training sample into the initial epileptic discharge model to be trained to obtain a model output result includes: Inputting the training sample into the initial epileptic discharge model to be trained to obtain a prediction label; the prediction label indicates whether the epilepsy patient is in an abnormal EEG discharge state; Determine the model output result corresponding to the training sample according to the predicted label.

9. The method according to claim 6, characterized in that The iterative training of the initial epileptic discharge model to be trained according to the loss value until a trained epileptic discharge model is obtained includes: According to the loss value, updating the model parameters of the initial epileptic discharge model to be trained; When the loss value satisfies a preset convergence condition, a trained epileptic discharge model is obtained.

10. An epileptic discharge determination device, characterized in that: The device comprises: An EEG characteristic data acquisition unit is used to acquire EEG characteristic data of a subject; A monitoring video acquisition unit is used to acquire the monitoring video of the object under test; An abnormal discharge determination unit, used for inputting the EEG characteristic data and the monitoring video into a trained epileptic discharge model, so as to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model; wherein the epileptic discharge model includes a three-dimensional convolutional neural network, a multi-layer perceptron and a fully connected layer; the three-dimensional convolutional neural network includes a three-dimensional convolutional layer, a pooling layer and a fully connected layer; The abnormal discharge determination unit is specifically used for: Using the three-dimensional convolutional layer to extract the EEG signal waveform features included in the EEG feature data to obtain a feature graph; Using the pooling layer to perform feature compression on the feature map to obtain a target feature map; Using the fully connected layer to perform dimensionality reduction processing on the target feature map to obtain EEG feature information; Inputting the human feature information and the motion feature information of the subject obtained from the monitoring video into the multi-layer perceptron to obtain motion feature information, and splicing the EEG feature information and the motion feature information to obtain a comprehensive feature; Inputting the comprehensive features into the fully connected layer to obtain a predicted probability value; wherein the predicted probability value is used to characterize whether the subject is in an abnormal EEG discharge state; Wherein, obtaining the human body feature information and motion feature information of the subject from the monitoring video includes: Extracting a plurality of frame images corresponding to the monitoring video respectively; Detecting object key points corresponding to the detected object and key point spatial coordinates corresponding to the object key points from the multiple frame images respectively; the object key points include object facial key points and body key points; Constructing human body feature information according to the spatial coordinates of all key points of the tested object for the key points of the target object; The human body feature information is subjected to human body motion analysis to obtain the motion feature information of the subject.

11. The device according to claim 10, characterized in that The EEG characteristic data is multi-channel EEG monitoring data collected from multiple parts of the subject's head; The step of obtaining the EEG characteristic data of the subject includes: Calculating the potential difference between each channel and the reference electrode according to the multi-channel EEG monitoring data to obtain initial characteristic data of the EEG signal; Generate a corresponding EEG signal waveform diagram according to the initial EEG signal characteristic data; The EEG signal waveform features are extracted according to the EEG signal waveform diagram.

12. The device according to claim 11, characterized in that After acquiring the monitoring video of the object under test, the device further includes: Obtaining human body feature information and motion feature information of the subject from the monitoring video; The EEG signal waveform characteristics, the human body characteristic information and the motion characteristic information are input into a trained epileptic discharge model, so as to determine whether the subject is in an abnormal EEG discharge state through the epileptic discharge model.

13. The device according to claim 12, characterized in that The step of inputting the EEG signal waveform feature, the human body feature information, and the motion feature information into a trained epileptic discharge model includes: Taking a preset time point as a starting time, and based on a preset time interval, dividing the EEG signal waveform feature, the human body feature information, and the motion feature information respectively to obtain a plurality of divided segments; The multiple segmented segments are input into the trained epileptic discharge model.

14. The device according to claim 10, characterized in that The three-dimensional convolutional layer includes multiple feature channels; The step of extracting features from the EEG feature data using the three-dimensional convolutional layer to obtain a feature map includes: Using the multiple channels in the three-dimensional convolutional layer to perform feature extraction on the EEG feature data respectively, to obtain channel feature maps corresponding to the multiple channels respectively; Determine preset weights set for the multiple channels respectively, and obtain a feature map based on the preset weights and a channel feature map corresponding to the preset weights.

15. The device according to claim 10, characterized in that The device also includes: Acquire training samples, wherein the training samples include a plurality of patient data corresponding to a plurality of epilepsy patients; the plurality of patient data respectively include EEG feature data, patient monitoring videos, and EEG abnormal discharge state labels; the EEG abnormal discharge state labels are used to indicate whether the epilepsy patient is in an EEG abnormal discharge state; Inputting the training sample into the initial epileptic discharge model to be trained to obtain a model output result; the model output result is used to characterize whether the epilepsy patient is in an abnormal EEG discharge state; Determining the loss value of the initial epileptic discharge model to be trained according to the model output result and the abnormal EEG discharge state label; According to the loss value, the initial epileptic discharge model to be trained is iteratively trained until a trained epileptic discharge model is obtained.

16. The device according to claim 15, characterized in that The training samples include positive samples and negative samples. The positive samples include multiple patient data corresponding to multiple epilepsy patients with abnormal EEG discharges, and the negative samples include multiple patient data corresponding to multiple epilepsy patients without abnormal EEG discharges.

17. The device according to claim 15, characterized in that The step of inputting the training sample into the initial epileptic discharge model to be trained to obtain a model output result includes: Inputting the training sample into the initial epileptic discharge model to be trained to obtain a prediction label; the prediction label indicates whether the epilepsy patient is in an abnormal EEG discharge state; Determine the model output result corresponding to the training sample according to the predicted label.

18. The device according to claim 15, characterized in that The iterative training of the initial epileptic discharge model to be trained according to the loss value until a trained epileptic discharge model is obtained includes: According to the loss value, updating the model parameters of the initial epileptic discharge model to be trained; When the loss value satisfies a preset convergence condition, a trained epileptic discharge model is obtained.

19. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images