Abnormal discharge detection method and device, electronic equipment, storage medium and product

Abstract

The present disclosure provides an abnormal discharge detection method, device, electronic equipment, storage medium and product, and relates to the technical field of auxiliary medical treatment. The abnormal discharge detection method comprises: preprocessing a received electroencephalogram signal of a measured object to obtain a multi-channel electroencephalogram signal; performing time-frequency domain and space-based feature extraction on the multi-channel electroencephalogram signal to obtain a multi-dimensional electroencephalogram feature vector; extracting a multi-dimensional facial motion feature vector and a multi-dimensional body motion feature vector in a received monitoring video of the measured object respectively; and performing abnormal discharge detection based on the multi-dimensional electroencephalogram feature vector, the multi-dimensional facial motion feature vector and the multi-dimensional body motion feature vector to obtain a detection result. Through the technical scheme of the present disclosure, comprehensive capture of abnormal discharge-related features can be achieved, thereby improving the accuracy and robustness of abnormal discharge detection.

Description

Technical Field

[0001] This disclosure relates to the field of assistive medical technology, and in particular to an abnormal discharge detection method, an abnormal discharge detection device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Epilepsy is a chronic neurological disorder caused by abnormal and excessive discharge of neurons in the brain. It is characterized by recurrent, spontaneous epileptic seizures and is particularly common in children. However, current detection methods require high accuracy in detecting the subject in the monitoring video. If there is a positioning deviation (such as occlusion or low resolution), effective features may be missed or noise may be introduced, resulting in the need to improve the accuracy of the detection results.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide an abnormal discharge detection method, an abnormal discharge detection device, an electronic device, a computer-readable storage medium, and a computer program product that can at least partially improve the problem of limited video required for epilepsy detection in related technologies.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of this disclosure, an abnormal discharge detection method is provided, comprising: preprocessing a received electroencephalogram (EEG) signal of a test subject to obtain a multi-channel EEG signal; performing feature extraction based on time-frequency domain and spatial domain on the multi-channel EEG signal to obtain a multi-dimensional EEG feature vector; extracting a multi-dimensional facial movement feature vector and a multi-dimensional body movement feature vector from a received monitoring video of the test subject, wherein the monitoring video and the EEG signal are time-synchronized; and performing abnormal discharge detection based on the multi-dimensional EEG feature vector, the multi-dimensional facial movement feature vector, and the multi-dimensional body movement feature vector to obtain a detection result.

[0007] In one embodiment of this disclosure, feature extraction based on time-frequency domain and spatial characteristics is performed on the multi-channel EEG signal to obtain a multi-dimensional EEG feature vector. This includes: inputting the multi-channel EEG signal into a two-dimensional convolutional 2D-CNN model to perform sliding operations on the two-dimensional convolutional layers to capture potential change patterns in different frequency ranges based on the sliding of the convolutional kernels on the time axis, thereby obtaining time-frequency domain features; and capturing potential correlations between different channels based on the sliding of the convolutional kernels on the channel axis, thereby obtaining spatial features; and performing hierarchical integration of the time-frequency domain features and the spatial features to obtain the multi-dimensional EEG feature vector.

[0008] In one embodiment of this disclosure, the two-dimensional convolutional 2D-CNN model uses EfficientNet-B0 as the backbone network to perform hierarchical integration of the time-frequency domain features and the spatial features to obtain the multidimensional EEG feature vector. This includes: performing hierarchical abstraction and fusion of the time-frequency domain features and spatial features through multi-layer convolution, residual connections and pooling operations of EfficientNet-B0, and outputting the multidimensional EEG feature vector of a preset dimension by the fully connected layer of EfficientNet-B0.

[0009] In one embodiment of this disclosure, the potential correlation between different channels is captured based on the sliding of the convolution kernel on the channel axis, including: capturing the synchronicity of potential changes between different channels and the potential difference distribution between different channels based on the sliding of the convolution kernel on the channel axis.

[0010] In one embodiment of this disclosure, the multidimensional facial motion feature vector and the multidimensional body motion feature vector are extracted from the received surveillance video of the subject being tested, including: segmenting the surveillance video into an image sequence based on a specified frame rate; inputting the image sequence into a feature detection model to perform facial and body feature detection, so that the feature detection model outputs the multidimensional facial motion feature vector and the multidimensional body motion feature vector.

[0011] In one embodiment of this disclosure, the feature detection model includes a YOLOv5-Patient model. The image sequence is input into the feature detection model for facial and body feature detection, so that the feature detection model outputs the multidimensional facial motion feature vector and the multidimensional body motion feature vector. This includes: inputting the image sequence into the YOLOv5-Patient model, where the YOLOv5-Patient model detects the location of objects in the image and determines the bounding box region of the object in the image, obtaining a cropped image of the object; performing facial and body recognition on the cropped image based on time sequence, obtaining facial video segments and body video segments respectively; extracting the multidimensional facial motion feature vector based on the facial video segments; and extracting the multidimensional body motion feature vector based on the body video segments.

[0012] In one embodiment of this disclosure, the feature detection model further includes a first Sapiens model and a first multilayer perceptron. Extracting the multidimensional facial motion feature vector based on the facial video segment includes: inputting the facial video segment into the first Sapiens model, where the first Sapiens model performs facial keypoint detection to obtain the spatial coordinates of the facial keypoints; performing facial motion analysis based on the temporal changes of the spatial coordinates of the facial keypoints to obtain facial motion feature information; and inputting the facial motion feature information into the first multilayer perceptron, where the first multilayer perceptron performs depth feature extraction to obtain the multidimensional facial motion feature vector.

[0013] In one embodiment of this disclosure, the feature detection model further includes a second Sapiens model and a second multilayer perceptron. Extracting the multidimensional body motion feature vector based on the body video segment includes: inputting the body video segment into the second Sapiens model, where the second Sapiens model performs body keypoint detection to obtain the spatial coordinates of the body keypoints; performing human motion analysis based on the temporal changes of the body keypoint spatial coordinates to obtain body motion feature information; and inputting the body motion feature information into the second multilayer perceptron, where the second multilayer perceptron performs depth feature extraction to obtain the multidimensional body motion feature vector.

[0014] In one embodiment of this disclosure, abnormal discharge detection is performed based on the multidimensional EEG feature vector, the multidimensional facial movement feature vector, and the multidimensional body movement feature vector to obtain a detection result, including: concatenating the multidimensional EEG feature vector, the multidimensional facial movement feature vector, and the multidimensional body movement feature vector to obtain a fusion vector; and performing abnormal discharge detection based on the fusion vector to obtain a detection result.

[0015] In one embodiment of this disclosure, abnormal discharge detection based on the fusion vector to obtain a detection result includes: inputting the fusion vector into an abnormal detection model constructed based on a classifier of a third multilayer perceptron, so as to map the weighted sum of the fusion vector after linear transformation and bias processing into a classification probability, and outputting the detection result based on the classification probability.

[0016] In one embodiment of this disclosure, the method further includes: using a specified duration as a sampling window, performing a sampling operation on historical EEG signals and corresponding historical surveillance videos of the object based on the sampling window without overlap; within one sampling window, constructing positive samples based on the historical EEG signals and the historical surveillance videos of the object located within a specified interval, and constructing negative samples based on the historical EEG signals and the historical surveillance videos of the object located on both sides of the specified interval, wherein the specified interval represents the interval of abnormal discharge occurrence; constructing a labeled fusion vector dataset based on the positive samples and the negative samples respectively; and training a neural network including an initial multilayer perceptron based on the fusion vector dataset to obtain the anomaly detection model.

[0017] In one embodiment of this disclosure, preprocessing the received electroencephalogram (EEG) signal from the subject to obtain a multi-channel EEG signal includes: resampling the EEG signal to a specified frequency; performing a second-order Butterworth filter based on the specified frequency to filter out low-frequency drift and high-frequency noise, thereby obtaining a filtered EEG signal; integrating the filtered EEG signal based on electrode configuration rules to generate a multi-channel average lead EEG signal; and truncating the amplitude of the multi-channel average lead EEG signal to obtain the multi-channel EEG signal.

[0018] According to another aspect of this disclosure, an abnormal discharge detection device is provided, comprising: a preprocessing module for preprocessing received electroencephalogram (EEG) signals from a test subject to obtain multi-channel EEG signals; a first feature extraction module for performing time-frequency and spatial feature extraction on the multi-channel EEG signals to obtain multi-dimensional EEG feature vectors; a second feature extraction module for extracting multi-dimensional facial movement feature vectors and multi-dimensional body movement feature vectors from received monitoring videos of the test subject, wherein the monitoring videos and the EEG signals are time-synchronized; and a detection module for detecting abnormal discharges based on the multi-dimensional EEG feature vectors, the multi-dimensional facial movement feature vectors, and the multi-dimensional body movement feature vectors to obtain detection results.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the abnormal discharge detection method of any one of the above-described methods by executing the executable instructions.

[0020] According to another aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the abnormal discharge detection method described above.

[0021] The abnormal discharge detection scheme provided in this disclosure extracts features including time-frequency domain information and spatial information from electroencephalogram (EEG) signals to generate multidimensional EEG feature vectors that directly reflect the nature of abnormal discharges in brain neurons. By extracting multidimensional facial movement feature vectors and multidimensional body movement feature vectors that characterize facial and body movement details from the received monitoring video of the subject, it is possible to capture abnormal movement features of key parts. Then, by combining the multidimensional EEG feature vectors, multidimensional facial movement feature vectors, and multidimensional body movement feature vectors, abnormal discharge detection is performed and the detection results are output. The extracted movement features of key parts of the face and body help to reduce the omission of effective features or the introduction of noise caused by overall positioning deviation. Moreover, when one side of the face or body is occluded, the other side can still provide key information. Combining multiple EEG feature vectors helps to achieve comprehensive capture of abnormal discharge-related features, thereby improving the accuracy and robustness of abnormal discharge detection.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 This diagram illustrates the structure of an abnormal discharge detection system according to an embodiment of the present disclosure. Figure 2 This diagram shows a flowchart of an abnormal discharge detection method according to an embodiment of the present disclosure; Figure 3 A schematic diagram of another abnormal discharge detection method in an embodiment of this disclosure is shown; Figure 4 This diagram shows a flowchart of yet another abnormal discharge detection method in an embodiment of the present disclosure; Figure 5 This diagram shows a flowchart of yet another abnormal discharge detection method according to an embodiment of the present disclosure; Figure 6 This diagram shows a flowchart of yet another abnormal discharge detection method according to an embodiment of the present disclosure; Figure 7 This diagram illustrates a flowchart of an abnormal discharge model training method according to an embodiment of the present disclosure. Figure 8 This diagram illustrates another abnormal discharge detection system structure in an embodiment of the present disclosure; Figure 9 This diagram illustrates the structure of another abnormal discharge detection system in an embodiment of the present disclosure. Figure 10 This diagram illustrates an abnormal discharge detection device according to an embodiment of the present disclosure; Figure 11 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] Epilepsy is a chronic neurological disorder caused by abnormal and excessive electrical discharges in the brain. It is characterized by recurrent, spontaneous seizures and is particularly common in children, being one of the most prevalent and serious neurological disorders in childhood. Childhood epilepsy not only severely impacts the quality of life, cognitive development, mental health, and education of affected children, but also places a heavy burden on families and society. Early, accurate, and timely detection and diagnosis of epilepsy are crucial for initiating effective treatment, improving prognosis, and mitigating the impact of the disease.

[0028] In some embodiments, the proposed intelligent auxiliary analysis method for pediatric motor epilepsy based on synchronous video and EEG involves simultaneously acquiring EEG signals and video data, extracting segments containing seizure-related information, filtering and preprocessing the EEG signals to remove interference noise, standardizing the video data format, extracting key features characterizing the frequency and temporal domains from the preprocessed EEG signals, locating the pediatric patient region from the video data using a YOLO neural network with a specific backbone and feature pyramid structure, focusing only on spatiotemporal points of interest within that region, constructing feature analysis units to extract video features reflecting movement dynamics and texture, quantifying the video features using a bag-of-words model, concatenating and fusing the EEG and video features, adjusting the positive and negative sample ratio using a sample balancing method to avoid model learning bias, and finally training the machine learning model through cross-validation to allow the model to simultaneously learn the association patterns between EEG physiological abnormalities and somatic movement abnormalities. However, the above scheme has the following technical drawbacks: Since the accuracy of YOLO bounding boxes directly affects the selection of spatiotemporal points of interest, if the patient's location in the video is off (such as occlusion or low resolution), it may lead to the omission of effective features or the introduction of noise. In addition, the video processing in the above scheme involves multi-layer neural networks (Darknet-53, FPN), spatiotemporal point of interest detection and multi-feature fusion, which has high requirements for hardware computing resources such as GPU and real-time processing capabilities, which may limit clinical deployment.

[0029] The solution provided in this application uses a deep learning model to extract the original waveform data as the original data features, and then fuses the frequency domain features and video features to maximize the use of the patient's pathological information. The above model is applied to the detection of a new type of infantile epileptic spasm syndrome. Users can learn about the results of abnormal brain discharges without leaving home. When encountering suspected epilepsy or other problems, the obtained results of abnormal brain discharges can be sent to the doctor, realizing remote diagnosis at the doctor's end.

[0030] Figure 1 This is a schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application. The system includes: a plurality of terminals 120 and a server cluster 140.

[0031] Terminal 120 can be a mobile terminal such as a mobile phone, game console, tablet computer, e-book reader, smart glasses, MP4 (Moving Picture Experts Group Audio Layer IV) player, smart home device, AR (Augmented Reality) device, VR (Virtual Reality) device, etc. Alternatively, terminal 120 can also be a personal computer (PC), such as a laptop computer and a desktop computer, etc.

[0032] The terminal 120 may contain an interactive application for providing based abnormal discharge detection.

[0033] Terminal 120 is connected to server cluster 140 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0034] Server cluster 140 may be a single server, a combination of several servers, a virtualization platform, or a cloud computing service center. Server cluster 140 provides background services for interactive applications that provide based abnormal discharge detection. Optionally, server cluster 140 undertakes the primary computing task, and terminal 120 undertakes secondary computing tasks; or, server cluster 140 undertakes secondary computing tasks, and terminal 120 undertakes the primary computing tasks; or, terminal 120 and server cluster 140 collaborate on computing using a distributed computing architecture.

[0035] In some alternative embodiments, server cluster 140 is used to store interactive program information based on abnormal discharge detection.

[0036] Optionally, the application clients installed on different terminals 120 may be the same, or the application clients installed on two terminals 120 may be clients of the same type of application on different control system platforms. Depending on the terminal platform, the specific form of the application client may also differ; for example, the application client may be a mobile client, a PC client, or a World Wide Web (WWW) client.

[0037] Those skilled in the art will understand that the number of terminals 120 described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0038] Optionally, the system may also include a management device ( Figure 1(Not shown), the management device is connected to the server cluster 140 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0039] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0040] The abnormal discharge detection method in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0041] like Figure 2 As shown, an abnormal discharge detection method according to an embodiment of this disclosure includes: Step S202: Preprocess the received EEG signal from the subject to obtain multi-channel EEG signal.

[0042] In some embodiments, electroencephalogram (EEG) signals refer to the weak electrical signals generated by the activity of brain neuron populations by placing electrodes on the scalp surface of the subject, which can directly reflect the electrophysiological state of the brain.

[0043] In some embodiments, the raw EEG signal is preprocessed to obtain a set of EEG signals including multiple electrode channels. For example, 19 electrodes are configured based on the international 10-20 standard, with each electrode corresponding to a specific region of the brain (such as the frontal, temporal, or occipital regions). The multi-channel EEG signals are presented in a two-dimensional matrix form of time and channels, which not only preserves the dynamic changes of the EEG signals over time but also includes the distribution of electrical activity in different brain regions.

[0044] Step S204: Perform feature extraction based on time-frequency domain and space on the multi-channel EEG signal to obtain a multi-dimensional EEG feature vector.

[0045] In some embodiments, abnormal discharges often manifest as rhythmic abnormalities at specific frequencies. For example, spikes and sharp waves during epileptic seizures are mostly concentrated in the high-frequency range of 10-25Hz, while slow wave abnormalities are mostly concentrated in the low-frequency range of 0.5-4Hz. These frequency abnormalities change dynamically over time. In addition, abnormal discharges usually originate from specific brain regions, such as temporal lobe epilepsy originating from the temporal region, and may spread to other brain regions. The channel dimensions of multichannel EEG signals correspond to different brain regions. By extracting spatial features, the brain region origin and spread patterns of abnormal discharges can be captured.

[0046] Step S206: Extract the multidimensional facial movement feature vector and multidimensional body movement feature vector from the received monitoring video of the subject, and synchronize the monitoring video and EEG signal in time.

[0047] In some embodiments, abnormal discharges during epileptic seizures not only lead to abnormal brain electrophysiology, but are also often accompanied by abnormal facial and body movements in the subject. These abnormal movements are external manifestations of abnormal discharges, which can cover facial and body abnormalities and help improve the reliability of abnormal discharge detection.

[0048] In some embodiments, the facial and body motion feature vectors extracted separately can be used to trace the contribution of the two types of motion features to the detection results in subsequent detection processes. If a detection deviation occurs, it can more specifically pinpoint whether there is room for optimization in the extraction process of facial motion features or body motion features.

[0049] In some embodiments, the multidimensionality of the multidimensional facial motion feature vector may include key point spatial location dimension, motion dynamic change dimension, expression shape dimension, and motion correlation dimension, etc., and the multidimensional body motion feature vector may include body key point spatial dimension, body key point spatial dimension, motion temporal dimension, and motion shape dimension, etc.

[0050] In some embodiments, during the monitoring of the subject, the monitoring video recording the subject's state and the electroencephalogram (EEG) signal acquiring the subject's brain electrical activity are aligned in the time dimension to ensure that the multidimensional facial movement feature vector and multidimensional body movement feature vector extracted from the video can be correlated with the EEG signal features at the corresponding time points.

[0051] Step S208: Abnormal discharge detection is performed based on multidimensional EEG feature vectors, multidimensional facial movement feature vectors, and multidimensional body movement feature vectors to obtain detection results.

[0052] In some embodiments, EEG signals are a direct reflection of the activity of brain neurons, and facial and body movement features are intuitive external manifestations of abnormal discharges. Through cross-verification of physiological essence and behavioral performance, the accuracy of abnormal discharge detection results can be guaranteed.

[0053] In this embodiment, by extracting features including time-frequency and spatial information from the electroencephalogram (EEG) signal, a multidimensional EEG feature vector that directly reflects the nature of abnormal neuronal discharges in the brain is generated. By extracting multidimensional facial movement feature vectors and multidimensional body movement feature vectors that characterize facial and body movement details from the received monitoring video of the subject, abnormal movement features of key parts can be captured. Then, by combining the multidimensional EEG feature vectors, multidimensional facial movement feature vectors, and multidimensional body movement feature vectors, abnormal discharge detection is performed and the detection results are output. The extracted movement features of key parts of the face and body help reduce the omission of effective features or the introduction of noise caused by overall positioning deviation. Moreover, when one side of the face or body is occluded, the other side can still provide key information. Combining multiple EEG feature vectors helps to achieve comprehensive capture of abnormal discharge-related features, thereby improving the accuracy and robustness of abnormal discharge detection.

[0054] In addition, video features are decomposed into facial motion feature vectors and body motion feature vectors, and core motion information is extracted only based on the temporal changes of key point spatial coordinates. This prevents redundant data calculations caused by parallel processing of multiple visual features such as HOG and LBP, thereby reducing the requirements for hardware computing power.

[0055] In one embodiment of this disclosure, feature extraction based on the time-frequency domain and space is performed on multi-channel EEG signals to obtain a multi-dimensional EEG feature vector, including: Multi-channel EEG signals are input into a two-dimensional convolutional 2D-CNN model to perform sliding operations on two-dimensional convolutional layers. This captures potential change patterns in different frequency ranges based on the sliding of the convolutional kernel on the time axis, resulting in time-frequency domain features. Additionally, it captures potential correlations between different channels based on the sliding of the convolutional kernel on the channel axis, resulting in spatial features. The time-frequency domain features and spatial features are then hierarchically integrated to obtain a multi-dimensional EEG feature vector.

[0056] In some embodiments, the multichannel EEG signal is a two-dimensional matrix of time and channel, where each matrix element represents the potential value of a certain channel at a certain moment.

[0057] In some embodiments, the sliding of the convolution kernel on the time axis (e.g., different sizes of convolution kernels correspond to different time windows) can locally aggregate potential changes in a continuous time series. Small-sized convolution kernels can capture short-term high-frequency potential fluctuations, such as waveforms during an epileptic seizure, while large-sized convolution kernels can capture long-term low-frequency potential trends, such as slow waves after a seizure. By sliding in a multi-scale manner, it is possible to extract time-frequency correlation patterns.

[0058] In some embodiments, by sliding the convolution kernel on the channel axis, the potential coordination relationship between different channels can be calculated, which can be used to identify spatial propagation characteristics during abnormal discharge, etc.

[0059] In this embodiment, the initial extracted time-frequency domain features and spatial features are obtained through 2D-CNN. The initial extracted time-frequency domain features and spatial features are further abstracted hierarchically. For example, the shallow network retains the original potential change details, while the deep network is fused to form more abstract high-order features. Then, the multi-level features are compressed into a multi-dimensional EEG feature vector that includes comprehensive information from the time-frequency domain and space, thereby achieving efficient and accurate multi-dimensional feature capture of EEG signals.

[0060] In one embodiment of this disclosure, the two-dimensional convolutional 2D-CNN model uses EfficientNet-B0 as the backbone network to hierarchically integrate time-frequency domain features and spatial features to obtain a multidimensional EEG feature vector. This includes: hierarchically abstracting and fusing time-frequency domain features and spatial features through multi-layer convolution, residual connections and pooling operations of EfficientNet-B0, and outputting a multidimensional EEG feature vector of a preset dimension by the fully connected layer of EfficientNet-B0.

[0061] In some embodiments, EfficientNet-B0 includes a Stem module, an MBConv module, and a Head module. The Stem module is used to increase the number of channels by passing a 3×3 convolution + BN + Swish activation on the input EEG time-channel matrix, thus completing the initial feature mapping. The MBConv module consists of 16 stacked modules (divided into 6 blocks, each containing a different number of MBConv modules), which are responsible for the main feature extraction and abstraction. The Head module includes a 1×1 convolution (to compress the number of channels), global average pooling (GAP), and a fully connected layer, ultimately outputting a 1000-dimensional vector.

[0062] In this embodiment, based on the depthwise separable convolution, residual connections, and pooling operations of the EfficientNet-B0 backbone network, the time-frequency domain features and spatial features of multi-channel EEG signals are progressively abstracted (from low-order local features to high-order global features) and fused across dimensions. The fused features are compressed into vectors of a preset dimension through fully connected layers. This can improve computational efficiency while ensuring the integrity of feature extraction, so that the output multi-dimensional EEG feature vector can more accurately represent EEG patterns related to abnormal discharges. The EEG feature extraction uses EfficientNet-B0 as the backbone network. Through efficient residual connections and pooling operations, this network can achieve feature abstraction with lower computational cost compared to MFCC+LPCC feature extraction in the process of hierarchically integrating time-frequency domain and spatial features, further reducing hardware resource consumption.

[0063] In one embodiment of this disclosure, the potential correlation between different channels is captured based on the sliding of the convolution kernel on the channel axis, including: capturing the synchronicity of potential changes between different channels and the potential difference distribution between different channels based on the sliding of the convolution kernel on the channel axis.

[0064] In this embodiment, based on the sliding of the convolutional kernel on the channel axis of the multi-channel EEG signal, different combinations of electrode channels correspond to different regions of the brain. By calculating the temporal synchronicity of potential changes between channels, such as the degree of coordination of potential fluctuations in multiple brain regions during abnormal discharge, and the spatial potential difference distribution, such as the difference in potential intensity between the lesion area and the surrounding normal area, the abstract inter-brain region correlation can be transformed into quantifiable features. This enables the capture of spatial features during the propagation of abnormal discharge in the brain region, thereby improving the ability to distinguish between normal EEG and abnormal discharge.

[0065] In one embodiment of this disclosure, multidimensional facial motion feature vectors and multidimensional body motion feature vectors are extracted from the received surveillance video of the subject, including: The surveillance video is segmented into image sequences based on a specified frame rate; the image sequences are input into a feature detection model to perform facial and body feature detection, so that the feature detection model outputs multidimensional facial motion feature vectors and multidimensional body motion feature vectors.

[0066] In this embodiment, by segmenting the surveillance video into a continuous image sequence at a specified frame rate, the dynamic video is transformed into a temporal set of static images. The image sequence is processed based on a feature detection model to selectively separate and extract facial and body movement features. First, the facial and body regions of the subject are located. Then, based on the temporal changes in each region, such as facial expression dynamics and body posture changes, key features are extracted, and multidimensional feature vectors that can respectively characterize facial and body movement patterns are output. The entire process is time-synchronized with the EEG signal to ensure the correspondence between movement features and EEG activity in the time dimension. The separation and extraction of facial and body movement features through the feature detection model not only preserves the specific information of movements of different body parts but also prevents information dilution caused by mixed features, thereby improving the accuracy of multimodal fusion detection.

[0067] like Figure 3 As shown, in one embodiment of this disclosure, the feature detection model includes a YOLOv5-Patient model. Image sequences are input into the feature detection model for facial and body feature detection, so that the feature detection model outputs multi-dimensional facial motion feature vectors and multi-dimensional body motion feature vectors, including: Step S302: Input the image sequence into the YOLOv5-Patient model. The YOLOv5-Patient model detects the location of objects in the image and determines the bounding box region of the object in the image, thus obtaining a cropped image of the object.

[0068] In some embodiments, the YOLOv5-Patient model is a target detection model based on the YOLOv5 architecture optimized for the detection task of the test subject (such as an epileptic patient).

[0069] In some embodiments, in the YOLOv5-Patient model, image features are extracted through the CSPDarknet53 backbone network, multi-scale feature fusion is performed through the PANet feature pyramid network, and the location of the test object in the image is located using bounding box regression and classification head. The output includes the bounding box coordinates of the object, realizing fast and accurate detection of the test object, and maintaining high localization robustness even in complex scenes such as occlusion or low resolution.

[0070] Step S304: Perform face and body recognition on the cropped image based on time sequence to obtain face video segments and body video segments respectively.

[0071] In some embodiments, based on the temporal correlation of continuously cropped images, a reference face and reference body region can be initially identified in the first frame or key frame by using morphological features, such as facial contours and body limb structures. Combining the continuity of movement of the same part in adjacent frames, the face and body regions in subsequent frames are dynamically tracked and corrected through feature matching, such as key point position changes and contour overlap. The face and body regions identified in consecutive frames are then integrated into face video segments and body video segments, respectively.

[0072] Extracting multidimensional facial motion feature vectors from facial video clips.

[0073] Extracting multidimensional body motion feature vectors from body video clips.

[0074] In this embodiment, the image sequence is input into the YOLOv5-Patient model to detect object bounding boxes and obtain cropped images. Based on temporal correlation, facial and body regions are identified from the cropped images to form corresponding video segments. Multidimensional motion feature vectors are then extracted from the two segments respectively. The use of the YOLOv5-Patient model can improve the accuracy and robustness of object localization in complex scenes. Combining temporal information to recognize facial and body regions provides targeted data for subsequent feature extraction, which is conducive to improving the representation ability of multidimensional motion feature vectors, thereby further improving the reliability of abnormal discharge detection.

[0075] Furthermore, the proposed solution employs the YOLOv5-Patient model for object localization and cropping. Compared to Darknet-53, it offers superior parameter and computational efficiency while maintaining detection accuracy. It also directly outputs cropped images of the target region, eliminating redundant computational steps such as spatiotemporal interest point selection, cube construction, and multi-feature extraction. Subsequent facial and body motion feature extraction relies solely on the Sapiens model (focused on keypoint detection) and the first multilayer perceptron (focused on deep feature extraction), eliminating the need for quantization processes involving multiple low-order visual features and bag-of-words models. This significantly simplifies the computational chain, reduces the high demand for GPU computing power, and is more suitable for the low-to-medium configuration hardware commonly found in clinical settings.

[0076] In one embodiment of this disclosure, the feature detection model further includes a first Sapiens model and a first multilayer perceptron, extracting multidimensional facial action feature vectors based on facial video segments, including: Step S306: Input the facial video clip into the first Sapiens model, and the first Sapiens model performs facial key point detection to obtain the spatial coordinates of the facial key points.

[0077] In some embodiments, the first Sapiens model is a model for human keypoint detection, capable of locating key anatomical positions of the face and torso. It extracts features from input images or video clips through a deep neural network architecture, combines prior knowledge of human structure, such as the relative positional relationships of facial features and the connection logic of torso joints, and uses methods such as heatmap prediction and coordinate regression to identify and output the two-dimensional or three-dimensional spatial coordinates of key points of the face or torso, such as shoulders, elbows, hips, and knees, in the image. This allows the first Sapiens model to still locate key points even when faced with the relative positional shift of key points due to pose changes or the loss of local information due to partial occlusion, thereby providing a spatial positional basis for motion analysis.

[0078] Step S308: Perform facial motion analysis based on the temporal changes of the spatial coordinates of facial key points to obtain facial motion feature information.

[0079] Step S310: The facial motion feature information is input into the first multilayer perceptron, and the first multilayer perceptron performs depth feature extraction to obtain a multidimensional facial motion feature vector.

[0080] In some embodiments, the first multilayer perceptron (MLP) is a neural network consisting of multiple fully connected layers, wherein the input layer receives facial or body motion feature information, the intermediate layers perform layer-by-layer transformation and abstraction of features through non-linear activation functions such as ReLU, and the output layer generates a vector of fixed dimensions.

[0081] In some embodiments, deep feature extraction refers to mapping the original action feature information, such as the coordinate changes and speed of key points, into higher-order features with higher discriminative power through multi-layer nonlinear transformation. Higher-order features can more effectively capture action patterns, thereby improving the accuracy of subsequent detection tasks.

[0082] In this embodiment, the spatial coordinates of key facial points are detected by the Sapiens model, and the changes in coordinates over time, such as displacement, velocity, and acceleration, are analyzed to obtain basic feature information of facial movements. The basic feature information is then processed by a first multilayer perceptron to extract multidimensional feature vectors that can characterize the essential patterns of facial movements. This allows for the efficient extraction of fine-grained and highly discriminative movement features from facial videos to capture subtle facial movements related to abnormal discharges, such as eyelid twitching and mouth drooping, providing a reliable basis for the facial movement dimension for abnormal discharge detection.

[0083] In one embodiment of this disclosure, the feature detection model further includes a second Sapiens model and a second multilayer perceptron, which extracts multidimensional body motion feature vectors based on body video segments, including: Step S312: Input the body video clip into the second Sapiens model, and the second Sapiens model performs body key point detection to obtain the spatial coordinates of the body key points.

[0084] Step S314: Perform human motion analysis based on the temporal changes of the spatial coordinates of key body points to obtain body motion feature information.

[0085] Step S316: Input the body motion feature information into the second multilayer perceptron, and the second multilayer perceptron performs deep feature extraction to obtain a multidimensional body motion feature vector.

[0086] In this embodiment, a body video clip is input into a second Sapiens model to obtain the spatial coordinates of key body points. Based on the temporal changes in the coordinates, the dynamic characteristics of body movements, such as joint angle changes and limb movement trajectories, are analyzed to obtain body movement feature information. This information is then extracted by a second multilayer perceptron and transformed into a multidimensional feature vector that reflects the core pattern of body movements. Through key point detection and depth feature extraction, abnormal body movements during abnormal discharges, such as limb rigidity and convulsions, can be captured, generating a multidimensional body movement feature vector. This multidimensional body movement feature vector can be combined with EEG features and facial movement features for multidimensional cross-validation. This reduces the risk of misjudgment caused by noise interference from a single feature mode while enhancing the robustness of abnormal discharge identification.

[0087] like Figure 4 As shown in the figure, the points represent key facial and torso points used for abnormal discharge detection. These key points were detected using the first Sapiens model and the second Sapiens model.

[0088] In the facial region (points within the head ellipse in the image), the first Sapiens model detects key points corresponding to the eyes, nose, and mouth, such as those labeled 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. By tracking the changes in the spatial coordinates of these facial key points over time, facial movement features can be analyzed, such as eyelid twitching and mouth drooping. This information is then input into the first multilayer perceptron for depth feature extraction, resulting in a multidimensional facial movement feature vector, which is used to determine the facial movement dimension in abnormal discharge detection.

[0089] In the body region (points below the head in the figure, including points on the upper limbs, trunk, and lower limbs, such as those labeled 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, etc.), the second Sapiens model detects key points in various parts of the body (such as shoulders, elbows, wrists, hips, knees, ankles, etc.). Based on the temporal changes in the spatial coordinates of these key body points, human movements can be analyzed, such as limb rigidity and involuntary twitching. This information is then input into the second multilayer perceptron for deep feature extraction, resulting in a multidimensional body movement feature vector. This provides a basis for the body movement dimension for abnormal discharge detection. The aforementioned key points on the face and body together constitute the basic elements for analyzing the movements of the tested object and assisting in the detection of abnormal discharge.

[0090] In one embodiment of this disclosure, abnormal discharge detection is performed based on multidimensional EEG feature vectors, multidimensional facial movement feature vectors, and multidimensional body movement feature vectors to obtain detection results, including: A fusion vector is obtained by concatenating multidimensional EEG feature vectors, multidimensional facial movement feature vectors, and multidimensional body movement feature vectors; abnormal discharges are detected based on the fusion vector to obtain the detection results.

[0091] In some embodiments, the EEG feature vector is 1024-dimensional, the facial action feature vector is 1024-dimensional, and the body action feature vector is 1024-dimensional. The three are concatenated to obtain a fusion vector with 1024+1024=2048 dimensions. This concatenation method allows the fusion vector to retain the original information of the three types of features, so as to simultaneously include the EEG physiological features of abnormal discharges and facial and body action behavioral features, providing a multimodal collaborative input basis for subsequent detection models.

[0092] In this embodiment, EEG features reflecting the intrinsic physiological nature of abnormal discharges and facial and body movement features reflecting external behavior are aggregated into a unified fusion vector by vector concatenation. This enables the detection model to learn the correlation patterns between the three types of features simultaneously, allowing the anomaly detection model to identify abnormal discharge patterns from the perspective of physiological-facial features-behavior linkage. This not only preserves the value of each modality feature, but also enhances the anomaly detection model's understanding and adaptability to complex scenarios through complementary correlations between features.

[0093] like Figure 5 As shown, the raw EEG monitoring data is first preprocessed to obtain the initial EEG signal feature data. The initial EEG signal feature data is used to calculate the potential difference in one process and to perform time-frequency conversion of the EEG signal in another process to obtain the time-frequency features of the EEG signal.

[0094] Meanwhile, the monitoring video is divided into facial video and body video of the subject being tested. The facial video of the subject being tested first obtains the spatial coordinates of key points, and then performs human motion analysis to obtain facial motion feature information. The body video of the subject being tested first obtains the spatial coordinates of key points, and then performs human motion analysis to obtain body motion feature information.

[0095] Potential difference, EEG signal time-frequency features, facial movement features, and body movement features are fused into fusion features. The fusion features are then processed by the attention layer to obtain embedded features, which are used to detect abnormal discharges.

[0096] In one embodiment of this disclosure, abnormal discharge detection based on a fusion vector is performed to obtain a detection result, including: inputting the fusion vector into an abnormal detection model constructed based on a classifier of a third multilayer perceptron, mapping the weighted sum of the fusion vector after linear transformation and bias processing to a classification probability, and outputting the detection result based on the classification probability.

[0097] In some embodiments, the third multilayer perceptron can be a deep learning model consisting of an input layer, multiple hidden layers, and an output layer. Each layer is connected in a fully connected manner. The hidden layers use non-linear activation functions to enhance feature learning capabilities, and the output layer uses a sigmoid function to map the output to the interval between 0 and 1 to represent the classification probability.

[0098] In this embodiment, a 2048-dimensional fusion vector is input into the input layer of a third multilayer perceptron. The features are then subjected to layer-by-layer linear transformation and bias processing through each hidden layer. High-order correlation features, such as the synchronization pattern between abnormal EEG and facial or body movement abnormalities, are extracted using a nonlinear activation function. The output layer maps the processed weighted sum to the classification probability of abnormal discharges using a sigmoid function. Based on a preset threshold, such as a probability ≥ 0.5, an abnormality is determined, and the detection result is output. This achieves a comprehensive classification judgment of multimodal fusion features. Since the EEG signals and monitoring videos of the tested subject can be measured and obtained by non-professionals, the detection of abnormal discharges is automatically performed based on the EEG signals and monitoring videos. This allows users to understand the results of abnormal brain discharges without leaving home. When encountering suspected epilepsy or other problems, the obtained results of abnormal brain discharges can be sent to doctors to facilitate diagnosis.

[0099] In one embodiment of this disclosure, the method further includes: using a specified duration as a sampling window, performing a sampling operation on historical EEG signals and corresponding historical surveillance videos of the object based on a non-overlapping sliding sampling window; within a sampling window, constructing positive samples based on historical EEG signals and historical surveillance videos of the object located within a specified interval, and constructing negative samples based on historical EEG signals and historical surveillance videos of the object located on both sides of the specified interval, wherein the specified interval represents the interval of abnormal discharge occurrence; constructing a labeled fusion vector dataset based on the positive and negative samples respectively; and training a neural network including an initial multilayer perceptron based on the fusion vector dataset to obtain an anomaly detection model.

[0100] In some embodiments, all video EEG data can be segmented into 4-second segments for training and testing. A non-overlapping sliding window strategy is used for sampling. In order to reduce the probability of label ambiguity caused by the epileptic region being located at the segment boundary, segments with epileptic time in the range of 0.5-3.5 seconds are labeled as positive samples, and the remaining segments are labeled as negative samples.

[0101] In this embodiment, by sampling through a non-overlapping sliding window of a specified duration, continuous historical data can be divided into sample units with temporal integrity. Each sample includes sufficient abnormal discharge-related features. Positive samples are constructed from data within the abnormal discharge occurrence interval (specified interval), and negative samples are constructed from normal data on both sides of the interval. This approach can minimize label ambiguity caused by incomplete features when the occurrence area is located at the beginning or end of a segment, making positive samples more accurately correspond to complete occurrence features. The training samples generated in this way have strong resistance to boundary interference, and the category labels in the constructed fusion vector dataset are more reliable. The anomaly detection model trained based on this can more accurately learn the complete multimodal feature patterns of abnormal discharge, reduce the bias in model learning caused by label ambiguity, and ensure the reliability of the trained model.

[0102] In one embodiment of this disclosure, the received electroencephalogram (EEG) signal from the subject is preprocessed to obtain a multi-channel EEG signal, including: resampling the EEG signal to a specified frequency; performing a second-order Butterworth filter based on the specified frequency to filter out low-frequency drift and high-frequency noise, thereby obtaining a filtered EEG signal; integrating the filtered EEG signal based on electrode configuration rules to generate a multi-channel average lead EEG signal; and performing amplitude truncation processing on the multi-channel average lead EEG signal to obtain a multi-channel EEG signal.

[0103] In this embodiment, resampling to a specified frequency can unify the data time scale, which is convenient for subsequent feature extraction. Second-order Butterworth filtering can specifically filter out low-frequency drift, such as baseline drift, and high-frequency noise, such as electromyographic interference, while retaining the effective frequency bands related to abnormal discharges in the EEG signal. Lead integration fuses multi-channel signals through electrode configuration rules, reducing data dimensionality while retaining spatial information of brain regions. Amplitude truncation can remove extreme outliers, such as spikes caused by poor electrode contact, and prevent noise interference. The signal-to-noise ratio of the preprocessed multi-channel EEG signal is improved, effectively preserving the temporal and spatial characteristics of abnormal discharges.

[0104] Those skilled in the art will understand that the training process of the abnormal discharge detection model is based on the same logic as the detection process.

[0105] like Figure 6 As shown, an abnormal discharge detection method according to another embodiment of this disclosure includes: Step S602: Synchronously acquire the multimodal raw data of the tested object.

[0106] The multimodal raw data includes EEG signal acquisition and video signal acquisition.

[0107] Electroencephalogram (EEG) signal acquisition can be performed using a multi-channel EEG machine, with electrodes configured based on a 10-20 standard EEG system, to acquire EEG signals from the patient in real time and record potential changes in different brain regions.

[0108] Video signal acquisition can be performed simultaneously with video equipment to capture the patient's facial and body movements at a frame rate of 15fps, ensuring that the video image and EEG signal are precisely aligned in the time dimension.

[0109] Step S604: Standardize the collected video EEG data and construct labeled training / test samples.

[0110] Specifically, a 4-second non-overlapping sliding window strategy is adopted to divide the synchronized video EEG data into several 4-second segments for model training and testing. This window duration ensures that each segment contains sufficient abnormal discharge-related features (such as EEG spikes and limb twitching movements) while preventing feature dilution due to excessively long segments. Non-overlapping sampling ensures sample independence, reduces data redundancy, and improves training efficiency.

[0111] In addition, to address the issue of ambiguous labels caused by the occurrence area being located at the segment boundary, label determination rules are set, including: if the occurrence time of abnormal discharge falls within the 0.5-3.5 second range of the segment, avoiding the boundary regions of 0.5 seconds at the beginning and end of the segment, then the segment is labeled as a positive sample (i.e., containing abnormal discharge); if the occurrence time does not fall within this range or the segment does not have abnormal discharge, then it is labeled as a negative sample (i.e., without abnormal discharge), ensuring clear distinction between positive and negative sample categories and reducing model learning bias.

[0112] Step S606 involves performing four preprocessing steps on each segmented EEG signal to obtain the channel EEG signal, specifically including: EEG signals were uniformly resampled to 500Hz to standardize the time scale and ensure that the temporal features of different samples could be compared.

[0113] A second-order Butterworth filter is used to filter out low-frequency drift below 0.5Hz (such as baseline drift) and high-frequency noise above 45Hz (such as electromyography interference), while retaining the effective frequency band signal associated with abnormal discharge.

[0114] Based on the electrode configuration rules of the 10-20 standard EEG system, the average lead signal is calculated, and the original multi-channel signal is transformed into a 19-channel average lead EEG signal, which reduces the data dimensionality while preserving the spatial correlation information of brain regions.

[0115] Extreme abnormal signals with amplitudes exceeding 1024mV (such as spikes caused by poor electrode contact) are truncated to prevent extreme values ​​from interfering with subsequent feature extraction and to maintain signal numerical stability.

[0116] In step S608, the preprocessed 19-channel EEG signal is input into EfficientNet-B0. Through the network's depthwise separable convolution, residual connections, and pooling operations, time-frequency domain features and spatial features are captured simultaneously, and a 1024-dimensional multidimensional EEG feature vector is output.

[0117] After feature abstraction and integration by multiple layers of the network, a 1024-dimensional multidimensional EEG feature vector is output by the fully connected layer. This vector retains the advantages of EfficientNet-B0 in terms of strong feature extraction ability, stable training and low inference latency, and can accurately represent abnormal discharge patterns in EEG signals.

[0118] Step S610: Convert the 4-second video clip into an image sequence, input it into the YOLOv5-Patient model, locate the patient's bounding box in each frame of the image, and crop the image to obtain an image containing only the patient.

[0119] Step S612: Input the cropped image sequence into the Sapiens model to detect key points on the patient's face and body, output the spatial coordinates of each key point, and capture the spatial position information of the action.

[0120] Step S614: The preliminary motion feature information generated by the temporal change analysis of the motion pattern based on the spatial coordinates of key points is input into the multilayer perceptron. Through nonlinear transformation and feature abstraction of the fully connected layer, a 1024-dimensional multidimensional video feature vector is finally output to characterize the motion behavior features of the tested object.

[0121] Step S616: Based on multidimensional EEG feature vectors and multidimensional video feature vectors, feature fusion and iterative training are performed to generate a detection model that can identify abnormal discharges.

[0122] Among them, the feature concatenation method is adopted to connect the 1024-dimensional EEG feature vector with the 1024-dimensional video feature vector end to end to form a 2048-dimensional fused feature vector. This fusion method completely preserves the original information of the two modalities, enabling the subsequent classification model to simultaneously learn the association pattern between EEG physiological abnormalities and motor behavior abnormalities, such as the temporal synchronization of EEG spikes and facial twitching.

[0123] In addition, a binary classifier based on the third multilayer perceptron (MLP) is adopted. The input layer receives a 2048-dimensional fusion vector, the hidden layer enhances the feature expression through nonlinear activation functions such as ReLU, and the output layer adopts the Sigmoid activation function. The weighted sum of the features after linear transformation and bias processing is mapped to the classification probability in the range of 0-1. The closer the probability is to 1, the higher the probability that the segment contains abnormal discharge.

[0124] The labeled fused feature vectors are used to construct a training dataset. The loss value (such as cross-entropy loss) between the classification probability and the true label of the sample (positive / negative sample) is used as the optimization objective. The model parameters are updated through backpropagation, and the training is iterated until the model loss converges and the classification accuracy is stable, finally obtaining the trained abnormal discharge detection model.

[0125] like Figure 7 As shown, EEG signals are input into a waveform feature model, while video signals are input into a video-based keypoint detection model.

[0126] Waveform feature models extract waveform features that characterize the properties of EEG signals; video-based keypoint detection models detect relevant keypoints in video signals and then extract video features.

[0127] The waveform features extracted from the EEG signal and the key point-related features extracted from the video signal are fused to obtain the fused features.

[0128] The loss value is calculated using the fused features and the annotation information (i.e., the known label data of whether it is abnormal).

[0129] Based on the calculated loss value, the gradient is calculated, and the parameters of the waveform feature model, the video-based keypoint detection model, and the feature fusion part are updated along the reverse direction of the gradient to minimize the loss. This allows the model to better learn the feature patterns related to abnormalities in EEG signals and video signals, improving the accuracy of subsequent abnormality detection. By continuously repeating this process (input data, feature extraction, feature fusion, loss calculation, backpropagation to update parameters), the model is gradually trained to accurately detect abnormal situations.

[0130] Step S618: Input the multidimensional EEG feature vector, multidimensional facial movement feature vector, and multidimensional body movement feature vector into the abnormality detection model, and output the abnormal discharge detection result.

[0131] After obtaining the abnormal discharge detection model, EEG and video signals from the tested patients are simultaneously acquired and segmented into segments in units of n seconds (usually using the 4 seconds from training to ensure matching with the model's input dimension). The segmented EEG segments undergo the same preprocessing procedures as the training phase, while the video segments undergo image sequence generation, target cropping, and key point detection processes to extract 1024-dimensional EEG feature vectors and 1024-dimensional video feature vectors, respectively. The two types of feature vectors yield a 2048-dimensional fusion vector, which is input into the trained abnormal discharge detection model. The model outputs classification probabilities. If the probability exceeds a preset threshold, the segment is determined to have an abnormal discharge; otherwise, it is determined to be normal. The detection results (abnormal / normal) for each segment every n seconds are output in real time, providing timely early warnings of abnormal discharges for clinical monitoring.

[0132] like Figure 8 As shown, the abnormal discharge detection method according to another embodiment of this disclosure is applied in a hospital clinical diagnosis and treatment scenario. When a patient is suspected of having epilepsy, and the doctor needs to monitor the patient's brain electrical state and whether there are abnormal discharges in real time, this method can be used to carry out the detection. The specific process is as follows: First, the EEG signal acquisition device 802 acquires multi-channel EEG signals from the patient 804. Multiple probe electrodes 806 are fixed at specific positions on the patient's head (such as electrode layout based on the 10-20 standard EEG system), and the electrodes are connected to the interface 808 of the EEG signal acquisition device 802. This allows for the real-time acquisition of multi-channel EEG signals reflecting the potential changes in different brain regions of the patient. The form of the EEG signal acquisition device is not limited. It can be a conventional multi-channel EEG machine, a helmet, hat, or bedding with integrated electrodes, or a packaged portable probe electrode, etc., depending on the actual needs.

[0133] While collecting EEG signals, the camera 810 simultaneously captures video of the patient's face and torso, fully recording the patient's limb movements and facial expression changes, ensuring that the EEG signals and video signals are accurately aligned in the time dimension, providing a foundation for subsequent multimodal analysis.

[0134] The EEG signal acquisition device 802 and the abnormal discharge detection device 812 are coupled (the two can be integrated into the same device, such as an intelligent EEG monitoring helmet with a camera). The synchronously acquired multi-channel EEG signals and patient video data are transmitted to the abnormal discharge detection device 812. The abnormal discharge detection device is loaded with an abnormal detection model. It performs preprocessing (such as resampling, filtering, lead integration, etc.) and feature extraction on the multi-channel EEG signals, and performs target localization, key point detection and action feature extraction on the video data. Then, it realizes abnormal discharge detection through feature fusion and classification model and generates detection results.

[0135] Furthermore, the abnormal discharge detection device 812 outputs the final detection results to the display 814, allowing doctors or patients to visually view whether the patient has abnormal discharges and related details, providing direct evidence for clinical diagnosis, treatment plan formulation, and efficacy evaluation.

[0136] like Figure 9 As shown, the abnormal discharge detection method according to another embodiment of this disclosure is applied to cloud-based remote detection in portable scenarios, such as portable monitoring scenarios in non-medical locations like homes and offices, to meet patients' daily self-monitoring needs. The specific process is as follows: Patients can autonomously complete data acquisition in any scenario using a portable EEG signal acquisition device 802 and a camera 810. The portable EEG signal acquisition device 802 is flexible in form and can be a portable EEG machine, a head-mounted device with electrodes (such as a lightweight EEG helmet), clothing or bedding with integrated electrodes, etc. Multi-channel EEG signals are acquired through electrodes 806 fixed to the head. The electrodes 806 are fixed at a specific position on the patient's head (such as an electrode layout based on the 10-20 standard EEG system) and connected to the interface 808 of the EEG signal acquisition device 802. At the same time, the camera 810 (which can be integrated into the acquisition device or use the camera of a mobile phone, tablet or other common device) is used to capture facial and trunk videos simultaneously, realizing data acquisition without scene limitations.

[0137] The portable EEG signal acquisition device is coupled with the data transceiver 902. After acquisition, the data transceiver uploads the multi-channel EEG signal and video data to the remote server 904. The type of server is not limited and can be a cloud server, distributed server, etc. After receiving the data, the server calls the brain abnormal discharge detection algorithm provided in this solution to perform standardization processing, feature extraction, feature fusion and classification judgment on the uploaded multimodal data, accurately identify whether there is abnormal discharge and generate detection results.

[0138] The server transmits the abnormal discharge detection results back to the patient's data transceiver, which then outputs the results to the patient's client display 906 (such as the screen of a mobile phone, computer, tablet, or other commonly used device). The patient can view their monitoring results in real time and seek medical attention promptly if any abnormalities are detected. At the same time, the detection results can also be synchronized to the doctor's client as needed, allowing the doctor to remotely monitor changes in the patient's condition and provide guidance for subsequent diagnosis and treatment.

[0139] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0140] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0141] The following reference Figure 10 The abnormal discharge detection device 1000 according to this embodiment of the present invention will be described. Figure 10 The abnormal discharge detection device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0142] The abnormal discharge detection device 1000 is manifested in the form of hardware modules. The components of the abnormal discharge detection device 1000 may include, but are not limited to: a preprocessing module 1002, used to preprocess the received EEG signal from the test subject to obtain a multi-channel EEG signal; a first feature extraction module 1004, used to perform time-frequency domain and spatial feature extraction on the multi-channel EEG signal to obtain a multi-dimensional EEG feature vector; a second feature extraction module 1006, used to extract multi-dimensional facial movement feature vectors and multi-dimensional body movement feature vectors from the received monitoring video of the test subject, respectively, with the monitoring video and EEG signal time-series synchronized; and a detection module 1008, used to detect abnormal discharges based on the multi-dimensional EEG feature vectors, multi-dimensional facial movement feature vectors, and multi-dimensional body movement feature vectors, and obtain the detection result.

[0143] The following reference Figure 11 To describe an electronic device 1100 according to this embodiment of the present invention. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0144] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110).

[0145] The storage unit stores program code, which can be executed by the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1110 can perform actions such as... Figure 2 Steps S202 and S208 shown, as well as other steps defined in the abnormal discharge detection method of this disclosure.

[0146] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 11201 and / or cache memory 11202, and may further include read-only memory (ROM) 11203.

[0147] Storage unit 1120 may also include a program / utility 11204 having a set (at least one) program module 11205, such program module 11205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0148] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0149] Electronic device 1100 can also communicate with one or more external devices 1160 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1140. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1150. As shown, network adapter 1150 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0151] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0152] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0154] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0155] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0156] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0157] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0158] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0159] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0160] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A method for detecting abnormal discharge, characterized in that, include: The received EEG signals from the test subjects are preprocessed to obtain multi-channel EEG signals; The multi-channel EEG signal is subjected to feature extraction based on time-frequency domain and spatial characteristics to obtain a multi-dimensional EEG feature vector. The multi-channel EEG signal is a two-dimensional matrix of time and electrode channels. The extraction process includes: inputting the multi-channel EEG signal into a two-dimensional convolutional 2D-CNN model to perform sliding operations of the two-dimensional convolutional layers to capture potential change patterns in different frequency ranges based on the sliding of the convolutional kernel on the time axis, thereby obtaining time-frequency domain features; and capturing potential correlations between different channels based on the sliding of the convolutional kernel on the channel axis, thereby obtaining spatial features; and performing hierarchical integration of the time-frequency domain features and the spatial features to obtain the multi-dimensional EEG feature vector. Multidimensional facial movement feature vectors and multidimensional body movement feature vectors are extracted from the received monitoring video of the test subject, respectively. The monitoring video and the electroencephalogram signal are synchronized in time. The extraction of the multidimensional facial movement feature vectors and the multidimensional body movement feature vectors are both performed based on the corresponding Sapiens model and multilayer perceptron. Abnormal discharges are detected based on the multidimensional EEG feature vector, the multidimensional facial movement feature vector, and the multidimensional body movement feature vector, and the detection results are obtained.

2. The abnormal discharge detection method according to claim 1, characterized in that, The two-dimensional convolutional 2D-CNN model uses EfficientNet-B0 as the backbone network to hierarchically integrate the time-frequency domain features and the spatial features to obtain the multidimensional EEG feature vector, including: Through multi-layer convolution, residual connections, and pooling operations of EfficientNet-B0, the time-frequency domain features and spatial features are hierarchically abstracted and fused, and the multi-dimensional EEG feature vector of a preset dimension is output by the fully connected layer of EfficientNet-B0.

3. The abnormal discharge detection method according to claim 1, characterized in that, Capturing the potential correlation between different channels based on the sliding of the convolution kernel along the channel axis includes: The system captures the synchronicity of potential changes between different channels and the distribution of potential differences between different channels by sliding the convolution kernel along the channel axis.

4. The abnormal discharge detection method according to claim 1, characterized in that, The multidimensional facial motion feature vector and multidimensional body motion feature vector are extracted from the received surveillance video of the tested object, including: The surveillance video is segmented into an image sequence based on a specified frame rate; The image sequence is input into a feature detection model to perform facial and body feature detection, so that the feature detection model outputs the multidimensional facial motion feature vector and the multidimensional body motion feature vector.

5. The abnormal discharge detection method according to claim 4, characterized in that, The feature detection model includes the YOLOv5-Patient model. The image sequence is input into the feature detection model for facial and body feature detection, and the model outputs the multidimensional facial motion feature vector and the multidimensional body motion feature vector, including: The image sequence is input into the YOLOv5-Patient model, which detects the location of objects in the image and determines the bounding box region of the object in the image, thus obtaining a cropped image of the object. Based on time sequence, face and body recognition are performed on the cropped image to obtain face video segments and body video segments, respectively. Extract the multidimensional facial motion feature vector based on the facial video segment; The multidimensional body motion feature vector is extracted based on the body video clip.

6. The abnormal discharge detection method according to claim 5, characterized in that, The feature detection model further includes a first Sapiens model and a first multilayer perceptron, and extracts the multidimensional facial action feature vector based on the facial video segment, including: The facial video clip is input into the first Sapiens model, and the first Sapiens model performs facial key point detection to obtain the spatial coordinates of the facial key points. Facial motion analysis is performed based on the temporal changes in the spatial coordinates of the facial key points to obtain facial motion feature information; The facial motion feature information is input into the first multilayer perceptron, which performs depth feature extraction to obtain the multidimensional facial motion feature vector.

7. The abnormal discharge detection method according to claim 5, characterized in that, The feature detection model further includes a second Sapiens model and a second multilayer perceptron, which extracts the multidimensional body motion feature vector based on the body video segment, including: The body video clip is input into the second Sapiens model, and the second Sapiens model performs body key point detection to obtain the spatial coordinates of the body key points. Human motion analysis is performed based on the temporal changes in the spatial coordinates of the key body points to obtain body motion feature information; The body motion feature information is input into the second multilayer perceptron, which performs depth feature extraction to obtain the multidimensional body motion feature vector.

8. The abnormal discharge detection method according to claim 1, characterized in that, Abnormal discharges are detected based on the multidimensional EEG feature vector, the multidimensional facial movement feature vector, and the multidimensional body movement feature vector, yielding detection results including: The multidimensional EEG feature vector, the multidimensional facial movement feature vector, and the multidimensional body movement feature vector are concatenated to obtain a fusion vector; Abnormal discharge detection is performed based on the fusion vector to obtain the detection results.

9. The abnormal discharge detection method according to claim 8, characterized in that, Abnormal discharge detection is performed based on the fused vector to obtain detection results, including: The fused vector is input into an anomaly detection model constructed based on a classifier of a third multilayer perceptron. The weighted sum of the fused vector after linear transformation and bias processing is mapped to a classification probability, and the detection result is output based on the classification probability.

10. The abnormal discharge detection method according to claim 9, characterized in that, Also includes: Using a specified duration as a sampling window, a sampling operation based on the non-overlapping sliding of the sampling window is performed on the historical EEG signal and the corresponding historical monitoring video of the object. Within a sampling window, positive samples are constructed based on the historical EEG signal and the historical monitoring video of the object located within a specified interval, and negative samples are constructed based on the historical EEG signal and the historical monitoring video of the object located on both sides of the specified interval. The specified interval represents the interval of abnormal discharge. A fused vector dataset with labels is constructed based on the positive samples and the negative samples respectively; The anomaly detection model is obtained by training a neural network including an initial multilayer perceptron based on the fused vector dataset.

11. The abnormal discharge detection method according to claim 1, characterized in that, The received EEG signals from the subject are preprocessed to obtain multi-channel EEG signals, including: The electroencephalogram (EEG) signal is resampled to a specified frequency; Based on the specified frequency, a second-order Butterworth filter is performed to filter out low-frequency drift and high-frequency noise, resulting in a filtered EEG signal. Based on electrode configuration rules, the filtered EEG signals are integrated into leads to generate multi-channel average lead EEG signals. The amplitude of the multi-channel average lead EEG signal is truncated to obtain the multi-channel EEG signal.

12. An abnormal discharge detection device, characterized in that, include: The preprocessing module is used to preprocess the received EEG signals from the test subject to obtain multi-channel EEG signals; The first feature extraction module is used to extract features from the multi-channel EEG signal based on the time-frequency domain and space to obtain a multi-dimensional EEG feature vector. The multi-channel EEG signal is a two-dimensional matrix of time and electrode channels. The module includes: inputting the multi-channel EEG signal into a two-dimensional convolutional 2D-CNN model to perform sliding operations of the two-dimensional convolutional layers to capture potential change patterns in different frequency ranges based on the sliding of the convolutional kernel on the time axis to obtain time-frequency domain features, and capturing potential correlations between different channels based on the sliding of the convolutional kernel on the channel axis to obtain spatial features; and performing hierarchical integration of the time-frequency domain features and the spatial features to obtain the multi-dimensional EEG feature vector. The second feature extraction module is used to extract multidimensional facial action feature vectors and multidimensional body action feature vectors from the received monitoring video of the test subject. The monitoring video and the electroencephalogram signal are synchronized in time. The extraction of the multidimensional facial action feature vectors and the multidimensional body action feature vectors are both performed based on the corresponding Sapiens model and multilayer perceptron. The detection module is used to detect abnormal discharges based on the multidimensional EEG feature vector, the multidimensional facial movement feature vector, and the multidimensional body movement feature vector, and to obtain the detection results.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the abnormal discharge detection method according to any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the abnormal discharge detection method according to any one of claims 1 to 11.

15. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the abnormal discharge detection method according to any one of claims 1 to 11.

Images