A method for constructing an epilepsy classification model based on the coordination of facial and posture features, an epilepsy monitoring device, equipment, medium and product

Abstract

This application discloses a method for constructing an epilepsy classification model that coordinates facial and posture features, an epilepsy monitoring device, equipment, medium, and products, and relates to the field of intelligent medical devices. The method first constructs a feature extraction module; then constructs an adaptive graph convolutional network model that integrates facial and posture information, namely a FAGCN network model; then constructs an epilepsy dataset and uses the epilepsy dataset to train the FAGCN network model to obtain a trained FAGCN network model; then connects the feature extraction module and the trained FAGCN network model to obtain an epilepsy classification model that coordinates facial and posture features. This application constructs an adaptive graph convolutional network model that integrates facial and posture information for epilepsy classification, which can directly realize epilepsy classification and monitoring based on video data.

Description

Technical Field

[0001] The present application relates to the field of medical devices, and in particular to a method for constructing an epilepsy classification model that coordinates facial and posture features, an epilepsy monitoring device, equipment, medium, and product. Background Art

[0002] Epilepsy is a chronic disease characterized by sudden abnormal discharges of brain neurons, leading to transient brain dysfunction. According to the latest epidemiological data in China, the overall prevalence of epilepsy in China is 7.0‰. An estimated 9 million people in China suffer from epilepsy, with approximately 400,000 new cases each year. Epilepsy has become the second most common neurological condition in China, second only to headaches.

[0003] Epilepsy specialists use visual signs such as involuntary body stiffness or spasms, facial loss, and flexed hands, caused by epileptic seizures, as well as other important features, to identify epilepsy. However, faced with a large number of cases, identifying epileptic seizures often requires considerable time and effort. Therefore, developing a more accurate epilepsy classification algorithm for patient monitoring, management, and diagnostic decision-making is an effective way to reduce the burden on specialists and assist them in diagnosis and treatment. By utilizing this epilepsy classification algorithm during a seizure, monitoring, and other facilities can ensure timely detection and treatment, effectively safeguarding patients' health.

[0004] Currently, deep learning-based classification methods have demonstrated exceptional performance across various fields. Neural networks can overcome these challenges by automatically learning features from training data that are more robust to changes in data distribution. Within the field of deep learning-based epilepsy classification, research primarily focuses on epilepsy classification based on electroencephalogram (EEG) data and visual data.

[0005] Deep learning-based epilepsy research has mostly been based on electroencephalograms (EEGs). Significant progress has been made in EEG detection, with studies widely using deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long-short-term memory (LSTMs) to process epilepsy EEG data. Numerous open-source EEG datasets annotated by experts are also available. However, using EEG to identify patient onset requires specialized equipment, which can restrict patient mobility. Meanwhile, video data, as a non-invasive information source, has shown great potential in disease detection, such as Parkinson's disease, Alzheimer's disease, and spinal muscular atrophy. However, due to the complex and variable visual manifestations of epileptic seizures, the visual features of videos used to distinguish different types of epileptic seizures are not clearly distinguishable, and some types of epileptic seizures lack specific visual features. Furthermore, the recording conditions of clinical surveillance videos vary widely, and epileptic seizures often begin and end abruptly. Consequently, there is a severe lack of annotated open-source epilepsy video datasets, leading to limited research. Summary of the Invention

[0006] The purpose of this application is to provide a method for constructing an epilepsy classification model that coordinates facial and posture features, an epilepsy monitoring device, equipment, medium and product, so as to realize epileptic seizure monitoring based on video data and overcome the defect that electroencephalogram monitoring restricts patient activities.

[0007] To achieve the above objectives, this application provides the following solutions.

[0008] In a first aspect, the present application provides a method for constructing an epilepsy classification model based on the synergy of facial and posture features, comprising:

[0009] Constructing a feature extraction module; the feature extraction module is used to extract the skeleton point feature sequence and facial feature sequence of the target person in the video data;

[0010] Constructing a FAGCN network model; the FAGCN network model is an adaptive graph convolutional network model that integrates face and posture, and the FAGCN network model includes: 10 FAGCN blocks connected in sequence and 1 fully connected layer, wherein the first, fifth and eighth FAGCN blocks are all first FAGCN blocks, the first FAGCN block is connected to the fifth FAGCN block through a Reshape function and a first convolution and tiling module, the fifth FAGCN block is connected to the eighth FAGCN block through a Reshape function and a first convolution and tiling module, and the remaining seven FAGCN blocks are all second FAGCN blocks; the first FAGCN block is used to process a skeleton point feature sequence and a facial feature sequence, and the second FAGCN block is used to process a skeleton feature sequence;

[0011] Construct an epilepsy dataset;

[0012] Based on the epilepsy dataset, the FAGCN network model is trained to obtain a trained FAGCN network model;

[0013] The feature extraction module is connected to the trained FAGCN network model to obtain an epilepsy classification model that coordinates facial and posture features.

[0014] In a second aspect, the present application provides an epilepsy monitoring device, the device comprising: a camera, a server, and a terminal;

[0015] The camera is connected to the server, which is connected to the terminal. The server is provided with an epilepsy classification model based on the coordination of facial and posture features, and the epilepsy classification model based on the coordination of facial and posture features is obtained using the above-mentioned method for constructing an epilepsy classification model based on the coordination of facial and posture features.

[0016] The camera is used to monitor the target person and obtain video data;

[0017] The server is configured to perform epilepsy recognition based on the video data using the epilepsy classification model coordinated with facial and posture features to obtain a recognition result;

[0018] The terminal is used to receive and display the recognition result.

[0019] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned method for constructing an epilepsy classification model based on the coordination of facial and posture features.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for constructing an epilepsy classification model that coordinates facial and posture features.

[0021] In a fifth aspect, the present application provides a computer program product, which, when executed by a processor, implements the above-mentioned method for constructing an epilepsy classification model that coordinates facial and posture features.

[0022] According to the specific embodiments provided in this application, this application has the following technical effects.

[0023] This application provides a method for constructing an epilepsy classification model that collaborates with facial and posture features, as well as an epilepsy monitoring device, equipment, medium, and product. First, a feature extraction module is constructed; then, an adaptive graph convolutional network model that integrates facial and posture information, namely, a FAGCN network model, is constructed; then, an epilepsy dataset is constructed and the FAGCN network model is trained using the epilepsy dataset to obtain a trained FAGCN network model. The feature extraction module and the trained FAGCN network model are then connected to obtain an epilepsy classification model that collaborates with facial and posture features. This application constructs an adaptive graph convolutional network model that integrates facial and posture information for epilepsy classification, which can directly implement epilepsy classification and monitoring based on video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a method for constructing an epilepsy classification model based on the collaboration of facial and posture features provided in one embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of the FAGCN network model provided in one embodiment of the present application;

[0027] Figure 3 A schematic diagram of skeleton information provided in an embodiment of the present application;

[0028] Figure 4 This is a graph showing the accuracy of each algorithm training process provided in one embodiment of the present application;

[0029] Figure 5 A Loss curve diagram of each algorithm training process provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0033] In an exemplary embodiment, Figure 1 As shown, a method for constructing an epilepsy classification model based on the coordination of facial and posture features is provided, including the following steps 101 to 105.

[0034] Step 101: construct a feature extraction module; the feature extraction module is used to extract the skeleton point feature sequence and facial feature sequence of the target person in the video data.

[0035] Step 102: construct a FAGCN network model; the FAGCN network model is an adaptive graph convolutional network model that integrates facial and posture information, and the FAGCN network model includes: 10 FAGCN blocks connected in sequence and 1 fully connected layer, wherein the 1st, 5th and 8th FAGCN blocks are all first FAGCN blocks, the 1st FAGCN block is connected to the 5th FAGCN block through a Reshape function and a first convolution and tiling module, the 5th FAGCN block is connected to the 8th FAGCN block through a Reshape function and a first convolution and tiling module, and the remaining 7 FAGCN blocks are all second FAGCN blocks; the first FAGCN block is used to process a skeleton point feature sequence and a facial feature sequence, and the second FAGCN block is used to process a skeleton feature sequence.

[0036] Step 103: construct an epilepsy dataset.

[0037] Step 104 : training the FAGCN network model based on the epilepsy dataset to obtain a trained FAGCN network model.

[0038] Step 105 : Connect the feature extraction module and the trained FAGCN network model to obtain an epilepsy classification model that coordinates facial and posture features.

[0039] By implementing the above steps 101 to 105, an adaptive graph convolutional network model integrating facial and posture information is constructed for epilepsy classification, which can directly realize epilepsy classification and monitoring based on video data.

[0040] Before building the adaptive graph convolutional network model that integrates facial and posture information in the above steps, a model that can use visual data for epilepsy classification was studied.

[0041] In recent years, advancements in computer technology, the development of deep learning techniques, and the increased computing power of graphics cards have made it possible to use deep learning techniques to solve video classification problems. Deep learning models hold considerable potential in video recognition and classification, and can be used to detect and identify signs or characteristics of certain conditions, including epilepsy. Some studies, such as the algorithm proposed by Ahmedt-Aristizabal D et al., use a convolutional neural network (CNN) to extract motion features from each frame of a video. A recurrent neural network then extracts information from the aggregated feature segments, ultimately performing classification through a fully connected layer. This approach has several challenges. First, to reduce computational complexity, the algorithm determines the bed's position in the first frame and then crops the image centered on the bed, which has limitations. Furthermore, bed displacement can significantly impact algorithm accuracy. Second, using CNNs to extract motion features on medical datasets, which are generally small, results in limitations in the extracted motion features.

[0042] In subsequent studies, such as P'erez- The epilepsy research algorithm proposed by F et al. does not use the motion model trained on the current medical dataset. Instead, it uses the CNN model STCNN trained on a large-scale natural human motion dataset to extract motion sequences, and then uses RNN to predict and classify the sequences, achieving an accuracy of about 90%.

[0043] The above methods are effective frameworks for studying the identification and classification of symptoms based on visual data. However, in epilepsy classification research, facial information during epileptic seizures is not utilized, and sequence classification models such as RNNs are relatively basic algorithms. With the development of deep learning technology, more time series classification models have been proposed, such as LSTM, Seq2Seq, and Transformer. There are also spatiotemporal sequence models that extract features from sequences in both the temporal and spatial domains, such as ST-GCN and AGCN. These models have achieved good accuracy in a wide range of areas of action recognition and classification.

[0044] By analyzing the movements and facial features during epileptic seizures and various time series classification algorithms in recent years, the embodiment of the present application is improved based on the AGCN model. On the basis of using skeleton point motion data, the facial area in the video data is located by the coordinates of the patient's facial skeleton points, so that facial features and motion features are used as input, and the processing and calculation are integrated in the algorithm to complete the classification of epilepsy.

[0045] Currently, most deep learning-based epilepsy recognition algorithms are based on EEG datasets. However, for patients, especially children, detection equipment can significantly restrict their movements, causing numerous inconveniences. However, currently, open-source, visual-based epilepsy datasets are extremely rare. Therefore, it is necessary to obtain visual data of epileptic seizures and, using a variety of methods, process this data into a dataset suitable for deep learning experiments.

[0046] In another exemplary embodiment, the epilepsy dataset construction in step 103 includes three steps: data screening, skeleton point extraction, and data processing, which are described in detail below.

[0047] 1. Data screening.

[0048] To ensure the accuracy and confidence of subsequent skeleton point extraction, which in turn affects classification precision, the epileptic seizure video data needs to be screened. The video data studied was extracted from video EEG examinations during clinical diagnosis and treatment activities. Objects obscuring the patient's body, or family members and medical staff interfering with the patient during the seizure, hinder the normal recognition of epileptic seizure movements, and therefore need to be eliminated.

[0049] 2. Skeleton point extraction.

[0050] Because epilepsy is a sudden illness, the video data is synchronized surveillance footage of video EEG examinations during clinical diagnosis and treatment. This creates a complex and diverse data environment, and surrounding medical staff and family members can also be disruptive. For situations where the patient's target is small and there is a lot of redundant data, this redundant information can be reduced by extracting the patient's skeletal point information. A pre-trained human skeletal point recognition model is used to process the epilepsy data and extract the patient's skeletal point information.

[0051] However, when using a pre-trained human skeleton point recognition model to process epilepsy data, the presence of the patient's family means that the skeleton points extracted in each frame are interfered with by other people. Therefore, it is necessary to accurately extract the patient's skeleton point information from the skeleton points of multiple people. This can be done by manually selecting the patient's skeleton point information in the first frame, and then calculating the Euclidean distance between the coordinates of each person's skeleton point in the next frame and the coordinates of the patient's skeleton point in the current frame to obtain the patient's skeleton point information in the next frame. Due to the complex data environment, there may be cases where the patient's skeleton point is not recognized in a certain frame. Calculating the Euclidean distance may select the wrong skeleton point information. Therefore, by increasing the threshold and determining whether the Euclidean distance exceeds the threshold, the extraction of incorrect skeleton point information can be prevented.

[0052] 3. Organize epilepsy data.

[0053] Because the videos contain both epileptic seizures and normal states, the output skeleton points are extracted based on the corresponding seizure times in the videos annotated by epilepsy experts. The corresponding video data is also sliced ​​based on the skeleton points to facilitate the acquisition of facial data corresponding to the skeleton points. The epilepsy data is sliced ​​into 210 frames based on the expert annotations to obtain both seizure and normal data. The skeleton point data, video data, and corresponding labels are finally organized into a dataset. The dataset is randomly divided into training, prediction, and test sets in a ratio of 7:2:1 to obtain the epilepsy dataset.

[0054] Epilepsy patient video data includes clinical videos of patients, their families, and / or medical staff. In order to obtain the patient's facial video data and input it into the subsequent algorithm steps, the epilepsy patient video data needs to be processed by the facial positioning algorithm. The facial positioning algorithm obtains all facial coordinate data from the skeleton point data corresponding to the video, which can be expressed as a set FS = {(x1, y1), (x2, y2), ... (x i ,y i )}, and then locate the patient's face area. The positioning algorithm formula is as follows:

[0055]

[0056]

[0057] Among them, FAX and FAY represent the cropping coordinates of the facial contour along the horizontal and vertical dimensions, respectively, and x1, x2 and x i are the horizontal coordinates of the first, second and i-th bone points on the target person’s face, y1, y2, y i are the ordinates of the first, second and i-th bone points on the target person’s face, V h and V wThe height and width of the video data respectively.

[0058] In formulas (1) and (2), MIN and MAX are minimum and maximum value functions.

[0059] After being processed by the facial positioning algorithm, the obtained facial area data is convolved and flattened, and then input into the subsequent FAGCN block together with the corresponding skeleton point motion data.

[0060] In another exemplary embodiment, in terms of extracting the skeleton point feature sequence and facial feature sequence of the target person in the video data, the above-mentioned feature extraction module is specifically used to: use a human posture estimation algorithm to extract the skeleton point information of each person in the n-th frame video image in the video data; n = 1, 2, ..., N, N is the number of frames of the video data; calculate the Euclidean distance between the skeleton point information of each person in the n-th frame video image and the skeleton point information of the target person in the n-1-th frame video image; determine the skeleton point information of the person with the smallest Euclidean distance in the n-th frame video image as the target person in the n-th frame video image skeleton point information of the target person in the n-th frame video image; human body pose estimation is performed based on the skeleton point information of the target person in the n-th frame video image, and the skeleton point features of the target person in the n-th frame video image are obtained; the face of the target person is located based on the skeleton point information of the target person in the n-th frame video image, and the facial contour of the target person in the n-th frame video image is obtained; the facial features of the target person in the n-th frame video image are extracted based on the facial contour of the target person in the n-th frame video image; based on the skeleton point features and facial features of the target person in each frame video image in the video data, a skeleton point feature sequence and a facial feature sequence of the target person are constructed.

[0061] In another exemplary embodiment, the overall framework of the FAGCN network model is as follows: Figure 2 As shown, Figure 2 (a) is a structural diagram of the input part of the FAGCN network model. Figure 2 (b) is a schematic diagram explaining the composition structure of the FAGCN network model. Figure 2 (c) in the figure is a schematic diagram of the overall structure of the FAGCN network model.

[0062] like Figure 2As shown in (c) of Figure 1, FAGCN mainly consists of 10 FAGCN blocks that are improved based on the AGCN block. Each FAGCN block contains a graph convolution module (GCN) and a temporal convolution module (TCN). In the graph convolution module GCN, the input data is first downsampled through convolution conv1 to extract features, and then the data dimension is adjusted using the Reshape and Transpose functions. The subsequent convolution conv2 is then used to determine the neighborhood of the skeleton points in the data and perform weighted summation to highlight the important features of the skeleton movement. The temporal convolution module (TCN) contains two normalization functions (BN) and an activation function (RELU) as well as a temporal convolution. The convolution uses a convolution kernel of size 9×1 to perform calculations in the temporal dimension to extract continuous motion features from the skeleton point sequence. The number of input and output channels of each FAGCN block in the algorithm is shown in Table 1.

[0063] Table 1 Number of input and output channels of FAGCN block

[0064]

[0065] Among the 10 FAGCN blocks, they can be divided into two categories according to the input. The first category is the 1st, 5th, and 8th FAGCN blocks represented by blue (i.e. Figure 2 The second category is the FAGCN block represented by green (i.e. Figure 2 Green Block in the ). Figure 2 Taking Block 1 in (a) as an example, we introduce the FAGCN block (shown in blue), the first FAGCN block. The input to these three first FAGCN blocks includes not only a sequence of skeletal feature points but also a sequence of facial feature points. The skeletal feature points are processed by a graph convolution module; the facial feature points undergo convolution and flattening, converting the two-dimensional image features into a one-dimensional sequence. The two outputs are then converted to the same dimension and concatenated before being fed into a temporal convolution module to extract temporal features. After processing by the temporal convolution module, the two feature sequences are split. The skeletal feature points are fed into subsequent FAGCN blocks (shown in green), while the facial feature points are converted back into a two-dimensional feature sequence using the Reshape function. After convolution and flattening, they are then fed into the fifth or eighth FAGCN block. The green FAGCN blocks only take the skeletal feature points as input, and residual convolution is also incorporated to improve algorithm stability.

[0066] Finally, the skeleton feature sequence processed by 10 FAGCN blocks and the facial feature sequence processed by 3 FAGCN blocks are input into the fully connected layer to obtain the final classification result.

[0067] Therefore, in an embodiment of the present application, the first FAGCN block includes: a first graph convolution module, a second convolution and tiling module, a connection module, a first time convolution module and a first post-processing module; the first graph convolution module and the second convolution and tiling module are both connected to the connection module, the connection module is connected to the first time convolution module, and the first time convolution module is connected to the first post-processing module; the first graph convolution module is used to perform deep feature extraction on the target skeleton point feature sequence to obtain a spatial target skeleton point deep feature sequence; the target skeleton point feature sequence is a skeleton point feature sequence or a skeleton point deep feature sequence output by the second FAGCN block located before the first FAGCN block; the second convolution The tiling module is used to extract deep features from the target facial feature sequence to obtain a spatial target facial depth feature sequence; the target facial feature sequence is a facial feature sequence or a facial depth feature sequence output by the first FAGCN block located before the first FAGCN block; the connection module is used to connect the spatial target bone point depth feature sequence and the spatial target facial depth feature sequence to obtain a facial-pose heterogeneous graph structure sequence; the first temporal convolution module is used to extract temporal features from the facial-pose heterogeneous graph structure sequence to obtain a facial-pose depth feature sequence; the first post-processing module is used to separate the facial-pose depth feature sequence to obtain a bone point depth feature sequence and a facial depth feature sequence.

[0068] The second FAGCN block includes a second graph convolution module and a second temporal convolution module connected in sequence; the second graph convolution module is used to perform deep feature extraction on the target bone point feature sequence to obtain a spatial target bone point deep feature sequence; the target bone point feature sequence is the bone point deep feature sequence output by the first FAGCN block or the second FAGCN block located before the second FAGCN block; the second temporal convolution module is used to perform temporal feature extraction on the spatial target bone point deep feature sequence to obtain a bone point deep feature sequence.

[0069] In order to illustrate the effect of the model constructed in this application, experimental verification was carried out, and the characteristics of the experimental data used in the experiment and the processing process of the experimental data into a data set were introduced. Then, the experimental results of the model of this application were displayed and compared with other mainstream algorithms to prove the effectiveness of the FAGCN network model, as follows.

[0070] The data used in the examples of this application are derived from clinical data from Beijing Children's Hospital affiliated to Capital Medical University. After being labeled by epilepsy specialists, a total of 38 video records of seizures from 30 patients were collected, including clinical information such as video frame number, seizure time node and seizure type. These video materials were collected from synchronized monitoring videos of patients undergoing video EEG examinations, and the average length of a single video segment was about 2 minutes. In view of the sudden nature of epileptic seizures, in addition to recording the patient's performance, the video also includes activity images of family members and medical staff during clinical treatment, and different patients exhibit different movement characteristics during seizures.

[0071] In order to extract the patient's kinematic characteristics, the present embodiment uses a pre-trained human skeleton point recognition model to analyze epileptic seizure videos. Specifically, the OpenPose deep learning human posture estimation tool is used, which can accurately detect and evaluate human key points and posture information from images or videos. Figure 3 As shown in the figure, the human skeleton consists of 25 key points (numbered 0-24), corresponding to: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, middle hip, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, left big toe, left little toe, left heel, right big toe, right little toe, and right heel. Skeletal point information was obtained through OpenPose processing, and the video clips were segmented into seizure and non-seizure time nodes based on the expert-annotated seizure time nodes. This was used to construct a binary classification dataset.

[0072] This experiment was conducted on a GPU server in the Baidu PaddlePaddle AI Studio environment, using the open-source Baidu PaddlePaddle open platform, framework version 2.2.2, and Python version 3.7. The dataset used was the self-developed epilepsy classification dataset described above. To demonstrate the effectiveness of incorporating facial features in this algorithm, it was compared with five time series algorithms: LSTM, Transformer, ST-GCN, AGCN, and CTR-GCN, which only input skeletal point sequences. The algorithm was trained for 100 epochs using the Adam optimizer, with a learning rate of 0.001, weight decay of 0.0001, and a cross-entropy loss function.

[0073] The training process of each algorithm in the experiment draws the Accuarcy and Loss curves, such as Figure 4 、 Figure 5 shown.

[0074] Figure 4 、 Figure 5The accuracy and loss curves for each algorithm model from 0 to 100 training rounds are shown in Table 2. It can be seen that the FAGCN algorithm's fitting performance is slightly higher than that of other algorithms in training rounds 80 to 100, indicating that the addition of facial features improves the algorithm's fitting performance. After 100 rounds of training, the experimental results are presented in Table 2.

[0075] The data in Table 2 are the training models of six algorithms on the epilepsy classification training set and the classification results on the test set. In Table 2, the algorithm FAGCN of this application is distinguished from the five comparison algorithms based on the input data, and the algorithm test results are displayed using four evaluation indicators: Precision, Recall, F1 Score, and Accuarcy. The best results are marked in bold. It can be seen that the FAGCN algorithm of this application surpasses the other five algorithms in all four evaluation indicators, proving that FAGCN has a high-precision classification effect on epileptic seizures. It also proves that adding facial features can assist motion features in improving the accuracy of the epilepsy classification algorithm.

[0076] Table 2 Experimental results

[0077]

[0078]

[0079] Epilepsy attacks often occur suddenly and stop suddenly. If they cannot be discovered and treated in time, it often leads to injury or even death of the patient. As a non-invasive monitoring method, real-time observation of whether the patient is ill through video monitoring can effectively reduce manpower and material resources. The embodiment of the present application analyzes the movement characteristics and facial features during epileptic seizures, and proposes an algorithm FAGCN that is improved based on AGCN and integrates facial features and skeletal point movement features to classify epilepsy. By extracting synchronized video data from video EEG monitoring of epilepsy patients, an epilepsy classification dataset was created, and experiments were conducted on this dataset using FAGCN and multiple comparison algorithms, which proved the effectiveness and superiority of the FAGCN algorithm. It also proved that adding facial features can effectively improve the algorithm's classification accuracy for epilepsy.

[0080] In an exemplary embodiment, the epilepsy classification model based on the coordinated facial and posture features constructed in the above-mentioned method embodiment is applied to an epilepsy monitoring device, which includes: a camera, a server and a terminal; the camera is connected to the server, and the server is connected to the terminal; the epilepsy classification model based on the coordinated facial and posture features is provided in the server, and the epilepsy classification model based on the coordinated facial and posture features is obtained by using the epilepsy classification model construction method based on the coordinated facial and posture features of the above-mentioned embodiment; the camera is used to monitor the target person and obtain video data; the server is used to perform epilepsy recognition based on the video data using the epilepsy classification model based on the coordinated facial and posture features to obtain recognition results; and the terminal is used to receive and display the recognition results.

[0081] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for constructing an epilepsy classification model that cooperates with facial and posture features is implemented.

[0082] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0083] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0084] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0085] In an exemplary embodiment, a computer program product is provided, which, when executed by a processor, implements the steps in the above-mentioned method embodiments.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0087] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0088] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0089] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing an epilepsy classification model based on the synergy of facial and posture features, characterized in that: include: Build feature extraction module; The feature extraction module is used to extract the skeleton feature sequence and facial feature sequence of the target person in the video data; Constructing a FAGCN network model; the FAGCN network model is an adaptive graph convolutional network model that integrates facial and posture features, and the FAGCN network model includes: 10 FAGCN blocks connected in sequence and 1 fully connected layer, wherein the first, fifth, and eighth FAGCN blocks are all first FAGCN blocks, the first FAGCN block is connected to the fifth FAGCN block through a Reshape function and a first convolution and tiling module, the fifth FAGCN block is connected to the eighth FAGCN block through a Reshape function and a first convolution and tiling module, and the remaining seven FAGCN blocks are all second FAGCN blocks; the first FAGCN block is used to process a skeleton point feature sequence and a facial feature sequence, and the second FAGCN block is used to process a skeleton point feature sequence; Construct an epilepsy dataset; Based on the epilepsy dataset, the FAGCN network model is trained to obtain a trained FAGCN network model; Connect the feature extraction module and the trained FAGCN network model to obtain an epilepsy classification model that combines facial and posture features. The first FAGCN block includes: a first graph convolution module, a second convolution and tiling module, a connection module, a first time convolution module and a first post-processing module; the first graph convolution module and the second convolution and tiling module are both connected to the connection module, the connection module is connected to the first time convolution module, and the first time convolution module is connected to the first post-processing module; The first graph convolution module is used to extract deep features from the target skeleton point feature sequence to obtain a spatial target skeleton point deep feature sequence; the target skeleton point feature sequence is a skeleton point feature sequence or a skeleton point deep feature sequence output by a second FAGCN block located before the first FAGCN block; The second convolution and tiling module is used to extract deep features from the target facial feature sequence to obtain a spatial target facial deep feature sequence; the target facial feature sequence is a facial feature sequence or a facial deep feature sequence output by a first FAGCN block located before the first FAGCN block; The connection module is used to connect the target skeleton point depth feature sequence in space and the target facial depth feature sequence in space to obtain a facial-posture heterogeneous graph structure sequence; The first time convolution module is used to extract temporal features from the face-posture heterogeneous graph structure sequence to obtain a face-posture depth feature sequence; The first post-processing module is used to separate the face-posture depth feature sequence to obtain a skeleton point depth feature sequence and a facial depth feature sequence; The second FAGCN block includes a second graph convolution module and a second temporal convolution module connected in sequence; The second graph convolution module is used to extract deep features from the target skeleton point feature sequence to obtain a spatial target skeleton point deep feature sequence; the target skeleton point feature sequence is a skeleton point deep feature sequence output by the first FAGCN block or the second FAGCN block located before the second FAGCN block; The second time convolution module is used to extract time features from the spatial target skeleton point depth feature sequence to obtain the skeleton point depth feature sequence.

2. The method for constructing an epilepsy classification model based on the coordination of facial and posture features according to claim 1, characterized in that: In terms of extracting the skeleton point feature sequence and facial feature sequence of the target person in the video data, the feature extraction module is specifically used to: A human posture estimation algorithm is used to extract the skeleton point information of each person in the n-th frame of the video data; n = 1, 2, ..., N, where N is the number of frames of the video data; Calculate the Euclidean distance between the skeleton point information of each person in the nth frame video image and the skeleton point information of the target person in the n-1th frame video image; Determine the skeleton point information of the person with the smallest Euclidean distance in the n-th frame of video image as the skeleton point information of the target person in the n-th frame of video image; Based on the skeleton point information of the target person in the n-th frame video image, the human body pose is estimated to obtain the skeleton point features of the target person in the n-th frame video image; Positioning the face of the target person based on the skeleton point information of the target person in the n-th frame video image, and obtaining the facial contour of the target person in the n-th frame video image; Extracting facial features of the target person in the n-th frame of video image based on the facial contour of the target person in the n-th frame of video image; Based on the skeleton point features and facial features of the target person in each frame of video image in the video data, a skeleton point feature sequence and a facial feature sequence of the target person are constructed.

3. The method for constructing an epilepsy classification model based on the coordination of facial and posture features according to claim 2, characterized in that: The formula for extracting the facial features of the target person in the n-th frame video image based on the facial contour of the target person in the n-th frame video image is: Among them, FAX and FAY represent the cropping coordinates of the facial contour along the horizontal and vertical dimensions, respectively, x1, x2 and x i are the horizontal coordinates of the first, second and i-th bone points on the target person’s face, y1, y2, y i are the ordinates of the first, second and i-th bone points on the target person’s face, V h and V w The height and width of the video data respectively.

4. The method for constructing an epilepsy classification model based on the coordination of facial and posture features according to claim 1, characterized in that: The constructing of the epilepsy dataset specifically includes: Obtain a video dataset of epilepsy patients; Filter the video data in the video data set, remove the video data in which the target person is occluded, and obtain a filtered video data set; A feature extraction module is used to extract the skeleton feature sequence and facial feature sequence of the target person in each video data in the filtered video data set; The skeleton feature sequence and facial feature sequence of the target person in each video data are marked as whether they are experiencing an epileptic seizure, so as to construct the epilepsy dataset.

5. An epilepsy monitoring device, characterized in that: The device includes: a camera, a server and a terminal; The camera is connected to the server, and the server is connected to the terminal. The server is provided with an epilepsy classification model coordinated by facial and posture features, and the epilepsy classification model coordinated by facial and posture features is obtained by using the method for constructing an epilepsy classification model coordinated by facial and posture features according to any one of claims 1 to 4; The camera is used to monitor the target person and obtain video data; The server is configured to perform epilepsy recognition based on the video data using the epilepsy classification model coordinated with facial and posture features to obtain a recognition result; The terminal is used to receive and display the recognition result.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing an epilepsy classification model based on the coordination of facial and posture features according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing an epilepsy classification model by coordinating facial and posture features according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for constructing an epilepsy classification model by coordinating facial and posture features according to any one of claims 1 to 4 is implemented.

Images