A method and device for intelligently determining epilepsy classification based on video

Abstract

The present application discloses a method and device for intelligently determining epilepsy classification based on video. The method comprises: acquiring an image sequence; extracting a region of interest for each image in the image sequence; extracting facial features from a facial region image sequence; extracting upper body image features from an upper body image sequence; extracting lower body image features from a lower body image sequence; acquiring virtual portrait image features; acquiring a trained epilepsy recognition neural network; fusing facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fused features; and inputting the fused features into the epilepsy recognition neural network to obtain epilepsy recognition results. On the one hand, the present application only uses video image information for recognition, omitting EEG signals, thereby making it possible to perform recognition without relying on EEG signals. On the other hand, recognition through multiple modal information can make the recognition results more accurate.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for intelligently determining epilepsy classification based on video and a device for intelligently determining epilepsy classification based on video. Background Art

[0002] Epilepsy is one of the most common neurological diseases, with a prevalence of approximately 0.8%. Approximately 10 million people in my country currently suffer from epilepsy. The incidence of epilepsy follows a U-shaped curve, with childhood being the first peak period. For children, recurrent epileptic seizures can lead to intellectual and cognitive developmental delays, even regression, and behavioral problems. Early and accurate diagnosis and treatment are crucial. Accurately determining the seizure type (generalized or focal) is particularly important for guiding treatment.

[0003] The diagnosis and treatment of epilepsy varies significantly across regions and medical institutions. Some institutions lack essential equipment like video EEG, making it difficult to accurately categorize epileptic seizures. Many patients with epilepsy may not receive appropriate treatment or even receive inappropriate medications, which can worsen their seizures. Currently, video EEG assessments rely primarily on manual evaluation by experienced clinicians. This process requires a high level of professional expertise, is subjective, and is inefficient, making it difficult to implement widely in primary care hospitals.

[0004] However, existing technologies usually identify epilepsy by combining EEG signals with videos. However, in reality, epileptic seizures occur at uncertain times, which means that EEG signals cannot necessarily be collected every time an epileptic seizure occurs.

[0005] Therefore, it is desired to have a technical solution to overcome or at least alleviate at least one of the above-mentioned deficiencies of the prior art.

[0006] Application Contents

[0007] The purpose of this application is to provide a method for intelligently determining epilepsy typing based on video to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0008] To achieve the above objectives, the present application provides a method for intelligently determining epilepsy classification based on video, the method comprising:

[0009] Get image sequence;

[0010] Extracting a region of interest from each image in the image sequence to obtain a region of interest image, wherein the region of interest image includes a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence;

[0011] extracting facial features from the facial region image sequence;

[0012] extracting upper body image features from the upper body image sequence;

[0013] extracting lower body image features from the lower body image sequence;

[0014] Obtaining virtual portrait image features;

[0015] Obtaining a trained epilepsy recognition neural network;

[0016] fusing the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fused features;

[0017] The fused features are input into the epilepsy recognition neural network to obtain an epilepsy recognition result.

[0018] Optionally, before acquiring the image sequence, the method for intelligently determining epilepsy typing based on video further comprises:

[0019] Get epileptic seizure video stream;

[0020] Each frame of the epileptic seizure video stream is acquired separately to form an image sequence.

[0021] Optionally, obtaining the virtual portrait image features includes:

[0022] Extracting skeleton point information from each image in the image sequence according to the image sequence;

[0023] Generate a virtual bone image corresponding to each image according to the information of each bone point;

[0024] The image features of the virtual skeleton image are extracted to obtain the image features of the virtual human.

[0025] Optionally, the region of interest extraction for each image in the image sequence is performed through a YOLO deep learning network.

[0026] Optionally, the method for intelligently determining epilepsy typing based on video further comprises:

[0027] Generate a matrix corresponding to facial expressions and upper body movements, which is called the first matrix;

[0028] Generate a matrix corresponding to facial expressions and lower body movements, which is called the second matrix;

[0029] Vectorize the first matrix to obtain a first matrix vector;

[0030] Vectorize the second matrix to obtain a second matrix vector;

[0031] The fusing of the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain the fused features further includes:

[0032] The facial features, upper body image features, lower body image features, virtual portrait image features, the first matrix vector and the second matrix vector are fused to obtain fused features.

[0033] Optionally, the generating of a matrix corresponding to facial expressions and upper body movements, which is referred to as a first matrix, includes:

[0034] Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups;

[0035] The following processing is performed on each group of whole-body image sequences:

[0036] The facial features and upper body image features of each whole-body image in each group of whole-body image sequence groups are extracted. If the whole-body image is a whole-body image of a generalized epileptic seizure, the matrix point is judged to be 1; otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and upper body movements of the training set.

[0037] Optionally, the generating of a matrix corresponding to facial expressions and lower body movements, which is referred to as a second matrix, includes:

[0038] Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups;

[0039] The following processing is performed for each full-body image:

[0040] The facial features and lower body image features of each full-body image are extracted. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1. If not, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and lower body movements of the training set.

[0041] The present application also provides a device for intelligently determining epilepsy classification based on video, the device comprising:

[0042] An image sequence acquisition module, wherein the image sequence acquisition module is used to acquire an image sequence;

[0043] a region of interest extraction and acquisition module, the region of interest extraction and acquisition module being configured to extract a region of interest from each image in the image sequence, thereby acquiring a region of interest image, the region of interest image comprising a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence;

[0044] A facial feature acquisition module, configured to extract facial features from the facial region image sequence;

[0045] an upper body image feature acquisition module, the upper body image feature acquisition module being used to extract upper body image features of the upper body image sequence;

[0046] a lower body image feature acquisition module, configured to extract lower body image features from the lower body image sequence;

[0047] A virtual portrait image feature acquisition module, wherein the virtual portrait image feature acquisition module is used to acquire virtual portrait image features;

[0048] A neural network acquisition module, wherein the neural network acquisition module is used to acquire a trained epilepsy recognition neural network;

[0049] A fusion feature acquisition module, configured to fuse the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fusion features;

[0050] An epilepsy recognition result acquisition module is used to input the fusion feature into the epilepsy recognition neural network to obtain an epilepsy recognition result.

[0051] The method of intelligently determining epilepsy classification based on video in the present application uses information from multiple modalities (face, upper body, lower body, and virtual portrait features) for identification. On the one hand, only video image information is used for identification, eliminating EEG signals, making it possible to perform identification without relying on EEG signals. On the other hand, identification through multiple modal information can make the identification results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 4 is a flow chart of a method for intelligently determining epilepsy classification based on video according to an embodiment of the present application.

[0053] Figure 2 It is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the implementation of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below in conjunction with the drawings in the embodiments of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be understood as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0055] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as limiting the scope of protection of this application.

[0056] Figure 1 4 is a flow chart of a method for intelligently determining epilepsy classification based on video according to an embodiment of the present application.

[0057] like Figure 1 The method for intelligently determining epilepsy classification based on video includes:

[0058] Get image sequence;

[0059] Extracting a region of interest from each image in the image sequence to obtain a region of interest image, wherein the region of interest image includes a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence;

[0060] extracting facial features from the facial region image sequence;

[0061] extracting upper body image features from the upper body image sequence;

[0062] extracting lower body image features from the lower body image sequence;

[0063] Obtaining virtual portrait image features;

[0064] Obtaining a trained epilepsy recognition neural network;

[0065] fusing the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fused features;

[0066] The fused features are input into the epilepsy recognition neural network to obtain an epilepsy recognition result.

[0067] The method of intelligently determining epilepsy classification based on video in the present application uses information from multiple modalities (face, upper body, lower body, and virtual portrait features) for identification. On the one hand, only video image information is used for identification, eliminating EEG signals, making it possible to perform identification without relying on EEG signals. On the other hand, identification through multiple modal information can make the identification results more accurate.

[0068] In this embodiment, before acquiring the image sequence, the method for intelligently determining epilepsy typing based on video further includes:

[0069] Get epileptic seizure video stream;

[0070] Each frame of the epileptic seizure video stream is acquired separately to form an image sequence.

[0071] In this embodiment, obtaining the virtual portrait image features includes:

[0072] Extracting skeleton point information from each image in the image sequence according to the image sequence;

[0073] Generate a virtual bone image corresponding to each image according to the information of each bone point;

[0074] The image features of the virtual skeleton image are extracted to obtain the image features of the virtual human.

[0075] In this embodiment, the skeleton point information can be selected according to the accuracy of the virtual portrait. In one embodiment, the skeleton point information includes 18 joint points of the human body, including the neck, nose, left eye, right eye, left ear, right ear, right shoulder, left shoulder, right hip, left hip, right hand joint, left hand joint, right knee, left knee, right wrist, left wrist, right ankle and left ankle.

[0076] In other embodiments, it can also be 20 points including the center point of human skeleton, the center point of spine, the center point of both shoulders, the center point of head, the center point of left shoulder, the center point of left elbow, the center point of left wrist, the center point of left hand, the center point of right shoulder, the center point of right elbow, the center point of right wrist, the center point of right hand, the center point of left hip, the center point of left knee, the center point of left ankle, the center point of left foot, the center point of right hip, the center point of right knee, the center point of right ankle, and the center point of right foot.

[0077] In this embodiment, the extraction of the region of interest for each image in the image sequence is performed through a YOLO deep learning network.

[0078] In this embodiment, the method for intelligently determining epilepsy typing based on video further includes:

[0079] Generate a matrix corresponding to facial expressions and upper body movements, which is called the first matrix;

[0080] Generate a matrix corresponding to facial expressions and lower body movements, which is called the second matrix;

[0081] Vectorize the first matrix to obtain a first matrix vector;

[0082] Vectorize the second matrix to obtain a second matrix vector;

[0083] The fusing of the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain the fused features further includes:

[0084] The facial features, upper body image features, lower body image features, virtual portrait image features, the first matrix vector and the second matrix vector are fused to obtain fused features.

[0085] In this embodiment, the generated facial expression and upper body movement correspondence matrix, referred to as the first matrix, includes:

[0086] Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups;

[0087] The following processing is performed on each group of whole-body image sequences:

[0088] The facial features and upper body image features of each whole-body image in each group of whole-body image sequence groups are extracted. If the whole-body image is a whole-body image of a generalized epileptic seizure, the matrix point is judged to be 1; otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and upper body movements of the training set.

[0089] Take the following matrix as an example:

[0090] The vertical rows can represent different facial features, and the horizontal rows can represent different upper body image features. The matrix points inside represent the epileptic seizure conditions corresponding to the facial features and upper body image features.

[0091] In this way, a prior interaction matrix can be provided to the neural network of the present application, thereby increasing the feature meaning.

[0092] In this embodiment, the generated matrix corresponding to facial expressions and lower body movements, referred to as a first matrix, includes:

[0093] Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups;

[0094] The following processing is performed for each full-body image:

[0095] The facial features and lower body image features of each full-body image are extracted. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1. If not, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and lower body movements of the training set.

[0096] The present application also provides a device for intelligently determining epilepsy classification based on video, the device comprising an image sequence acquisition module, a region of interest extraction acquisition module, a facial feature acquisition module, an upper body image feature acquisition module, a lower body image feature acquisition module, a virtual portrait image feature acquisition module, a neural network acquisition module, a fusion feature acquisition module, and an epilepsy recognition result acquisition module, wherein:

[0097] The image sequence acquisition module is used to acquire image sequences;

[0098] The region of interest extraction acquisition tool is used to extract the region of interest from each image in the image sequence, thereby acquiring region of interest images, wherein the region of interest images include the patient's facial region, the patient's upper body region, and the patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence;

[0099] The facial feature acquisition module is used to extract facial features of the facial region image sequence;

[0100] The upper body image feature acquisition module is used to extract the upper body image features of the upper body image sequence;

[0101] The lower body image feature acquisition module is used to extract the lower body image features of the lower body image sequence;

[0102] The virtual portrait image feature acquisition module is used to acquire virtual portrait image features;

[0103] The neural network acquisition module is used to acquire the trained epilepsy recognition neural network;

[0104] The fusion feature acquisition module is used to fuse the facial features, upper body image features, lower body image features and virtual portrait image features to obtain fusion features;

[0105] The epilepsy recognition result acquisition module is used to input the fusion features into the epilepsy recognition neural network, thereby acquiring an epilepsy recognition result.

[0106] It should be noted that the above explanations of the method embodiment are also applicable to the device of this embodiment and will not be repeated here.

[0107] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the above method for intelligently determining epilepsy typing based on video is implemented.

[0108] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above method for intelligently determining epilepsy typing based on video.

[0109] Figure 2 This is an exemplary structural diagram of an electronic device capable of implementing the method for intelligently determining epilepsy typing based on video provided in accordance with one embodiment of the present application.

[0110] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processing unit 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processing unit 503, the memory 504, and the output interface 505 are interconnected via a bus 507. The input device 501 and the output device 506 are connected to the bus 507 via the input interface 502 and the output interface 505, respectively, and are then connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside and transmits the input information to the central processing unit 503 via the input interface 502; the central processing unit 503 processes the input information based on the computer-executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 via the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.

[0111] That is to say, Figure 2 The electronic device shown may also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination of the computer executable instructions when executing the computer executable instructions. Figure 1The method described is for intelligently determining epilepsy classification based on video.

[0112] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program code; and one or more processors 503 configured to run the executable program code stored in the memory 504 to execute the method for intelligently determining epilepsy typing based on video in the above embodiment.

[0113] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0115] Computer-readable media include permanent and non-permanent, removable and non-removable media, and media can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices recited in the device claims can also be implemented by one unit or the entire device through software or hardware.

[0118] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and a part of the module, program segment or code includes one or more executable instructions for realizing the prescribed logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or overall flow chart can be implemented using a dedicated hardware-based system that performs the prescribed function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices recited in the device claims can also be implemented by one unit or the entire device through software or hardware.

[0121] Although the present application is disclosed as above with reference to preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0122] Finally, it should be pointed out that the above embodiments are intended only to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they may modify the technical solutions described in the above embodiments or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intelligently determining epilepsy typing based on video, characterized in that: The method for intelligently determining epilepsy typing based on video includes: Get image sequence; Extracting a region of interest from each image in the image sequence to obtain a region of interest image, wherein the region of interest image includes a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence; extracting facial features from the facial region image sequence; extracting upper body image features from the upper body image sequence; extracting lower body image features from the lower body image sequence; Obtaining virtual portrait image features; Obtaining a trained epilepsy recognition neural network; fusing the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fused features; Inputting the fused features into the epilepsy recognition neural network to obtain an epilepsy recognition result; Before acquiring the image sequence, the method for intelligently determining epilepsy typing based on video further comprises: Get epileptic seizure video stream; Each frame image of the epileptic seizure video stream is acquired separately to form an image sequence; The acquiring of the virtual portrait image features includes: Extracting skeleton point information from each image in the image sequence according to the image sequence; Generate a virtual bone image corresponding to each image according to the information of each bone point; Extracting image features of the virtual skeleton image, thereby obtaining image features of the virtual human; The region of interest of each image in the image sequence is extracted by a YOLO deep learning network; The method for intelligently determining epilepsy typing based on video further comprises: Generate a matrix corresponding to facial expressions and upper body movements, which is called the first matrix; Generate a matrix corresponding to facial expressions and lower body movements, which is called the second matrix; Vectorize the first matrix to obtain a first matrix vector; Vectorize the second matrix to obtain a second matrix vector; The fusing of the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain the fused features further includes: Fusing the facial features, upper body image features, lower body image features, virtual portrait image features, the first matrix vector, and the second matrix vector to obtain a fused feature; The matrix corresponding to facial expressions and upper body movements is generated, which is called the first matrix and includes: Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups; Each whole-body image sequence group is processed as follows: Extract the facial features and upper body image features of each full-body image in each full-body image sequence group. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1; otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and upper body movements of the training set; The generated matrix corresponding to facial expressions and lower body movements is called the second matrix and includes: Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups; The following processing is performed for each full-body image: The facial features and lower body image features of each full-body image are extracted. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1. If not, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and lower body movements of the training set.

2. A device for intelligently determining epilepsy types based on video, characterized in that: The device for intelligently determining epilepsy typing based on video comprises: An image sequence acquisition module, wherein the image sequence acquisition module is used to acquire an image sequence; a region of interest extraction and acquisition module, the region of interest extraction and acquisition module being configured to extract a region of interest from each image in the image sequence, thereby acquiring a region of interest image, the region of interest image comprising a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence; A facial feature acquisition module, configured to extract facial features from the facial region image sequence; an upper body image feature acquisition module, the upper body image feature acquisition module being used to extract upper body image features of the upper body image sequence; a lower body image feature acquisition module, configured to extract lower body image features from the lower body image sequence; A virtual portrait image feature acquisition module, wherein the virtual portrait image feature acquisition module is used to acquire virtual portrait image features; A neural network acquisition module, wherein the neural network acquisition module is used to acquire a trained epilepsy recognition neural network; A fusion feature acquisition module, configured to fuse the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fusion features; an epilepsy recognition result acquisition module, configured to input the fusion feature into the epilepsy recognition neural network, thereby acquiring an epilepsy recognition result; Before acquiring the image sequence, the method for intelligently determining epilepsy typing based on video further comprises: Get epileptic seizure video stream; Each frame image of the epileptic seizure video stream is acquired separately to form an image sequence; The acquiring of the virtual portrait image features includes: Extracting skeleton point information from each image in the image sequence according to the image sequence; Generate a virtual bone image corresponding to each image according to the information of each bone point; Extracting image features of the virtual skeleton image, thereby obtaining image features of the virtual human; The region of interest of each image in the image sequence is extracted by a YOLO deep learning network; The method for intelligently determining epilepsy typing based on video further comprises: Generate a matrix corresponding to facial expressions and upper body movements, which is called the first matrix; Generate a matrix corresponding to facial expressions and lower body movements, which is called the second matrix; Vectorize the first matrix to obtain a first matrix vector; Vectorize the second matrix to obtain a second matrix vector; The fusing of the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain the fused features further includes: Fusing the facial features, upper body image features, lower body image features, virtual portrait image features, the first matrix vector, and the second matrix vector to obtain a fused feature; The matrix corresponding to facial expressions and upper body movements is generated, which is called the first matrix and includes: Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups; Each whole-body image sequence group is processed as follows: Extract the facial features and upper body image features of each full-body image in each full-body image sequence group. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1; otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and upper body movements of the training set; The generated matrix corresponding to facial expressions and lower body movements is called the second matrix and includes: Acquire a training set, wherein the training set includes multiple groups of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest are focal epileptic seizure image sequence groups; The following processing is performed for each full-body image: The facial features and lower body image features of each full-body image are extracted. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1. If not, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and lower body movements of the training set.

Images