Method and device for intelligently judging epilepsy typing according to video

Abstract

The invention discloses a method and a device for intelligently judging epilepsy typing according to a video. The method comprises the following steps: acquiring an image sequence; performing region-of-interest extraction on each image in the image sequence; extracting facial features of the facial region image sequence; extracting upper body image features of the upper body image sequence; extracting lower body image features of the lower body image sequence; obtaining virtual portrait image features; acquiring a trained epilepsy recognition neural network; fusing the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain fused features; and inputting the fusion features into the epilepsy recognition neural network to obtain an epilepsy recognition result. According to the method, on one hand, only the video image information is used for identification, electroencephalogram signals are omitted, so that identification can be carried out without depending on the electroencephalogram signals, and on the other hand, identification is carried out through multiple pieces of modal information, so that the identification result is more accurate.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for intelligently determining epilepsy classification based on video and a device for intelligently determining epilepsy classification based on video. Background Art

[0002] Epilepsy is one of the most common diseases in neurology, with a prevalence of about 0.8%. There are about 10 million epilepsy patients in my country. The incidence curve of epilepsy is "U"-shaped, and childhood is the first peak of epilepsy. For children, repeated epileptic seizures can lead to intellectual and cognitive development retardation or even regression and behavioral problems. Early correct diagnosis and treatment are very important. Correctly determining the type of epileptic seizures (generalized and focal) is particularly important for guiding treatment.

[0003] The diagnosis and treatment level of epilepsy varies greatly between different regions and medical institutions. Some medical institutions lack necessary equipment such as video EEG, making it difficult to correctly classify epileptic seizures. Many epilepsy patients cannot receive correct treatment, and even use inappropriate drugs, which worsens the seizures. At present, the judgment of video EEG mainly relies on manual judgment by experienced clinical personnel. This process requires a high level of professional knowledge of clinical personnel, is subjective and inefficient, and cannot be widely carried out in grassroots hospitals.

[0004] However, existing technologies usually identify epilepsy by combining EEG signals with videos. However, in reality, epileptic seizures occur at uncertain times, which means that EEG signals may not be collected every time an epileptic seizure occurs.

[0005] Therefore, it is desired to have a technical solution to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0006] Application Contents

[0007] The purpose of the present application is to provide a method for intelligently determining epilepsy classification based on video to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0008] To achieve the above-mentioned purpose, the present application provides a method for intelligently determining epilepsy classification based on video, and the method for intelligently determining epilepsy classification based on video includes:

[0009] Get image sequence;

[0010] Extracting a region of interest from each image in the image sequence, thereby obtaining a region of interest image, wherein the region of interest image includes a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence;

[0011] Extracting facial features of the facial region image sequence;

[0012] extracting upper body image features of the upper body image sequence;

[0013] extracting lower body image features of the lower body image sequence;

[0014] Acquire virtual portrait image features;

[0015] Obtaining a trained epilepsy recognition neural network;

[0016] Fusing the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fused features;

[0017] The fused features are input into the epilepsy recognition neural network to obtain an epilepsy recognition result.

[0018] Optionally, before acquiring the image sequence, the method for intelligently determining epilepsy typing based on video further comprises:

[0019] Get epileptic seizure video stream;

[0020] Each frame image of the epileptic seizure video stream is acquired separately to form an image sequence.

[0021] Optionally, obtaining the virtual portrait image features includes:

[0022] Extracting skeleton point information in each image in the image sequence according to the image sequence;

[0023] Generate a virtual bone image corresponding to each image according to the information of each bone point;

[0024] The image features of the virtual skeleton image are extracted to obtain the image features of the virtual human.

[0025] Optionally, the extraction of the region of interest for each image in the image sequence is performed through a YOLO deep learning network.

[0026] Optionally, the method for intelligently determining epilepsy typing based on video further comprises:

[0027] Generate a matrix corresponding to facial expressions and upper body movements, which is called the first matrix;

[0028] Generate a matrix corresponding to facial expressions and lower body movements, which is called the second matrix;

[0029] Vectorize the first matrix to obtain a first matrix vector;

[0030] Vectorize the second matrix to obtain a second matrix vector;

[0031] The step of fusing the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain the fused features further includes:

[0032] The facial features, upper body image features, lower body image features, virtual portrait image features, the first matrix vector and the second matrix vector are fused to obtain fused features.

[0033] Optionally, the generating of the facial expression and upper body movement correspondence matrix, which is called the first matrix, includes:

[0034] Acquire a training set, wherein the training set includes a plurality of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest of which are focal epileptic seizure image sequence groups;

[0035] The following processing is performed for each group of whole body image sequences:

[0036] The facial features and upper body image features of each whole-body image in each group of whole-body image sequence groups are extracted. If the whole-body image is a whole-body image of a generalized epileptic seizure, the matrix point is judged to be 1, otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and upper body movements of the training set.

[0037] Optionally, the generating of the facial expression and lower body movement correspondence matrix, which is called the second matrix, includes:

[0038] Acquire a training set, wherein the training set includes a plurality of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest of which are focal epileptic seizure image sequence groups;

[0039] The following processing is performed for each full-body image:

[0040] The facial features and lower body image features of each full-body image are extracted. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1, otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and lower body movements of the training set.

[0041] The present application also provides a device for intelligently determining epilepsy classification based on video, the device for intelligently determining epilepsy classification based on video comprising:

[0042] An image sequence acquisition module, wherein the image sequence acquisition module is used to acquire an image sequence;

[0043] An area of ​​interest extraction and acquisition module, the area of ​​interest extraction and acquisition module is used to extract the area of ​​interest from each image in the image sequence, so as to acquire an area of ​​interest image, the area of ​​interest image includes a patient's facial area, a patient's upper body area, and a patient's lower body area, each of the patient's facial areas constitutes a facial area image sequence, each of the patient's upper body areas constitutes an upper body image sequence, and each of the patient's lower body areas constitutes a lower body image sequence;

[0044] A facial feature acquisition module, the facial feature acquisition module is used to extract facial features of the facial region image sequence;

[0045] An upper body image feature acquisition module, the upper body image feature acquisition module is used to extract upper body image features of the upper body image sequence;

[0046] A lower body image feature acquisition module, the lower body image feature acquisition module is used to extract lower body image features of the lower body image sequence;

[0047] A virtual portrait image feature acquisition module, wherein the virtual portrait image feature acquisition module is used to acquire virtual portrait image features;

[0048] A neural network acquisition module, wherein the neural network acquisition module is used to acquire a trained epilepsy recognition neural network;

[0049] A fusion feature acquisition module, the fusion feature acquisition module is used to fuse the facial features, upper body image features, lower body image features and virtual portrait image features to obtain fusion features;

[0050] An epilepsy recognition result acquisition module is used to input the fusion feature into the epilepsy recognition neural network to obtain an epilepsy recognition result.

[0051] The method of intelligently determining epilepsy classification based on video in the present application uses information from multiple modalities (face, upper body, lower body, and virtual portrait features) for identification. On the one hand, only video image information is used for identification, eliminating EEG signals, thereby making it possible to perform identification without relying on EEG signals. On the other hand, identification through multiple modal information can make the identification result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of a method for intelligently determining epilepsy classification based on video according to an embodiment of the present application.

[0053] Figure 2 It is a schematic diagram of the structure of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0055] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the scope of protection of the present application.

[0056] Figure 1 It is a flowchart of a method for intelligently determining epilepsy classification based on video according to an embodiment of the present application.

[0057] like Figure 1 The method for intelligently determining epilepsy classification based on video includes:

[0058] Get image sequence;

[0059] Extracting a region of interest from each image in the image sequence, thereby obtaining a region of interest image, wherein the region of interest image includes a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence;

[0060] Extracting facial features of the facial region image sequence;

[0061] extracting upper body image features of the upper body image sequence;

[0062] extracting lower body image features of the lower body image sequence;

[0063] Acquire virtual portrait image features;

[0064] Obtaining a trained epilepsy recognition neural network;

[0065] Fusing the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fused features;

[0066] The fused features are input into the epilepsy recognition neural network to obtain an epilepsy recognition result.

[0067] The method of intelligently determining epilepsy classification based on video in the present application uses information from multiple modalities (face, upper body, lower body, and virtual portrait features) for identification. On the one hand, only video image information is used for identification, eliminating EEG signals, thereby making it possible to perform identification without relying on EEG signals. On the other hand, identification through multiple modal information can make the identification result more accurate.

[0068] In this embodiment, before acquiring the image sequence, the method for intelligently determining epilepsy typing based on video further includes:

[0069] Get epileptic seizure video stream;

[0070] Each frame image of the epileptic seizure video stream is acquired separately to form an image sequence.

[0071] In this embodiment, obtaining the virtual portrait image features includes:

[0072] Extracting skeleton point information in each image in the image sequence according to the image sequence;

[0073] Generate a virtual bone image corresponding to each image according to the information of each bone point;

[0074] The image features of the virtual skeleton image are extracted to obtain the image features of the virtual human.

[0075] In this embodiment, the skeleton point information can be selected according to the accuracy of the virtual portrait. In one embodiment, the skeleton point information includes 18 joint points of the human body, including neck, nose, left eye, right eye, left ear, right ear, right shoulder, left shoulder, right hip, left hip, right hand joint, left hand joint, right knee, left knee, right wrist, left wrist, right ankle and left ankle.

[0076] In other embodiments, it can also be 20 points including the center point of human skeleton, the center point of spine, the center point of both shoulders, the center point of head, the center point of left shoulder, the center point of left elbow, the center point of left wrist, the center point of left hand, the center point of right shoulder, the center point of right elbow, the center point of right wrist, the center point of right hand, the center point of left hip, the center point of left knee, the center point of left ankle, the center point of left foot, the center point of right hip, the center point of right knee, the center point of right ankle, and the center point of right foot.

[0077] In this embodiment, the extraction of the region of interest for each image in the image sequence is performed through a YOLO deep learning network.

[0078] In this embodiment, the method for intelligently determining epilepsy typing based on video further includes:

[0079] Generate a matrix corresponding to facial expressions and upper body movements, which is called the first matrix;

[0080] Generate a matrix corresponding to facial expressions and lower body movements, which is called the second matrix;

[0081] Vectorize the first matrix to obtain a first matrix vector;

[0082] Vectorize the second matrix to obtain a second matrix vector;

[0083] The step of fusing the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain the fused features further includes:

[0084] The facial features, upper body image features, lower body image features, virtual portrait image features, the first matrix vector and the second matrix vector are fused to obtain fused features.

[0085] In this embodiment, the facial expression and upper body movement correspondence matrix is ​​generated, and the matrix is ​​called the first matrix, and includes:

[0086] Acquire a training set, wherein the training set includes a plurality of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest of which are focal epileptic seizure image sequence groups;

[0087] The following processing is performed for each group of whole body image sequences:

[0088] The facial features and upper body image features of each whole-body image in each group of whole-body image sequence groups are extracted. If the whole-body image is a whole-body image of a generalized epileptic seizure, the matrix point is judged to be 1, otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and upper body movements of the training set.

[0089] Take the following matrix as an example:

[0090] The vertical rows can represent different facial features, the horizontal rows can represent different upper body image features, and the matrix points inside represent the epileptic seizure conditions corresponding to the facial features and the upper body image features.

[0091] In this way, a prior interaction matrix can be provided to the neural network of the present application, thereby increasing the feature meaning.

[0092] In this embodiment, the facial expression and lower body movement correspondence matrix is ​​generated, and the matrix is ​​called the first matrix, and includes:

[0093] Acquire a training set, wherein the training set includes a plurality of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest of which are focal epileptic seizure image sequence groups;

[0094] The following processing is performed for each full-body image:

[0095] The facial features and lower body image features of each full-body image are extracted. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1, otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and lower body movements of the training set.

[0096] The present application also provides a device for intelligently determining epilepsy classification based on video, the device for intelligently determining epilepsy classification based on video includes an image sequence acquisition module, an area of ​​interest extraction acquisition module, a facial feature acquisition module, an upper body image feature acquisition module, a lower body image feature acquisition module, a virtual portrait image feature acquisition module, a neural network acquisition module, a fusion feature acquisition module and an epilepsy recognition result acquisition module, wherein:

[0097] The image sequence acquisition module is used to acquire the image sequence;

[0098] The region of interest extraction acquisition tool is used to extract the region of interest from each image in the image sequence, so as to obtain the region of interest image, wherein the region of interest image includes the patient's facial region, the patient's upper body region and the patient's lower body region, each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence;

[0099] The facial feature acquisition module is used to extract facial features of the facial region image sequence;

[0100] The upper body image feature acquisition module is used to extract the upper body image features of the upper body image sequence;

[0101] The lower body image feature acquisition module is used to extract the lower body image features of the lower body image sequence;

[0102] The virtual portrait image feature acquisition module is used to acquire the virtual portrait image features;

[0103] The neural network acquisition module is used to acquire the trained epilepsy recognition neural network;

[0104] The fusion feature acquisition module is used to fuse the facial features, upper body image features, lower body image features and virtual portrait image features to obtain fusion features;

[0105] The epilepsy recognition result acquisition module is used to input the fusion feature into the epilepsy recognition neural network, so as to obtain the epilepsy recognition result.

[0106] It should be noted that the above explanations of the method embodiment are also applicable to the device of this embodiment and will not be repeated here.

[0107] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the method for intelligently determining epilepsy classification based on video is implemented as described above.

[0108] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above method for intelligently determining epilepsy classification based on video.

[0109] Figure 2 It is an exemplary structural diagram of an electronic device capable of implementing the method for intelligently determining epilepsy classification based on video provided according to one embodiment of the present application.

[0110] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.

[0111] That is to say, Figure 2 The electronic device shown can also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination when executing the computer executable instructions. Figure 1A method for intelligently determining epilepsy classification based on video is described.

[0112] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program codes; and one or more processors 503 configured to run the executable program codes stored in the memory 504 to execute the method for intelligently determining epilepsy classification based on video in the above embodiment.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0117] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or the total flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0120] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.

[0121] Although the present application is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0122] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for intelligently determining epilepsy typing based on video, characterized in that: The method for intelligently determining epilepsy typing based on video comprises: Get image sequence; Extracting a region of interest from each image in the image sequence, thereby obtaining a region of interest image, wherein the region of interest image includes a patient's facial region, a patient's upper body region, and a patient's lower body region, wherein each of the patient's facial regions constitutes a facial region image sequence, each of the patient's upper body regions constitutes an upper body image sequence, and each of the patient's lower body regions constitutes a lower body image sequence; Extracting facial features of the facial region image sequence; Extracting upper body image features of the upper body image sequence; extracting lower body image features of the lower body image sequence; Acquire virtual portrait image features; Obtaining a trained epilepsy recognition neural network; Fusing the facial features, upper body image features, lower body image features, and virtual portrait image features to obtain fused features; The fused features are input into the epilepsy recognition neural network to obtain an epilepsy recognition result.

2. The method for intelligently determining epilepsy typing based on video according to claim 1, characterized in that: Before acquiring the image sequence, the method for intelligently determining epilepsy typing based on video further comprises: Get epileptic seizure video stream; Each frame image of the epileptic seizure video stream is acquired separately to form an image sequence.

3. The method for intelligently determining epilepsy typing based on video according to claim 2, characterized in that: The acquiring of virtual portrait image features comprises: Extracting skeleton point information in each image in the image sequence according to the image sequence; Generate a virtual bone image corresponding to each image according to the information of each bone point; The image features of the virtual skeleton image are extracted to obtain the image features of the virtual human.

4. The method for intelligently determining epilepsy typing based on video according to claim 3, characterized in that: The region of interest extraction for each image in the image sequence is performed through a YOLO deep learning network.

5. The method for intelligently determining epilepsy typing based on video according to claim 4, characterized in that: The method for intelligently determining epilepsy typing based on video further comprises: Generate a matrix corresponding to facial expressions and upper body movements, which is called the first matrix; Generate a matrix corresponding to facial expressions and lower body movements, which is called the second matrix; Vectorize the first matrix to obtain a first matrix vector; Vectorize the second matrix to obtain a second matrix vector; The step of fusing the facial features, the upper body image features, the lower body image features and the virtual portrait image features to obtain the fused features further includes: The facial features, upper body image features, lower body image features, virtual portrait image features, the first matrix vector and the second matrix vector are fused to obtain fused features.

6. The method for intelligently determining epilepsy typing based on video according to claim 5, characterized in that: The generated facial expression and upper body movement correspondence matrix, which is called the first matrix, includes: Acquire a training set, wherein the training set includes a plurality of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest of which are focal epileptic seizure image sequence groups; Each whole body image sequence group is processed as follows: The facial features and upper body image features of each whole-body image in each whole-body image sequence group are extracted. If the whole-body image is a whole-body image of a generalized epileptic seizure, the matrix point is judged to be 1, otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and upper body movements of the training set.

7. The method for intelligently determining epilepsy typing based on video according to claim 6, characterized in that: The generated facial expression and lower body movement correspondence matrix, which is called the second matrix, includes: Acquire a training set, wherein the training set includes a plurality of whole-body image sequence groups, a portion of which are generalized epileptic seizure image sequence groups, and the rest of which are focal epileptic seizure image sequence groups; The following processing is performed for each full-body image: The facial features and lower body image features of each full-body image are extracted. If the full-body image is a full-body image of a generalized epileptic seizure, the matrix point is judged to be 1, otherwise, the matrix point is judged to be 0, thereby obtaining the corresponding matrix of facial expressions and lower body movements of the training set.

8. A device for intelligently determining epilepsy typing based on video, characterized in that: The device for intelligently determining epilepsy typing based on video comprises: An image sequence acquisition module, wherein the image sequence acquisition module is used to acquire an image sequence; An area of ​​interest extraction and acquisition module, the area of ​​interest extraction and acquisition module is used to extract the area of ​​interest from each image in the image sequence, so as to acquire an area of ​​interest image, the area of ​​interest image includes a patient's facial area, a patient's upper body area, and a patient's lower body area, each of the patient's facial areas constitutes a facial area image sequence, each of the patient's upper body areas constitutes an upper body image sequence, and each of the patient's lower body areas constitutes a lower body image sequence; A facial feature acquisition module, the facial feature acquisition module is used to extract facial features of the facial region image sequence; An upper body image feature acquisition module, the upper body image feature acquisition module is used to extract upper body image features of the upper body image sequence; A lower body image feature acquisition module, the lower body image feature acquisition module is used to extract lower body image features of the lower body image sequence; A virtual portrait image feature acquisition module, wherein the virtual portrait image feature acquisition module is used to acquire virtual portrait image features; A neural network acquisition module, wherein the neural network acquisition module is used to acquire a trained epilepsy recognition neural network; A fusion feature acquisition module, the fusion feature acquisition module is used to fuse the facial features, upper body image features, lower body image features and virtual portrait image features to obtain fusion features; An epilepsy recognition result acquisition module is used to input the fusion feature into the epilepsy recognition neural network to obtain an epilepsy recognition result.

Images