Human brain abnormal discharge detection method, device, storage medium and electronic equipment

Abstract

The disclosed embodiments relate to a method, device, storage medium and electronic device for detecting abnormal discharges in the human brain, and relate to the fields of artificial intelligence and multimodal technology. The method comprises: obtaining biomedical feature data of the subject; obtaining face video data of the subject, and extracting face feature data based on the face video data; obtaining body key point data of the subject collected by a motion capture device bound to the body of the subject, and obtaining body feature data based on the body key point data; using a pre-trained abnormal discharge detection model to process the biomedical feature data, the face feature data, and the body feature data, and obtain abnormal discharge detection results of the subject. The disclosed method realizes the automated detection of abnormal discharges in the human brain and can improve detection accuracy.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence and multimodal technology. More specifically, the embodiments of the present disclosure relate to a method for detecting abnormal discharges in the human brain, a device for detecting abnormal discharges in the human brain, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.

[0003] Abnormal discharges in the human brain are caused by abnormal activity of brain cells and can cause neurological diseases such as epilepsy. Detecting abnormal discharges in the human brain through examinations such as electroencephalograms can help doctors identify patients' diseases. For example, detecting epileptic-like discharges in an electroencephalogram is one of the important criteria for diagnosing epilepsy.

[0004] The detection of abnormal brain discharges involves EEG and other test data. The work of data detection and analysis is arduous and consumes a lot of manpower and time. Therefore, the industry has an increasing demand for automated detection of abnormal brain discharges. Summary of the invention

[0005] However, the accuracy of detecting abnormal discharges in the human brain needs to be improved.

[0006] In one solution of the related technology, artificial intelligence technology is used to detect abnormal discharges in the human brain, for example, an intelligent network for identifying epileptiform discharges is trained based on deep learning to detect epileptiform discharges in the human brain. Related literature reports that its accuracy rate is only 70%, and professional doctors and technicians are required to post-process the results of artificial intelligence reading.

[0007] Another solution of the related technology combines video and EEG synchronous data to establish a deep learning-based reading system, with the intention of improving the accuracy of the existing artificial intelligence EEG interpretation system. This solution is highly dependent on video quality. When the video clarity is not high or there is occlusion of the patient's body, face, etc., the accuracy of the reading results will be affected.

[0008] Therefore, there is a great need for an improved method for detecting abnormal brain discharges, which can realize automatic detection of abnormal brain discharges and improve the accuracy of detection.

[0009] In this context, embodiments of the present disclosure are intended to provide a method for detecting abnormal discharges in the human brain, an apparatus for detecting abnormal discharges in the human brain, a computer-readable storage medium, and an electronic device.

[0010] According to a first aspect of the present disclosure, a method for detecting abnormal discharges in the human brain is provided, the method comprising: obtaining biomedical feature data of a test subject; obtaining facial video data of the test subject, and extracting facial feature data based on the facial video data; obtaining body key point data of the test subject collected by a motion capture device bound to the body of the test subject, and obtaining body feature data based on the body key point data; and processing the biomedical feature data, the facial feature data, and the body feature data using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the test subject.

[0011] In one embodiment, the obtaining of biomedical characteristic data of the subject includes: obtaining original monitoring data of the subject collected by a biomedical monitoring device; preprocessing the original monitoring data; and obtaining the biomedical characteristic data based on the preprocessed original monitoring data.

[0012] In one embodiment, the original monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; obtaining the biomedical feature data based on the preprocessed original monitoring data includes: calculating the potential difference between each channel and a reference electrode of the preprocessed multi-channel EEG monitoring data to obtain initial feature data of the EEG signal; and extracting the biomedical feature data based on the initial feature data of the EEG signal.

[0013] In one embodiment, the biomedical feature data includes EEG signal waveform features; extracting the biomedical feature data based on the EEG signal initial feature data includes: processing the EEG signal initial feature data using a pre-trained waveform feature extraction model to extract the EEG signal waveform features.

[0014] In one embodiment, the biomedical feature data includes time-frequency features of EEG signals; extracting the biomedical feature data based on the initial EEG signal feature data includes: performing time-frequency transformation on the initial EEG signal feature data to obtain EEG signal time-frequency data corresponding to the initial EEG signal feature data; and extracting the EEG signal time-frequency features based on the EEG signal time-frequency data.

[0015] In one embodiment, the preprocessing of the original monitoring data includes at least one of the following processes: resampling, filtering, noise data removal, and numerical standardization.

[0016] In one embodiment, extracting facial feature data based on the facial video data includes: detecting facial key point data from the facial video data; and obtaining the facial feature data based on the facial key point data.

[0017] In one embodiment, the facial feature data includes first facial feature data; obtaining the facial feature data based on the facial key point data includes: according to the facial key point data, cropping facial area images from multiple frames of the facial video data to obtain a facial image sequence; processing the facial image sequence using a facial feature representation model to obtain the first facial feature data.

[0018] In one embodiment, the facial feature data includes second facial feature data; obtaining the facial feature data based on the facial key point data includes: determining facial motion information corresponding to the facial key point data based on the facial key point data; and obtaining the second facial feature data based on the facial motion information.

[0019] In one embodiment, the body feature data includes first body feature data; and obtaining the body feature data based on the body key point data includes: encoding the body key point data to obtain the first body feature data.

[0020] In one embodiment, the body feature data includes second body feature data; obtaining the body feature data based on the body key point data includes: determining body movement information corresponding to the body key point data based on the body key point data; and obtaining the second body feature data based on the body movement information.

[0021] In one embodiment, the abnormal discharge detection model includes a feature fusion layer, an attention layer, and a classification layer; the use of a pre-trained abnormal discharge detection model to process the biomedical feature data, the facial feature data, and the body feature data to obtain the abnormal discharge detection result of the subject includes: inputting the biomedical feature data, the facial feature data, and the body feature data into the abnormal discharge detection model; using the fusion layer to fuse the biomedical feature data, the facial feature data, and the body feature data to obtain a fused feature; using the attention layer to characterize the fused feature to obtain an embedded feature; using the classification layer to map the embedded feature to an output space to obtain the abnormal discharge detection result of the subject.

[0022] In one embodiment, before processing the biomedical feature data, the facial feature data, and the body feature data using a pre-trained abnormal discharge detection model, the method further includes: time-aligning the biomedical feature data, the facial feature data, and the body feature data.

[0023] In one embodiment, the time alignment of the biomedical feature data, the facial feature data, and the body feature data includes: detecting one or more biomedical feature data suspected of abnormal discharge and the corresponding one or more first time points from the biomedical feature data; detecting one or more facial feature data suspected of abnormal discharge and the corresponding one or more second time points from the facial feature data; detecting one or more body feature data suspected of abnormal discharge and the corresponding one or more third time points from the body feature data; matching the biomedical feature data suspected of abnormal discharge, the facial feature data suspected of abnormal discharge, and the body feature data suspected of abnormal discharge, and determining the correspondence between the first time point, the second time point, and the third time point according to the matching result; determining the time calibration parameters based on the correspondence between the first time point, the second time point, and the third time point, and using the time calibration parameters to time align the biomedical feature data, the facial feature data, and the body feature data.

[0024] In one embodiment, the matching of the biomedical feature data of the suspected abnormal discharge, the facial feature data of the suspected abnormal discharge, and the body feature data of the suspected abnormal discharge includes: determining a first relative value between the biomedical feature data of the suspected abnormal discharge and the other biomedical feature data; determining a second relative value between the facial feature data of the suspected abnormal discharge and the other facial feature data; determining a third relative value between the body feature data of the suspected abnormal discharge and the other body feature data; and obtaining a matching result between the biomedical feature data of the suspected abnormal discharge and the facial feature data of the suspected abnormal discharge and the body feature data of the suspected abnormal discharge by comparing the first relative value, the second relative value, and the third relative value.

[0025] According to a second aspect of the present disclosure, a human brain abnormal discharge detection device is provided, the device comprising: a first acquisition module, configured to acquire biomedical feature data of a test subject; a second acquisition module, configured to acquire face video data of the test subject, and extract face feature data based on the face video data; a third acquisition module, configured to acquire body key point data of the test subject collected by a motion capture device bound to the body of the test subject, and obtain body feature data based on the body key point data; a data processing module, configured to process the biomedical feature data, the face feature data, and the body feature data using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the test subject.

[0026] In one embodiment, the obtaining of biomedical characteristic data of the subject includes: obtaining original monitoring data of the subject collected by a biomedical monitoring device; preprocessing the original monitoring data; and obtaining the biomedical characteristic data based on the preprocessed original monitoring data.

[0027] In one embodiment, the original monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; obtaining the biomedical feature data based on the preprocessed original monitoring data includes: calculating the potential difference between each channel and a reference electrode of the preprocessed multi-channel EEG monitoring data to obtain initial feature data of the EEG signal; and extracting the biomedical feature data based on the initial feature data of the EEG signal.

[0028] In one embodiment, the biomedical feature data includes EEG signal waveform features; extracting the biomedical feature data based on the EEG signal initial feature data includes: processing the EEG signal initial feature data using a pre-trained waveform feature extraction model to extract the EEG signal waveform features.

[0029] In one embodiment, the biomedical feature data includes time-frequency features of EEG signals; extracting the biomedical feature data based on the initial EEG signal feature data includes: performing time-frequency transformation on the initial EEG signal feature data to obtain EEG signal time-frequency data corresponding to the initial EEG signal feature data; and extracting the EEG signal time-frequency features based on the EEG signal time-frequency data.

[0030] In one embodiment, the preprocessing of the original monitoring data includes at least one of the following processes: resampling, filtering, noise data removal, and numerical standardization.

[0031] In one embodiment, extracting facial feature data based on the facial video data includes: detecting facial key point data from the facial video data; and obtaining the facial feature data based on the facial key point data.

[0032] In one embodiment, the facial feature data includes first facial feature data; obtaining the facial feature data based on the facial key point data includes: according to the facial key point data, cropping facial area images from multiple frames of the facial video data to obtain a facial image sequence; processing the facial image sequence using a facial feature representation model to obtain the first facial feature data.

[0033] In one embodiment, the facial feature data includes second facial feature data; obtaining the facial feature data based on the facial key point data includes: determining facial motion information corresponding to the facial key point data based on the facial key point data; and obtaining the second facial feature data based on the facial motion information.

[0034] In one embodiment, the body feature data includes first body feature data; and obtaining the body feature data based on the body key point data includes: encoding the body key point data to obtain the first body feature data.

[0035] In one embodiment, the body feature data includes second body feature data; obtaining the body feature data based on the body key point data includes: determining body movement information corresponding to the body key point data based on the body key point data; and obtaining the second body feature data based on the body movement information.

[0036] In one embodiment, the abnormal discharge detection model includes a feature fusion layer, an attention layer, and a classification layer; the use of a pre-trained abnormal discharge detection model to process the biomedical feature data, the facial feature data, and the body feature data to obtain the abnormal discharge detection result of the subject includes: inputting the biomedical feature data, the facial feature data, and the body feature data into the abnormal discharge detection model; using the fusion layer to fuse the biomedical feature data, the facial feature data, and the body feature data to obtain a fused feature; using the attention layer to characterize the fused feature to obtain an embedded feature; using the classification layer to map the embedded feature to an output space to obtain the abnormal discharge detection result of the subject.

[0037] In one embodiment, the data processing module is further configured to: time-align the biomedical feature data, the facial feature data, and the body feature data before processing the biomedical feature data, the facial feature data, and the body feature data using a pre-trained abnormal discharge detection model.

[0038] In one embodiment, the time alignment of the biomedical feature data, the facial feature data, and the body feature data includes: detecting one or more biomedical feature data suspected of abnormal discharge and the corresponding one or more first time points from the biomedical feature data; detecting one or more facial feature data suspected of abnormal discharge and the corresponding one or more second time points from the facial feature data; detecting one or more body feature data suspected of abnormal discharge and the corresponding one or more third time points from the body feature data; matching the biomedical feature data suspected of abnormal discharge, the facial feature data suspected of abnormal discharge, and the body feature data suspected of abnormal discharge, and determining the correspondence between the first time point, the second time point, and the third time point according to the matching result; determining the time calibration parameters based on the correspondence between the first time point, the second time point, and the third time point, and using the time calibration parameters to time align the biomedical feature data, the facial feature data, and the body feature data.

[0039] In one embodiment, the matching of the biomedical feature data of the suspected abnormal discharge, the facial feature data of the suspected abnormal discharge, and the body feature data of the suspected abnormal discharge includes: determining a first relative value between the biomedical feature data of the suspected abnormal discharge and the other biomedical feature data; determining a second relative value between the facial feature data of the suspected abnormal discharge and the other facial feature data; determining a third relative value between the body feature data of the suspected abnormal discharge and the other body feature data; and obtaining a matching result between the biomedical feature data of the suspected abnormal discharge and the facial feature data of the suspected abnormal discharge and the body feature data of the suspected abnormal discharge by comparing the first relative value, the second relative value, and the third relative value.

[0040] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting abnormal discharges in the human brain according to the first aspect and its possible implementation methods are implemented.

[0041] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the human brain abnormal discharge detection method and its possible implementation method of the above-mentioned first aspect by executing the executable instructions.

[0042] In the scheme disclosed in the present invention, on the one hand, biomedical feature data, facial feature data, and body feature data are obtained, and abnormal brain discharge detection is performed by combining the three data, which can improve the accuracy of the detection results, and no manual processing such as manual reading is required throughout the process, thereby realizing the automatic detection of abnormal brain discharge. On the other hand, the body feature data comes from the key point data of the body collected by the motion capture device, which can accurately obtain the key point position coordinates of the required body parts, avoid the problem of inaccurate data being obtained due to body occlusion of the subject, etc., improve the accuracy and stability of the data, and further ensure the quality of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1A A schematic diagram showing a system architecture in this exemplary embodiment.

[0044] Figure 1B A schematic diagram showing another system architecture in this exemplary embodiment.

[0045] Figure 2 A flow chart showing a method for detecting abnormal discharges in the human brain in this exemplary embodiment.

[0046] Figure 3 A flow chart of acquiring biomedical feature data in this exemplary embodiment is shown.

[0047] Figure 4 A schematic diagram showing a multi-channel EEG signal in this exemplary embodiment.

[0048] Figure 5 A schematic diagram showing feature extraction from multi-channel EEG monitoring data in this exemplary embodiment.

[0049] Figure 6 A schematic diagram showing facial key points and body key points in this exemplary embodiment.

[0050] Figure 7 A schematic diagram showing feature extraction from face video data in this exemplary embodiment.

[0051] Figure 8 A flow chart showing how to obtain abnormal discharge detection results using the abnormal discharge detection model in this exemplary embodiment.

[0052] Fig. 9 A schematic diagram showing the method of obtaining a training data set in this exemplary embodiment.

[0053] Fig.10 A schematic diagram showing the training of an abnormal discharge detection model in this exemplary embodiment.

[0054] Fig.11A schematic diagram showing abnormal discharge detection in the human brain in this exemplary embodiment.

[0055] Fig.12 A schematic structural diagram of a device for detecting abnormal discharges in the human brain in this exemplary embodiment is shown.

[0056] Fig.13 A schematic structural diagram of an electronic device in this exemplary embodiment is shown.

[0057] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0058] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] The embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0060] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0062] The inventors have found that the accuracy of abnormal brain discharge detection needs to be improved. Specifically, in one solution of the related technology, artificial intelligence technology is used to detect abnormal brain discharges, such as training an intelligent network for epileptiform discharge recognition based on deep learning to detect epileptiform discharges in the human brain. Relevant literature reports that its accuracy rate is only 70%, and professional doctors and technicians are required to post-process the artificial intelligence reading results. In another solution of the related technology, a deep learning-based reading system is established in combination with video and EEG synchronization data, with the intention of improving the accuracy of existing artificial intelligence EEG reading systems. This solution is highly dependent on video quality. When the video clarity is not high, or there is occlusion on the patient's body, face and other parts, the accuracy of the reading results is affected.

[0063] In view of the above, the present disclosure provides a method for detecting abnormal discharges in the human brain, a device for detecting abnormal discharges in the human brain, a computer-readable storage medium, and an electronic device. On the one hand, biomedical feature data, facial feature data, and body feature data are obtained, and abnormal discharges in the human brain are detected by combining the three data, which can improve the accuracy of the detection results, and no manual processing such as manual reading is required throughout the process, thereby realizing the automatic detection of abnormal discharges in the human brain. On the other hand, the body feature data comes from the key point data of the body collected by the motion capture device, which can accurately obtain the key point position coordinates of the required body parts, avoid the problem of not being able to obtain accurate data due to body occlusion of the subject, etc., improve the accuracy and stability of the data, and further ensure the quality of the detection results.

[0064] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0065] Application Scenario Overview

[0066] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0067] The embodiments of the present disclosure can be applied to the detection of suspected patients and related scenarios of auxiliary diagnosis and treatment. For example, a user is suspected of having epilepsy. During the treatment in the hospital, the doctor needs to understand the user's EEG state, especially whether there is abnormal discharge in the brain. The detection can be achieved by the human brain abnormal discharge detection method disclosed in the present disclosure. The application scenario is specifically described below in combination with the system architecture.

[0068] Figure 1A A schematic diagram of a system architecture for abnormal brain discharge detection is shown. The system architecture includes a test object 101, a biomedical monitoring device 102, a video acquisition device 103, a motion capture device 104, and a processing device 105. When abnormal brain discharge detection is required for the test object 101, the biomedical monitoring device 102, the video acquisition device 103, and the motion capture device 104 can be set for the test object 101 to collect corresponding data, and the data is processed by the processing device 105.

[0069] The biomedical monitoring device 102 may include detection electrodes 1021, 1022, 1023 and a main device 1024. The detection electrodes 1021, 1022, 1023 may contact different parts of the subject 101 to collect electrical signals. The main device 1024 aggregates and processes the electrical signals to obtain original monitoring data or biomedical characteristic data. Exemplarily, the biomedical monitoring device 102 may be an electroencephalogram (EEG) monitoring device, and the detection electrodes 1021, 1022, 1023 may be fixed to various positions on the head of the subject 101 to collect multi-channel EEG signals. Of course, Figure 1A The biomedical monitoring device 102 shown is merely exemplary and may also include a helmet, a hat, bedding, packaged portable detection electrodes, and other components, which are not limited in the present disclosure.

[0070] The video acquisition device 103 is used to acquire facial video data of the object under test 101, such as a surveillance camera, a mobile phone with a shooting function, etc. It can be set in a position where the camera faces the object under test 101 to continuously monitor and shoot facial video data.

[0071] The motion capture device 104 may include sensors 1041, 1042, and 1043, which may be bound to the key parts of the body of the subject 101 to sense the movement of each part and collect the key point data of the body of the subject 101. For example, sensors 1041 and 1042 may be bound to the arm joints of the subject 101, and sensor 1043 may be bound to the fingers and palms of the subject 101 (such as sensor 1043 is set at the fingers and palms of the motion capture gloves, when the subject 101 wears the motion capture gloves, sensor 1043 is located at the fingers and palms of the subject 101), and the motion capture device 104 may collect the real-time position data of the arm joints, fingers, and palms of the subject 101, that is, the key point data of the body. Of course, the present disclosure does not limit the number of sensors and the bound body parts. In addition to the above-mentioned arm joints, fingers, and palms, the sensors of the motion capture device 104 may be bound to other key parts of the subject 101 according to specific needs to collect the corresponding key point data of the body.

[0072] The biomedical monitoring device 102, the video acquisition device 103, and the motion capture device 104 can be connected to the processing device 105 for communication, such as through a wired or wireless communication link, so that the biomedical monitoring device 102, the video acquisition device 103, and the motion capture device 104 send the collected data to the processing device 105. The processing device 105 can execute the human brain abnormal discharge detection method in this exemplary embodiment, process the acquired multimodal data, and obtain the abnormal discharge detection result of the subject. In one embodiment, the processing device 105 may include a display, which can display the abnormal discharge detection result, and can also display one or more of the original monitoring data, face video data, and body key point data.

[0073] Any two or more of the biomedical monitoring device 102, the video acquisition device 103, the motion capture device 104, and the processing device 105 can be integrated into the same device. Exemplarily, the biomedical monitoring device 102 and the processing device 105 can be integrated into the same device, or more specifically, the main device 1024 of the biomedical monitoring device 102 can serve as the processing device 105. Alternatively, the video acquisition device 103 can be integrated in the biomedical monitoring device 102 or the motion capture device 104. For example, the biomedical monitoring device 102 can include a head-mounted device, in which detection electrodes 1021, 1022, and 1023 can be set. When the subject 101 wears the head-mounted device, the head contacts the detection electrodes 1021, 1022, and 1023. A fixed bracket can also be set on the head-mounted device, and the video acquisition device 103 is fixedly placed on the other end of the fixed bracket (for example, a mobile phone can be fixedly placed by a clamp). The video acquisition device 103 faces the face of the subject 101 and can capture and collect facial video data.

[0074] Figure 1BA schematic diagram of another system architecture for abnormal discharge detection of the human brain is shown. The system architecture includes a test object 101, a biomedical monitoring device 102, a video acquisition device 103, a motion capture device 104, a data transceiver device 106, and a server 107. The biomedical monitoring device 102, the video acquisition device 103, and the motion capture device 104 can be connected to the data transceiver device 106 for communication, and the data transceiver device 106 is connected to the server 107 for communication. Thus, after the data transceiver device 106 obtains the original monitoring data or biomedical feature data, face video data, and body key point data, it sends it to the server 107. The server 107 may include any form of data processing server such as a cloud server and a distributed server. After obtaining the multimodal data sent by the data transceiver device 106, the server 107 obtains the abnormal discharge detection result of the test object by executing the abnormal discharge detection method of the human brain in this exemplary embodiment. In one implementation, the server 107 may return the abnormal discharge detection result to the data transceiver device 106 for display on the data transceiver device 106 or the biomedical monitoring device 102 .

[0075] Figure 1B The system architecture shown is suitable for portable scenarios. For example, the subject 101 can use the portable biomedical monitoring device 102, video acquisition device 103, and motion capture device 104 to collect data at home, office, or any other place. The data transceiver device 106 sends the data to the server 107 to realize abnormal brain discharge detection. In this way, the user can learn about the abnormal brain discharge detection results without leaving home. When encountering problems such as suspected epilepsy, the obtained abnormal brain discharge detection results can be sent to the doctor to help the doctor judge the condition.

[0076] Exemplary Methods

[0077] The exemplary embodiment of the present disclosure provides a method for detecting abnormal discharges in the human brain. Figure 2 As shown, the method may include steps S210 to S240. Figure 2 Provide detailed instructions for each step.

[0078] refer to Figure 2 In step S210, biomedical characteristic data of the subject is obtained.

[0079] Among them, the subject refers to a person who needs to undergo abnormal brain discharge detection, such as a possible epileptic patient. Biomedical signals are signals generated by the physiological process of the human body, which can reflect the physiological state or physical signs of a person. Biomedical feature data is feature data extracted from biomedical signals. Biomedical signals include but are not limited to any one or more of the following signals: physiological signals such as electrocardiogram signals, electroencephalogram signals, electromyography signals, electrooculogram signals and electrogastric signals, and may also include non-electrical signals such as body temperature, blood pressure, pulse, and respiration. The following is an exemplary description of biomedical signals and their acquisition process.

[0080] ECG signals can be collected by a multi-channel ECG machine. ECG (Electrocardiogram, ECG) refers to the graph of various forms of potential changes drawn from the body surface by an electrocardiograph during each cardiac cycle, with the pacemaker, atrium, and ventricle being excited successively, accompanied by changes in ECG bioelectricity. ECG is an objective indicator of the occurrence, propagation, and recovery process of cardiac excitement.

[0081] EEG signals can be electroencephalograms collected by multi-channel electroencephalogram machines. Electroencephalograms (EEGs) are graphs obtained by amplifying and recording the spontaneous bioelectric potential of the cerebral cortex from the scalp through sophisticated instruments. They are spontaneous and rhythmic electrical activities of brain cell groups recorded by electrodes. This electrical activity is a plane diagram of the relationship between the potential and time, with the potential as the vertical axis and time as the horizontal axis. The frequency (period), amplitude and phase of brain waves constitute the basic characteristics of EEGs.

[0082] Electromyography (EMG) is the temporal and spatial superposition of action potentials of motor units in numerous muscle fibers, which can be obtained by sticking electromyography sensors on the skin.

[0083] Electrooculogram (EOG) is a bioelectric signal caused by the potential difference between the cornea and the retina of the eye. It is very easy to collect and can be completed with a small number of electrodes.

[0084] Electrogastrogram (EGG) is an electrical signal generated by the contraction of stomach muscles, which can be collected on the surface of the abdominal skin using electrodes.

[0085] The present disclosure does not limit the specific method of extracting feature data from biomedical signals, such as including but not limited to: preprocessing, data statistics, extracting feature data through neural networks, etc.

[0086] In one embodiment, reference Figure 3As shown, the above-mentioned acquisition of the biomedical characteristic data of the subject may include the following steps S310 to S330:

[0087] Step S310, obtaining original monitoring data of the subject collected by the biomedical monitoring equipment.

[0088] The original monitoring data may be the original biomedical signal. For example, the biomedical monitoring device may be an electrocardiograph or an electroencephalograph, and the original monitoring data may be an electrocardiograph signal or electrocardiogram collected by the electrocardiograph, or an electroencephalogram signal or electroencephalogram collected by the electroencephalogram. The electrocardiograph and electroencephalogram are essentially graphs drawn from electrocardiograph signals and electroencephalogram signals at different times. The electrocardiograph signal may be considered equivalent to the electrocardiograph, and the electroencephalogram signal may be considered equivalent to the electroencephalogram. Figure 4 The schematic diagram of multi-channel EEG signals is shown. The original EEG monitoring data is plotted into a graph to obtain Figure 4 The waveform shown is an electroencephalogram. An electroencephalogram machine may have 23 electrodes, and each electrode collects the signal of one channel.

[0089] Step S320, pre-processing the original monitoring data.

[0090] Exemplarily, pre-processing may include but is not limited to one or more of the following processing methods:

[0091] ① Resampling. Resampling can change the sampling rate of the signal and can convert non-uniformly sampled signals into uniformly sampled signals. For example, the EEG monitoring data can be resampled at 500 Hz.

[0092] ② Filtering. For example, the EEG monitoring data, ECG monitoring data, and EMG monitoring data include 29 channels in total, of which the EEG monitoring data includes 23 channels. All 29 channels can be subjected to 50Hz notch filtering to reduce interference from AC signals, and the 23 EEG channels can be subjected to bandpass filtering (such as using a 0.1-70Hz frequency band) to reduce signal interference in non-EEG signal frequency bands.

[0093] ③ Eliminate noise data. Noise data may be caused by factors such as poor contact (such as poor contact between the biomedical monitoring equipment and the parts of the object being tested, poor contact between the equipment cables, etc.). Eliminating noise data can improve data quality and the accuracy of test results. For example, channel data with more noise can be eliminated.

[0094] ④ Numerical standardization processing. Numerical standardization processing can map data of different modalities and types into the same appropriate numerical range for unified processing. For example, different types of original monitoring data such as EEG monitoring data, ECG monitoring data, and EMG monitoring data can be de-dimensionalized first, and then normalized to an appropriate numerical range, such as by multiplying by corresponding coefficients for numerical mapping to complete numerical standardization processing.

[0095] Step S330, obtaining biomedical characteristic data based on the preprocessed original monitoring data.

[0096] The preprocessed raw monitoring data can be used as biomedical feature data, or further feature extraction processing can be performed on the preprocessed raw monitoring data. For example, the preprocessed raw monitoring data can be processed by a pre-trained feature extraction model to obtain biomedical feature data.

[0097] In one embodiment, the raw monitoring data may include: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject. The electroencephalograph may have multiple electrodes (such as the above-mentioned detection electrodes 1021, 1022, 1023), and each electrode may collect EEG monitoring data of one channel. One or more electrodes may be set on each part of the scalp of the subject, so that a total of multiple electrodes are used to collect multi-channel EEG monitoring data. Accordingly, the above-mentioned biomedical feature data obtained from the pre-processed raw monitoring data may include the following steps:

[0098] The potential difference between each channel and the reference electrode is calculated using the preprocessed multi-channel EEG monitoring data to obtain the initial characteristic data of the EEG signal;

[0099] Extract biomedical feature data based on initial feature data of EEG signals.

[0100] Among them, the reference electrode is introduced to ensure the EEG signal with potential difference, which can reduce the influence of noise. The selection of the reference electrode includes but is not limited to the following methods: using a unipolar lead, setting an active electrode and an irrelevant electrode on the subject, and the irrelevant electrode can be set at a position such as the earlobe, which is equivalent to using the irrelevant electrode as the reference electrode. The collected multi-channel EEG monitoring data includes the potential difference of each channel relative to the irrelevant electrode, and the initial characteristic data of the EEG signal is obtained after preprocessing. Using a bipolar lead method, no irrelevant electrode is set on the subject, and each active electrode uses other active electrodes as the reference electrode. The collected multi-channel EEG monitoring data includes the potential difference of each channel relative to other active electrodes, and the initial characteristic data of the EEG signal is obtained after preprocessing. Average reference, setting active electrodes and grounding electrodes on the subject, the collected multi-channel EEG monitoring data includes the potential difference of each channel relative to the ground end, and one or more of the electrodes of the multi-channel EEG monitoring data can be used as the reference electrode. The preprocessed multi-channel EEG monitoring data calculates the potential difference between each channel and the reference electrode to obtain the initial characteristic data of the EEG signal. For example, the c electrodes of the electroencephalograph are set on the scalp of the subject, and any one of the c electrodes can be selected as the reference electrode, or multiple electrodes can be used as reference electrodes, and the average value of the preprocessed multi-channel EEG monitoring data is calculated as the reference value, and the difference between the preprocessed multi-channel EEG monitoring data of each channel and the reference value is calculated to obtain the initial characteristic data of the EEG signal. In one embodiment, different electrodes can be selected as reference electrodes, and the potential difference of each channel is calculated under different reference electrodes, and then the average value is calculated, etc., to serve as the initial characteristic data of the EEG signal.

[0101] After obtaining the initial feature data of the EEG signal, the initial feature data of the EEG signal can be used as the biomedical feature data obtained in step S210, or the initial feature data of the EEG signal can be further processed, such as further extracting effective information therein to obtain biomedical feature data.

[0102] In one embodiment, the biomedical feature data may include the waveform features of the EEG signal. The above-mentioned extraction of the biomedical feature data based on the initial feature data of the EEG signal may include the following steps:

[0103] The initial feature data of the EEG signal is processed using a pre-trained waveform feature extraction model to extract the waveform features of the EEG signal.

[0104] The waveform feature extraction model may be a model of a structure such as a pre-trained Transformer, which can extract time series features and other features from the initial feature data of the EEG signal. For example, the initial feature data of the EEG signal may be input into the waveform feature extraction model, and the waveform feature extraction model performs embedding and other processing on the initial feature data of the EEG signal to obtain the waveform features of the EEG signal.

[0105] In one embodiment, the biomedical feature data may include EEG signal image features. Image features may be extracted from an EEG signal image that has been preprocessed and / or calculated based on a potential difference of a reference electrode, such as by inputting the EEG signal image into a pre-trained convolutional neural network or other model to obtain the EEG signal image features.

[0106] In one embodiment, the biomedical feature data may include the time-frequency features of the EEG signal. The above-mentioned extraction of the biomedical feature data based on the initial feature data of the EEG signal may include the following steps:

[0107] Performing time-frequency transformation on the initial characteristic data of the EEG signal to obtain the time-frequency data of the EEG signal corresponding to the initial characteristic data of the EEG signal;

[0108] Extract EEG signal time-frequency features based on EEG signal time-frequency data.

[0109] Among them, the initial feature data of the EEG signal is usually data in the time domain, expressing the changes of the electrical signal over time, and the information is relatively simple. Through time-frequency transformation, the initial feature data of the EEG signal can be converted to the joint time-frequency domain to obtain the EEG signal time-frequency data corresponding to the initial feature data of the EEG signal, which can provide the joint distribution information of the signal in the time domain and the frequency domain. The time-frequency transformation methods include but are not limited to short-time Fourier transform, wavelet transform, etc. The present disclosure does not limit which specific method is adopted. Furthermore, the time-frequency features of the EEG signal can be extracted from the time-frequency data of the EEG signal, such as extracting image features from the time-frequency diagram, or the time-frequency data of the EEG signal can be statistically analyzed and features extracted to obtain the time-frequency features of the EEG signal.

[0110] Figure 5A schematic diagram of extracting feature data from multi-channel EEG monitoring data is shown. Exemplarily, the original monitoring data includes multi-channel EEG monitoring data. The multi-channel EEG monitoring data can be preprocessed first, and then the potential difference between each channel and the reference electrode is calculated for the preprocessed multi-channel EEG monitoring data to obtain the initial feature data of the EEG signal. Next, on the one hand, the initial feature data of the EEG signal is input into a pre-trained waveform feature extraction model, such as Transformer, etc., to obtain the waveform features of the EEG signal. On the other hand, the initial feature data of the EEG signal is short-time Fourier transformed to obtain the time-frequency data of the EEG signal, and then the time-frequency data of the EEG signal is input into a pre-trained time-frequency feature extraction model, such as EfficientNetv2, to obtain the time-frequency features of the EEG signal.

[0111] In addition to the above-mentioned EEG signal waveform characteristics and EEG signal time-frequency characteristics, other EEG feature data can also be extracted, such as EEG statistical feature data.

[0112] In addition, the method of extracting EEG feature data can be used to extract biomedical feature data from original monitoring data such as electrocardiogram, electromyography, electrooculography, and electrogastric electrogram.

[0113] Exemplarily, the original monitoring data may include multi-channel ECG monitoring data. The potential difference between each channel and the reference electrode may be calculated from the preprocessed multi-channel ECG monitoring data to obtain initial feature data of the ECG signal; and biomedical feature data may be extracted based on the initial feature data of the ECG signal. Among them, the biomedical feature data includes ECG signal waveform features. The initial feature data of the ECG signal may be processed using a pre-trained waveform feature extraction model to extract the ECG signal waveform features. The biomedical feature data includes ECG signal time-frequency features. The initial feature data of the ECG signal may be subjected to time-frequency transformation to obtain ECG signal time-frequency data corresponding to the ECG signal feature data; and the ECG signal time-frequency features may be extracted based on the ECG signal time-frequency data.

[0114] Exemplarily, the original monitoring data may include multi-channel electromyography monitoring data. The potential difference between each channel and the reference electrode may be calculated from the preprocessed multi-channel electromyography monitoring data to obtain initial electromyography signal feature data; and biomedical feature data may be extracted based on the initial electromyography signal feature data. Among them, the biomedical feature data include electromyography signal waveform features. The initial electrocardiogram signal feature data may be processed using a pre-trained waveform feature extraction model to extract the electrocardiogram signal waveform features. The biomedical feature data include electromyography signal time-frequency features. The initial electromyography signal feature data may be subjected to time-frequency transformation to obtain electromyography signal time-frequency data corresponding to the electromyography signal feature data; and the electromyography signal time-frequency features may be extracted based on the electromyography signal time-frequency data.

[0115] In one embodiment, after obtaining the original biomedical monitoring data, the biomedical monitoring data can be segmented into segments with a certain time length (such as 4 seconds), such as obtaining biomedical monitoring data segments with 4 seconds as a unit, preprocessing the biomedical monitoring data of each segment, extracting features from the preprocessed biomedical monitoring data, and obtaining biomedical feature data of each segment. Subsequently, the subject is detected in segments to see whether abnormal discharge occurs in each segment.

[0116] Continue to refer Figure 2 In step S220, facial video data of the subject is obtained, and facial feature data is extracted based on the facial video data.

[0117] The face video data of the subject can be obtained by shooting with a video acquisition device, and can be a face video of the subject shot over a period of time. The face video data contains a large amount of information, from which information data related to abnormal brain discharges, i.e., face feature data, can be extracted. For example, a pre-trained image or video detection model can be used to process the face video data, extract abstract features, and obtain face feature data in the form of vectors, matrices, etc.

[0118] In one implementation, the above-mentioned extraction of facial feature data based on facial video data may include the following steps:

[0119] Detect facial key point data from face video data;

[0120] The facial feature data is obtained according to the facial key point data.

[0121] Figure 6 A schematic diagram of the key points of the face and the body is shown. The 33 key points of the face and the body can be numbered and recorded as key points 0 to 32. Among them, key points 0 to 10 are key points of the face, and key points 11 to 32 are key points of the body. Of course, the present disclosure does not limit the number of key points, and can be increased or decreased according to specific circumstances.

[0122] The facial key point data can be detected by using a pre-trained face detection model or a face detection algorithm. For example, one or more frames of images can be captured from the face video data, and the images can be analyzed by target detection and key point detection algorithms to detect the positions of the facial key points. For example, the positions of the facial key points can be determined by detecting the positions of facial organs such as eyes, nose, mouth, ears, etc. The position data of the facial key points are the facial key point data.

[0123] After obtaining the facial key point data, the facial feature data can be further extracted. The facial key point data represents the static position of the facial key points at one or more frames, and the facial feature data can represent the dynamic features of the face, such as the serialized features reflected by the position changes of the facial key points.

[0124] In one embodiment, the facial feature data includes first facial feature data. The above-mentioned obtaining facial feature data according to facial key point data may include the following steps:

[0125] According to the facial key point data, the facial region images are cropped from multiple frames of the facial video data to obtain a facial image sequence;

[0126] The facial image sequence is processed using a facial feature representation model to obtain first facial feature data.

[0127] Among them, the face video data does not necessarily contain only the face area, and other screen contents may also be captured, such as capturing part of the body of the subject, or capturing the environment around the subject, etc. After obtaining the key face data, the face position can be determined, thereby capturing the face area image from the face video data. For example, a face bounding box can be generated based on the face key point data, including the eyes, nose, mouth, ears and other human face organs, and the image of the face bounding box can be captured, or the face bounding box can be appropriately enlarged (such as multiplying the width and height of the face bounding box by a proportional coefficient greater than 1, the proportional coefficient can be 1.1, 1.2, etc., which can be determined based on experience or specific needs), and the image of the enlarged face bounding box can be captured.

[0128] It should be understood that the face area image can be intercepted only for some frames in the face video data without intercepting every frame, thereby reducing the number of face area images and reducing the amount of calculation. Exemplarily, after acquiring the face video data, key frame images can be extracted, such as extracting one frame every certain number of frames, or detecting the difference between adjacent frames, and when the difference reaches a predetermined value, extracting the next frame in the adjacent frames as the key frame image. Perform face key point detection on the key frame image to obtain face key point data, and then crop the face area image from the key frame image based on the face key point data.

[0129] The facial region images of different frames form a facial image sequence. The facial feature representation model is a pre-trained machine learning model for processing image sequences, such as a neural network based on structures such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit). The first facial feature data is a feature extracted from the facial image sequence, which can be a high-dimensional feature. The facial image sequence is input into the pre-trained facial feature representation model to obtain the first facial feature data.

[0130] In one embodiment, the facial feature data includes second facial feature data. The above-mentioned obtaining facial feature data according to the facial position and facial key point data may include the following steps:

[0131] Determine facial action information corresponding to the facial key point data according to the facial key point data;

[0132] Second facial feature data is obtained based on the facial motion information.

[0133] The second facial feature data is feature data that characterizes facial motion information, and may be low-dimensional features. From the facial key point data of different frames, the position change information of the facial key points may be obtained, and then the corresponding facial motion information may be identified. For example, the position change information of the facial key points may be identified according to the rules to obtain a facial motion recognition result (or classification result), such as identifying a facial motion as blinking, shaking the head, frowning, opening the mouth, etc., and the recognition result may be used as the second facial feature data.

[0134] In one implementation, the above-mentioned determining the facial action information corresponding to the facial key point data according to the facial key point data may include the following steps:

[0135] According to the facial key point data, the facial key points that are in motion and their displacement information are determined to obtain facial motion information corresponding to the facial key point data.

[0136] Among them, the facial key point data can represent the position of the facial key points at different times, and can be used to determine the facial key points that move within a period of time, and record the identifiers of these facial key points (such as Figure 6 The key point numbers in the image are collected and their displacement information is used to form the facial motion information of the object under test. The facial motion information can be used as the second facial feature data.

[0137] In one embodiment, after acquiring the face video data, the face video data can be segmented by a certain time length (such as 4 seconds, which can be the same as the time length of the biomedical monitoring data segmentation), such as obtaining face video data segments in units of 4 seconds. In each segment of the face video data, one or more key frames are determined, such as the first frame, the midpoint frame, the last frame, etc., and the face key points are detected in the key frames to obtain the displacement information of the face key points, thereby forming the face motion information of the subject under test as the second face feature data. The position of the face area is determined based on the detected face key points, and the face area image is intercepted from the face video data to form a face image sequence. The face image sequence is processed using the face feature representation model to obtain the first face feature data. Subsequently, the first face feature data, the second face feature data and other information are input into the abnormal discharge detection model in units of segments to detect whether the subject under test has abnormal discharge in each segment.

[0138] In one embodiment, the positional relationship between different facial key points can be determined based on the facial key point data to generate the facial motion data to be identified; the facial motion data to be identified is matched with the facial motion data of multiple standard facial motions, and the facial motion information corresponding to the facial key point data is determined based on the matching results. Among them, the facial motion data to be identified can be the positional relationship sequence data of facial key points. For example, the positional relationship between facial key points can be represented as a vector, and the different dimensions of the vector represent the distance, orientation, etc. between different key points. According to the facial key point data of each frame, the facial key point positional relationship data of each frame can be generated in the form of a vector. The facial key point position relationship data of a single frame is used as the facial action data to be identified, and matched with the facial action data of the pre-set standard facial action. The facial action data of the standard facial action can also be a vector of facial key point position data. By calculating the similarity between the vectors, the facial action data of the standard facial action that matches the facial action data to be identified is obtained. If the similarity between the two reaches a similarity threshold, it is determined that the facial action data to be identified corresponds to the standard facial action, thereby obtaining the facial action information corresponding to the facial key point data. Alternatively, the facial key point position relationship data of different frames are formed into a sequence as the facial action data to be identified. The facial action data of the standard facial action can also be a sequence. By matching between the sequences, the facial action information corresponding to the facial key point data is identified.

[0139] Figure 7A schematic diagram of extracting features from face video data is shown. After acquiring face video data, face key point data is first determined by target detection or other methods. Next, on the one hand, the face region in the face video data, i.e., the ROI region (Region Of Interest) is determined based on the face key point data, and the face region image is intercepted from the face video data to form a face image sequence, and the face image sequence is input into a pre-trained face feature representation model to obtain first face feature data. On the other hand, face action information is identified based on the face key point data, and second face feature data is obtained based on the face action information.

[0140] In addition to the above-mentioned first facial feature data and second facial feature data, facial feature data of other aspects or dimensions may also be extracted by other methods, which is not limited in the present disclosure.

[0141] Continue to refer Figure 2 In step S230, the key point data of the body of the subject collected by the motion capture device bound to the body of the subject is obtained, and the body feature data is obtained according to the key point data of the body.

[0142] Among them, the key points of the body can be referred to Figure 6 As shown. According to specific needs and hardware conditions, it can be determined at which key points of the subject's body the motion capture device is set. For example, the motion capture device can be set only at key points such as arm joints, palms, fingers, etc. to collect the key point data of these parts of the body.

[0143] The body key point data may include the positions of the body key points at different times. The body key point data itself may be used as body feature data, or the body key point data may be further processed, such as extracting statistical features, extracting features through a machine learning model, etc., to obtain body feature data.

[0144] In one embodiment, the body feature data includes first body feature data. The above-mentioned obtaining the body feature data according to the body key point data may include the following steps:

[0145] The body key point data is encoded to obtain first body feature data.

[0146] The encoding can be used as a method of extracting features, and the first body feature data is the body feature data obtained by encoding. The body key point data can be encoded by using an encoding method such as embedding to obtain the first body feature data.

[0147] In one embodiment, the body feature data includes second body feature data. The above-mentioned obtaining of the body feature data according to the body key point data may include the following steps:

[0148] Determine body motion information corresponding to the body key point data according to the body key point data;

[0149] The second body characteristic data is obtained based on the body motion information.

[0150] The second body feature data is feature data that characterizes body motion information, and may be a low-dimensional feature. From the body key point data of different frames, the position change information of the body key points may be obtained, and then the corresponding body motion information may be identified. For example, the position change information of the body key points may be identified according to the rules to obtain a body motion recognition result (or classification result), such as identifying a body motion as raising a hand, clapping, etc., and the recognition result may be used as the second body feature data.

[0151] In one implementation, the above-mentioned determining the body motion information corresponding to the body key point data according to the body key point data may include the following steps:

[0152] According to the body key point data, the body key points that are in motion and their displacement information are determined to obtain the body motion information corresponding to the body key point data.

[0153] Among them, the body key point data can represent the position of the body key points at different times, and the body key points that move within a period of time can be determined based on this, and the identifiers of these body key points can be recorded (such as Figure 6 The key point numbers in the image) and their displacement information are used to form the body motion information of the subject. The body motion information can be used as the second body feature data.

[0154] In one embodiment, after obtaining the key point data of the body, the key point data of the body can be segmented according to a certain time length (such as 4 seconds, which can be the same as the time length of the biomedical monitoring data segmentation), such as obtaining body key point data segments with 4 seconds as the unit. The key point data of each segment is encoded respectively to obtain the first body feature data of each segment. And the corresponding body motion information is determined according to the key point data of each segment respectively, to obtain the second body feature data of each segment. Subsequently, the first body feature data, the second body feature data and other information are input into the abnormal discharge detection model in units of segments to detect whether abnormal discharge occurs in each segment of the subject.

[0155] In one embodiment, the positional relationship between different body key points can be determined based on the body key point data to generate the body motion data to be identified; the body motion data to be identified is matched with the body motion data of multiple standard body motions, and the body motion information corresponding to the body key point data is determined based on the matching results. Among them, the body motion data to be identified can be the positional relationship sequence data of the body key points. For example, the positional relationship between the body key points can be represented as a vector, and the different dimensions of the vector represent the distance, orientation, etc. between different key points. According to the body key point data of each frame, the body key point positional relationship data of each frame can be generated correspondingly in the form of a vector. The body key point positional relationship data of a single frame is used as the body motion data to be identified, and matched with the body motion data of the pre-set standard body motion. The body motion data of the standard body motion can also be the vector of the body key point position data. By calculating the similarity between the vectors, the body motion data of the standard body motion that matches the body motion data to be identified is obtained. If the similarity between the two reaches the similarity threshold, it is determined that the body motion data to be identified corresponds to the standard body motion, thereby obtaining the body motion information corresponding to the body key point data. Alternatively, the body key point position relationship data of different frames are formed into a sequence as the body movement data to be identified. The body movement data of standard body movements can also be a sequence. The body movement information corresponding to the body key point data is identified by matching between sequences.

[0156] In one embodiment, the body feature data may include third body feature data. The above-mentioned obtaining of the body feature data according to the body key point data may include the following steps:

[0157] Generate a body image sequence based on body key point data;

[0158] The body image sequence is processed using the body feature representation model to obtain third body feature data.

[0159] Among them, the body key point data includes the position information of the body key points. Without shooting the body video of the subject, the virtual human model (such as a digital human) can be controlled to move according to the body key point data. Specifically, according to the body key point data at a certain moment, the body key points of the virtual human model are moved to the position indicated by the body key point data to obtain a body image of a frame of the virtual human model. Multiple frames of body images are obtained from the body key point data at multiple moments, and arranged to form a body image sequence. Alternatively, the body video data of the subject can also be collected by a video acquisition device, and the body image can be captured from one or more frames of the body video data according to the body key point data to form a body image sequence.

[0160] The body feature representation model is a pre-trained machine learning model for processing image sequences, such as a neural network based on LSTM, GRU, etc. The third body feature data is a feature extracted from the body image sequence, which can be a high-dimensional feature. The body image sequence is input into the pre-trained face feature representation model to obtain the third body feature data.

[0161] In addition to the above-mentioned first physical feature data, second physical feature data, and third physical feature data, physical feature data of other aspects or dimensions may also be extracted by other means, and the present disclosure does not limit this.

[0162] Continue to refer Figure 2 In step S240, the biomedical feature data, facial feature data, and body feature data are processed using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0163] Biomedical feature data, facial feature data, and body feature data are feature data of different modalities. The abnormal discharge detection model can comprehensively process the three feature data to obtain the final abnormal discharge detection result.

[0164] In one embodiment, the abnormal discharge detection model includes a feature fusion layer, an attention layer, and a classification layer. Figure 8 As shown, the above-mentioned process of using the pre-trained abnormal discharge detection model to process the biomedical feature data, the facial feature data, and the body feature data to obtain the abnormal discharge detection result of the tested object may include the following steps S810 to S840:

[0165] Step S810, inputting biomedical feature data, facial feature data, and body feature data into an abnormal discharge detection model;

[0166] Step S820, using a fusion layer to fuse the biomedical feature data, the facial feature data, and the body feature data to obtain a fusion feature;

[0167] Step S830, using the attention layer to characterize the fused features to obtain embedded features;

[0168] Step S840, using the classification layer to map the embedded features to the output space, to obtain the abnormal discharge detection result of the object under test.

[0169] Among them, the fusion layer, the attention layer, and the classification layer are the three main parts of the abnormal discharge detection model, and each part can include one or more intermediate layers. The fusion layer can use MLP (multi-layer perceptron), splicing and other methods to perform feature fusion. Exemplarily, the fusion layer can include one or more fully connected layers and splicing layers, and the biomedical feature data, facial feature data, and body feature data are aligned by feature dimensions through the fully connected layer, and then the splicing operation is performed through the splicing layer to obtain fusion features. In the attention layer, the fusion features can be selected, such as re-characterizing the fusion features through the attention weight to obtain embedded features. The embedded features can be dense features, each dimension of which can fuse information of three modalities. The classification layer can include a fully connected layer, etc., by performing a fully connected operation on the embedded features, it is gradually mapped to the input space, and the probability value of abnormal discharge detection can also be obtained through activation functions such as sigmoid (S-type function) and softmax (normalized exponential function). The probability value can be used as the final abnormal discharge detection result, or it can be judged whether there is abnormal discharge based on the probability value, so as to obtain the final abnormal discharge detection result.

[0170] Fig. 9 A schematic diagram of obtaining a training data set is shown. Exemplarily, the original monitoring sample data can be obtained, and the EEG signal can be collected by an electroencephalograph to obtain multi-channel EEG monitoring sample data, which is preprocessed and then abnormal discharge is marked. The continuous data can be cut according to a fixed time length, such as cutting into segments of 4 seconds in length, classified into positive and negative samples, and the time period of each sample on the original continuous data is recorded, and the segments containing abnormal discharge marks are extracted as positive samples, and the segments without abnormal discharge marks are used as negative samples. Samples marked as seizures or suspicious (doctors are not sure whether it is abnormal discharge) are discarded. Medical feature training data is further extracted. The result of the mark of abnormal discharge is used as the marked data. The face video sample data and body key point sample data of the same time period as the original monitoring sample data are obtained, and face key point detection and face feature extraction are performed on the face video sample data to obtain face feature training data, and body feature extraction is performed on the body key point sample data to obtain body feature training data. Biomedical feature training data, facial feature training data, body feature training data and corresponding labeled data in the same time period form a group of supervised data, and a large amount of supervised data forms a training data set.

[0171] Fig.10A schematic diagram of training an abnormal discharge detection model is shown. The biomedical feature training data, facial feature training data, and body feature training data in the training data set are input into the abnormal discharge detection model to be trained to obtain the corresponding abnormal discharge detection sample data, and the loss function value is calculated based on the abnormal discharge detection sample data and the labeled data, and the parameters of the abnormal discharge detection model are updated according to the loss function value. The update process is iteratively performed until the abnormal discharge detection model meets the predetermined training completion condition, such as the accuracy rate on the validation set or the test set reaches the standard, and the trained abnormal discharge detection model is obtained.

[0172] In one embodiment, before using the pre-trained abnormal discharge detection model to process the biomedical feature data, the facial feature data, and the body feature data, the human brain abnormal discharge detection method may further include the following steps:

[0173] Temporally align biomedical feature data, facial feature data, and body feature data.

[0174] Among them, biomedical feature data, facial feature data, and body feature data come from different devices, and different devices may have time differences, resulting in time asynchrony between biomedical feature data, facial feature data, and body feature data, affecting the results of abnormal brain discharge detection. Therefore, the biomedical feature data, facial feature data, and body feature data can be time aligned. For example, biomedical monitoring equipment, video acquisition equipment, and motion capture equipment can be time calibrated.

[0175] In one implementation, the time alignment of the biomedical feature data, the facial feature data, and the body feature data may include the following steps:

[0176] Detecting one or more biomedical feature data suspected of abnormal discharge and the corresponding one or more first time points from the biomedical feature data;

[0177] Detecting one or more facial feature data suspected of abnormal discharge and the corresponding one or more second time points from the facial feature data;

[0178] Detecting one or more body feature data suspected of abnormal discharge and the corresponding one or more third time points from the body feature data;

[0179] Matching the biomedical feature data of the suspected abnormal discharge, the facial feature data of the suspected abnormal discharge, and the body feature data of the suspected abnormal discharge, and determining the corresponding relationship between the first time point, the second time point, and the third time point according to the matching result;

[0180] Based on the correspondence between the first time point, the second time point, and the third time point, a time calibration parameter is determined, and the time calibration parameter is used to perform time alignment on the biomedical feature data, the facial feature data, and the body feature data.

[0181] Among them, the suspected abnormal discharge may occur at one or more time points, or may occur in one or more time periods, and the starting time point and the ending time point of the time period can be used as the first, second or third time point mentioned above. For example, in the time period in which the signal value is abnormally increased in the biomedical feature data, the biomedical feature data in the time period can be used as the biomedical feature data of the suspected abnormal discharge, and the starting time point and the ending time point of the time point can be used as the first time point. Based on a similar method, the facial feature data of the suspected abnormal discharge and the second time point, and the body feature data of the suspected abnormal discharge and the third time point can be determined.

[0182] The biomedical feature data of the suspected abnormal discharge, the facial feature data of the suspected abnormal discharge, and the body feature data of the suspected abnormal discharge are matched, and then the time difference between the first time point, the second time point, and the third time point having a corresponding relationship is calculated to obtain the time calibration parameter. For example, the biomedical feature data of the suspected abnormal discharge at each first time point can be obtained, and the first time point and the second time point and the third time point closest thereto can be matched to the same set of time calibration data.

[0183] In one embodiment, the matching of the biomedical feature data of suspected abnormal discharge, the facial feature data of suspected abnormal discharge, and the body feature data of suspected abnormal discharge includes:

[0184] Determining a first relative value between the biomedical characteristic data of the suspected abnormal discharge and other biomedical characteristic data;

[0185] Determine a second relative value between the facial feature data of the person suspected of abnormal discharge and other facial feature data;

[0186] determining a third relative value between the body feature data of suspected abnormal discharge and other body feature data;

[0187] By comparing the first relative value, the second relative value, and the third relative value, a matching result between the biomedical feature data of the suspected abnormal discharge and the facial feature data of the suspected abnormal discharge and the body feature data of the suspected abnormal discharge is obtained.

[0188] Among them, the first relative value represents the relative difference between the biomedical feature data of the suspected abnormal discharge and the biomedical feature data under the normal state, such as in the form of a percentage. The second relative value represents the relative difference between the facial feature data of the suspected abnormal discharge and the facial feature data under the normal state. The second relative value represents the relative difference between the body feature data of the suspected abnormal discharge and the body feature data under the normal state. The closest first relative value, second relative value, and third relative value can be matched together to obtain the matching result between the biomedical feature data of the suspected abnormal discharge and the facial feature data of the suspected abnormal discharge and the body feature data of the suspected abnormal discharge, so as to form a set of time calibration data with the corresponding first time point, second time point, and third time point. This can improve the accuracy of the matching.

[0189] The time difference is calculated for the first time point, the second time point, and the third time point in each group of time calibration data to obtain the time calibration parameter. The time calibration parameters of each group can be averaged to obtain the final time calibration parameter. Exemplarily, the first time point can be used as a reference, and the time calibration parameter includes the time difference between the second time point and the first time point, and the time difference between the third time point and the first time point. According to the time difference between the second time point and the first time point, the timestamp of the facial feature data is adjusted, and according to the time difference between the third time point and the first time point, the timestamp of the body feature data is adjusted. In this way, the time alignment of the biomedical feature data, the facial feature data, and the body feature data is achieved.

[0190] Fig.11 The schematic diagram of abnormal discharge detection of the human brain is shown. The EEG machine collects multi-channel EEG monitoring data, and obtains biomedical feature data after preprocessing and feature extraction; the video acquisition device collects face video data, and obtains face feature data after face key point detection and feature extraction; the motion capture device collects body key point data, and obtains body feature data after feature extraction. The biomedical feature data, face feature data, and body feature data are input into the trained abnormal discharge detection model, and the abnormal discharge detection results are output.

[0191] Exemplary Devices

[0192] The exemplary embodiment of the present disclosure also provides a device for detecting abnormal discharges in the human brain. Fig.12As shown, the human brain abnormal discharge detection device 1200 may include the following program modules: a first acquisition module 1210, configured to acquire biomedical feature data of the subject; a second acquisition module 1220, configured to acquire face video data of the subject, and extract face feature data based on the face video data; a third acquisition module 1230, configured to acquire body key point data of the subject collected by a motion capture device bound to the body of the subject, and obtain body feature data based on the body key point data; a data processing module 1240, configured to process the biomedical feature data, face feature data, and body feature data using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0193] In one embodiment, obtaining biomedical characteristic data of the subject includes: obtaining original monitoring data of the subject collected by a biomedical monitoring device; preprocessing the original monitoring data; and obtaining biomedical characteristic data based on the preprocessed original monitoring data.

[0194] In one embodiment, the original monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; obtaining biomedical feature data based on the preprocessed original monitoring data, including: calculating the potential difference between each channel and the reference electrode of the preprocessed multi-channel EEG monitoring data to obtain initial feature data of the EEG signal; extracting biomedical feature data based on the initial feature data of the EEG signal.

[0195] In one embodiment, the biomedical feature data includes EEG signal waveform features; extracting the biomedical feature data based on the initial EEG signal feature data includes: processing the initial EEG signal feature data using a pre-trained waveform feature extraction model to extract the EEG signal waveform features.

[0196] In one embodiment, the biomedical feature data includes time-frequency features of EEG signals; extracting the biomedical feature data based on the initial feature data of the EEG signals includes: performing time-frequency transformation on the initial feature data of the EEG signals to obtain EEG signal time-frequency data corresponding to the initial feature data of the EEG signals; and extracting the time-frequency features of the EEG signals based on the time-frequency data of the EEG signals.

[0197] In one embodiment, the raw monitoring data is preprocessed, including at least one of the following processes: resampling, filtering, noise data removal, and numerical standardization.

[0198] In one implementation, extracting facial feature data based on facial video data includes: detecting facial key point data from the facial video data; and obtaining facial feature data based on the facial key point data.

[0199] In one embodiment, facial feature data includes first facial feature data; obtaining facial feature data based on facial key point data includes: according to the facial key point data, cropping facial area images from multiple frames of facial video data to obtain a facial image sequence; processing the facial image sequence using a facial feature representation model to obtain the first facial feature data.

[0200] In one embodiment, the facial feature data includes second facial feature data; obtaining the facial feature data based on the facial key point data includes: determining facial motion information corresponding to the facial key point data based on the facial key point data; and obtaining the second facial feature data based on the facial motion information.

[0201] In one embodiment, the body feature data includes first body feature data; obtaining the body feature data based on the body key point data includes: encoding the body key point data to obtain the first body feature data.

[0202] In one embodiment, the body feature data includes second body feature data; obtaining the body feature data based on the body key point data includes: determining body motion information corresponding to the body key point data based on the body key point data; and obtaining the second body feature data based on the body motion information.

[0203] In one embodiment, the abnormal discharge detection model includes a feature fusion layer, an attention layer, and a classification layer; the biomedical feature data, facial feature data, and body feature data are processed using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the object under test, including: inputting the biomedical feature data, facial feature data, and body feature data into the abnormal discharge detection model; using the fusion layer to fuse the biomedical feature data, facial feature data, and body feature data to obtain a fused feature; using the attention layer to characterize the fused feature to obtain an embedded feature; using the classification layer to map the embedded feature to the output space to obtain the abnormal discharge detection result of the object under test.

[0204] In one embodiment, the data processing module 1240 is further configured to: time-align the biomedical feature data, facial feature data, and body feature data before processing the biomedical feature data, facial feature data, and body feature data using a pre-trained abnormal discharge detection model.

[0205] In one embodiment, time alignment is performed on biomedical feature data, facial feature data, and body feature data, including: detecting one or more biomedical feature data suspected of abnormal discharge and corresponding one or more first time points from the biomedical feature data; detecting one or more facial feature data suspected of abnormal discharge and corresponding one or more second time points from the facial feature data; detecting one or more body feature data suspected of abnormal discharge and corresponding one or more third time points from the body feature data; matching the biomedical feature data suspected of abnormal discharge, the facial feature data suspected of abnormal discharge, and the body feature data suspected of abnormal discharge, and determining the correspondence between the first time point, the second time point, and the third time point according to the matching results; determining time calibration parameters based on the correspondence between the first time point, the second time point, and the third time point, and using the time calibration parameters to time align the biomedical feature data, the facial feature data, and the body feature data.

[0206] In one embodiment, matching biomedical feature data of suspected abnormal discharge, facial feature data of suspected abnormal discharge, and body feature data of suspected abnormal discharge includes: determining a first relative value between the biomedical feature data of suspected abnormal discharge and other biomedical feature data; determining a second relative value between the facial feature data of suspected abnormal discharge and other facial feature data; determining a third relative value between the body feature data of suspected abnormal discharge and other body feature data; and obtaining a matching result between the biomedical feature data of suspected abnormal discharge and the facial feature data of suspected abnormal discharge and the body feature data of suspected abnormal discharge by comparing the first relative value, the second relative value, and the third relative value.

[0207] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the embodiments of the above method and will not be repeated here.

[0208] Exemplary Storage Media

[0209] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method of the present disclosure when the computer program is executed by a processor. The above-mentioned method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0210] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0211] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0212] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0213] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0214] Exemplary Electronic Devices

[0215] The exemplary embodiment of the present disclosure also provides an electronic device, including a processor and a memory, wherein the memory is used to store executable instructions of the processor. The processor is configured to execute the above method of the present disclosure by executing the executable instructions.

[0216] refer to Fig.13An electronic device according to an exemplary embodiment of the present disclosure is described. Fig.13 The electronic device 1300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0217] like Fig.13 As shown, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, and a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310).

[0218] The storage unit stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1310 can perform the following steps: Figure 2 The method steps shown, etc.

[0219] The storage unit 1320 may include a volatile storage unit, such as a random access storage unit (RAM) 1321 and / or a cache storage unit 1322 , and may further include a read-only storage unit (ROM) 1323 .

[0220] The storage unit 1320 may also include a program / utility 1324 having a set (at least one) of program modules 1325, such program modules 1325 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0221] The bus 1330 may include a data bus, an address bus, and a control bus.

[0222] The electronic device 1300 may also communicate with one or more external devices 1400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed via an input / output (I / O) interface 1340. The electronic device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1350. As shown, the network adapter 1350 communicates with other modules of the electronic device 1300 via a bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0223] It should be noted that, although several modules or submodules of the device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0224] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0225] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for detecting abnormal discharges in the human brain, characterized in that: include: Obtaining biomedical characteristic data of the subject; Acquire facial video data of the subject, and extract facial feature data based on the facial video data; Acquire body key point data of the subject collected by a motion capture device bound to the body of the subject, and obtain body feature data according to the body key point data; Processing the biomedical feature data, the facial feature data, and the body feature data using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject; Wherein, before processing the biomedical feature data, the facial feature data, and the body feature data using the pre-trained abnormal discharge detection model, the method further includes: Detecting one or more biomedical feature data suspected of abnormal discharge and the corresponding one or more first time points from the biomedical feature data; Detecting one or more facial feature data suspected of abnormal discharge and the corresponding one or more second time points from the facial feature data; Detecting one or more body feature data suspected of abnormal discharge and corresponding one or more third time points from the body feature data; Determine a first relative value between the biomedical feature data of the suspected abnormal discharge and the other biomedical feature data; determine a second relative value between the facial feature data of the suspected abnormal discharge and the other facial feature data; determine a third relative value between the body feature data of the suspected abnormal discharge and the other body feature data; obtain a matching result between the biomedical feature data of the suspected abnormal discharge and the facial feature data of the suspected abnormal discharge and the body feature data of the suspected abnormal discharge by comparing the first relative value, the second relative value, and the third relative value; determine a corresponding relationship between the first time point, the second time point, and the third time point according to the matching result; Based on the correspondence between the first time point, the second time point, and the third time point, a time calibration parameter is determined, and the time calibration parameter is used to time-align the biomedical feature data, the facial feature data, and the body feature data.

2. The method according to claim 1, characterized in that The step of obtaining biomedical characteristic data of the subject includes: Acquiring original monitoring data of the subject collected by biomedical monitoring equipment; Preprocessing the original monitoring data; The biomedical characteristic data is obtained according to the preprocessed original monitoring data.

3. The method according to claim 2, characterized in that The original monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; the biomedical feature data is obtained according to the pre-processed original monitoring data, including: The potential difference between each channel and the reference electrode is calculated using the preprocessed multi-channel EEG monitoring data to obtain the initial characteristic data of the EEG signal; The biomedical feature data is extracted according to the initial feature data of the electroencephalogram signal.

4. The method according to claim 3, characterized in that The biomedical feature data includes an electroencephalogram signal waveform feature; the step of extracting the biomedical feature data based on the initial electroencephalogram signal feature data includes: The initial feature data of the EEG signal is processed using a pre-trained waveform feature extraction model to extract the waveform features of the EEG signal.

5. The method according to claim 3, characterized in that: The biomedical feature data includes time-frequency features of EEG signals; and extracting the biomedical feature data according to the initial feature data of the EEG signals includes: Performing time-frequency transformation on the initial characteristic data of the EEG signal to obtain time-frequency data of the EEG signal corresponding to the initial characteristic data of the EEG signal; The time-frequency features of the EEG signal are extracted according to the time-frequency data of the EEG signal.

6. The method according to claim 2, characterized in that The preprocessing of the original monitoring data includes at least one of the following processing: resampling, filtering, removing noise data, and numerical standardization.

7. The method according to claim 1, characterized in that The extracting facial feature data according to the facial video data comprises: Detecting facial key point data from the facial video data; The facial feature data is obtained according to the facial key point data.

8. The method according to claim 7, characterized in that The facial feature data includes first facial feature data; the step of obtaining the facial feature data according to the facial key point data includes: According to the facial key point data, cropping facial region images from multiple frames of the facial video data to obtain a facial image sequence; The facial image sequence is processed using a facial feature representation model to obtain the first facial feature data.

9. The method according to claim 7, characterized in that: The facial feature data includes second facial feature data; the facial feature data is obtained according to the facial key point data, including: Determine the facial action information corresponding to the facial key point data according to the facial key point data; The second facial feature data is obtained based on the facial motion information.

10. The method according to claim 1, characterized in that The body feature data includes first body feature data; the body feature data is obtained according to the body key point data, including: The body key point data is encoded to obtain the first body feature data.

11. The method according to claim 1, characterized in that: The body feature data includes second body feature data; the body feature data is obtained according to the body key point data, including: Determine body motion information corresponding to the body key point data according to the body key point data; The second body feature data is obtained based on the body motion information.

12. The method according to claim 1, characterized in that The abnormal discharge detection model includes a feature fusion layer, an attention layer, and a classification layer; the pre-trained abnormal discharge detection model is used to process the biomedical feature data, the facial feature data, and the body feature data to obtain the abnormal discharge detection result of the subject, including: Inputting the biomedical feature data, the facial feature data, and the body feature data into the abnormal discharge detection model; Using the fusion layer to fuse the biomedical feature data, the facial feature data, and the body feature data, to obtain a fusion feature; Using the attention layer to characterize the fused features to obtain embedded features; The classification layer is used to map the embedded features to an output space to obtain an abnormal discharge detection result of the object under test.

13. A device for detecting abnormal discharge of human brain, characterized in that: include: A first acquisition module is configured to acquire biomedical characteristic data of the subject; A second acquisition module is configured to acquire face video data of the subject and extract face feature data according to the face video data; A third acquisition module is configured to acquire body key point data of the subject collected by a motion capture device bound to the body of the subject, and obtain body feature data according to the body key point data; a data processing module configured to process the biomedical feature data, the facial feature data, and the body feature data using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject; The data processing module is further configured to: before processing the biomedical feature data, the facial feature data, and the body feature data using the pre-trained abnormal discharge detection model, perform the following steps: Detect one or more biomedical feature data suspected of abnormal discharge and corresponding one or more first time points from the biomedical feature data; detect one or more facial feature data suspected of abnormal discharge and corresponding one or more second time points from the facial feature data; detect one or more body feature data suspected of abnormal discharge and corresponding one or more third time points from the body feature data; determine a first relative value between the biomedical feature data suspected of abnormal discharge and other biomedical feature data; determine a second relative value between the facial feature data suspected of abnormal discharge and other facial feature data; determine a third relative value between the body feature data suspected of abnormal discharge and other body feature data; obtain a matching result between the biomedical feature data suspected of abnormal discharge and the facial feature data suspected of abnormal discharge and the body feature data suspected of abnormal discharge by comparing the first relative value, the second relative value, and the third relative value; determine a corresponding relationship between the first time point, the second time point, and the third time point according to the matching result; determine a time calibration parameter based on the corresponding relationship between the first time point, the second time point, and the third time point, and use the time calibration parameter to time align the biomedical feature data, the facial feature data, and the body feature data.

14. The device according to claim 13, characterized in that The obtaining of biomedical characteristic data of the tested object comprises: obtaining original monitoring data of the tested object collected by biomedical monitoring equipment; preprocessing the original monitoring data; and obtaining the biomedical characteristic data according to the preprocessed original monitoring data.

15. The device according to claim 14, characterized in that The original monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; obtaining the biomedical feature data based on the preprocessed original monitoring data includes: calculating the potential difference between each channel and a reference electrode of the preprocessed multi-channel EEG monitoring data to obtain initial feature data of the EEG signal; and extracting the biomedical feature data based on the initial feature data of the EEG signal.

16. The device according to claim 15, characterized in that The biomedical feature data includes EEG signal waveform features; extracting the biomedical feature data based on the EEG signal initial feature data includes: processing the EEG signal initial feature data using a pre-trained waveform feature extraction model to extract the EEG signal waveform features.

17. The device according to claim 15, characterized in that The biomedical feature data includes time-frequency features of EEG signals; extracting the biomedical feature data based on the initial EEG signal feature data includes: performing time-frequency transformation on the initial EEG signal feature data to obtain EEG signal time-frequency data corresponding to the initial EEG signal feature data; and extracting the EEG signal time-frequency features based on the EEG signal time-frequency data.

18. The device according to claim 14, characterized in that The preprocessing of the original monitoring data includes at least one of the following processing: resampling, filtering, removing noise data, and numerical standardization.

19. The device according to claim 13, characterized in that The extracting of facial feature data based on the facial video data includes: detecting facial key point data from the facial video data; and obtaining the facial feature data based on the facial key point data.

20. The device according to claim 19, characterized in that The facial feature data includes first facial feature data; obtaining the facial feature data based on the facial key point data includes: according to the facial key point data, cropping facial area images from multiple frames of the facial video data to obtain a facial image sequence; and processing the facial image sequence using a facial feature representation model to obtain the first facial feature data.

21. The device according to claim 19, characterized in that The facial feature data includes second facial feature data; obtaining the facial feature data based on the facial key point data includes: determining facial motion information corresponding to the facial key point data based on the facial key point data; and obtaining the second facial feature data based on the facial motion information.

22. The device according to claim 13, characterized in that The body feature data includes first body feature data; obtaining the body feature data according to the body key point data includes: encoding the body key point data to obtain the first body feature data.

23. The device according to claim 13, characterized in that The body feature data includes second body feature data; obtaining the body feature data based on the body key point data includes: determining body movement information corresponding to the body key point data based on the body key point data; and obtaining the second body feature data based on the body movement information.

24. The device according to claim 13, characterized in that The abnormal discharge detection model includes a feature fusion layer, an attention layer, and a classification layer; the method of using the pre-trained abnormal discharge detection model to process the biomedical feature data, the facial feature data, and the body feature data to obtain the abnormal discharge detection result of the object under test includes: inputting the biomedical feature data, the facial feature data, and the body feature data into the abnormal discharge detection model; using the fusion layer to fuse the biomedical feature data, the facial feature data, and the body feature data to obtain a fused feature; using the attention layer to characterize the fused feature to obtain an embedded feature; and using the classification layer to map the embedded feature to an output space to obtain the abnormal discharge detection result of the object under test.

25. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

26. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 12 by executing the executable instructions.

Images