Method, device, storage medium and electronic device for detecting abnormal brain discharges

Abstract

The embodiments of the present disclosure relate to a method, apparatus, storage medium, and electronic device for detecting abnormal brain discharges, and relate to the fields of artificial intelligence and multimodal technologies. The method includes: obtaining biomedical feature data of a subject to be tested; obtaining video monitoring data of the subject to be tested; detecting action information of the subject to be tested according to the video monitoring data; extracting an image sequence of interest for characterizing the actions of the subject to be tested from the video monitoring data; and processing the biomedical feature data, the action information, and the image sequence of interest by using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject to be tested. The present disclosure can achieve automatic detection of abnormal brain discharges and improve the accuracy of detection.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the fields of artificial intelligence and multimodal technologies. More specifically, embodiments of the present disclosure relate to a method for detecting abnormal brain discharges, a device for detecting abnormal brain discharges, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section aims to provide background or context for the embodiments of the present disclosure recited in the claims. The description herein is not admitted to be prior art merely because it is included in this section.

[0003] Abnormal brain discharges are caused by abnormal activities of brain cells and can cause neurological diseases such as epilepsy. Detecting abnormal brain discharges through examination means such as electroencephalograms can help doctors identify the diseases of patients. For example, detecting epileptiform discharges in electroencephalograms is one of the important criteria for diagnosing epilepsy.

[0004] The detection of abnormal brain discharges involves detection data such as electroencephalograms. The work of data detection and analysis is heavy, consuming a large amount of human and time costs. Therefore, the industry has an increasing demand for the automated detection of abnormal brain discharges. Summary of the Invention

[0005] However, the accuracy of current abnormal brain discharge detection needs to be improved.

[0006] In related technologies, artificial intelligence technologies are used for detecting abnormal brain discharges. For example, an epileptiform discharge recognition intelligent network is trained based on deep learning to detect epileptiform discharges in the human brain. Relevant literature reports that its accuracy rate only reaches 70%, and professional doctors and technicians are required to post-process the results read by artificial intelligence.

[0007] Therefore, there is a great need for an improved method for detecting abnormal brain discharges, which can realize the automated detection of abnormal brain discharges and improve the accuracy of detection.

[0008] In this context, embodiments of the present disclosure are expected to provide a method for detecting abnormal brain discharges, a device for detecting abnormal brain discharges, a computer-readable storage medium, and an electronic device.

[0009] According to a first aspect of the present disclosure, there is provided a method for detecting abnormal brain discharges, including: obtaining biomedical feature data of a subject; obtaining video monitoring data of the subject; detecting action information of the subject according to the video monitoring data; extracting an interesting image sequence for characterizing the actions of the subject from the video monitoring data; and processing the biomedical feature data, the action information, and the interesting image sequence by using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0010] In one embodiment, the obtaining of the biomedical feature data of the subject to be tested includes: obtaining the biomedical monitoring data of the subject to be tested collected by a biomedical monitoring device; preprocessing the biomedical monitoring data; and obtaining the biomedical feature data according to the preprocessed biomedical monitoring data.

[0011] In one embodiment, the biomedical monitoring data includes: multi-channel electroencephalogram (EEG) monitoring data collected from multiple parts of the scalp of the subject to be tested; and the obtaining of the biomedical feature data according to the preprocessed biomedical monitoring data includes: calculating the potential difference between each channel and a reference electrode from the preprocessed multi-channel EEG monitoring data to obtain initial EEG signal feature data; and extracting the biomedical feature data according to the initial EEG signal feature data.

[0012] In one embodiment, the biomedical feature data includes EEG signal waveform features; and the extracting of the biomedical feature data according to the initial EEG signal feature data includes: using a pre-trained waveform feature extraction model to process the initial EEG signal feature data to extract the EEG signal waveform features.

[0013] In one embodiment, the biomedical feature data includes EEG signal time-frequency features; and the extracting of the biomedical feature data according to the initial EEG signal feature data includes: performing time-frequency transformation on the initial EEG signal feature data to obtain EEG signal time-frequency data corresponding to the initial EEG signal feature data; and extracting the EEG signal time-frequency features according to the EEG signal time-frequency data.

[0014] In one embodiment, the preprocessing of the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical normalization processing.

[0015] In one embodiment, the video monitoring data includes face video monitoring data; and the detecting of the action information of the subject to be tested according to the video monitoring data includes: detecting face key point data from the face video monitoring data; and obtaining the face action information of the subject to be tested according to the face key point data.

[0016] In one embodiment, the image sequence of interest includes a face image sequence; and the extracting of the image sequence of interest for characterizing the action of the subject to be tested from the video monitoring data includes: cropping face region images from multiple frames of the face video monitoring data according to the face key point data to obtain a face image sequence.

[0017] In one embodiment, obtaining the facial action information of the subject according to the facial key point data includes: determining the facial key points undergoing movement and their displacement information according to the facial key point data, so as to obtain the facial action information of the subject.

[0018] In one embodiment, the video monitoring data includes body video monitoring data; detecting the action information of the subject according to the video monitoring data includes: detecting body key point data from the body video monitoring data; obtaining the body action information of the subject according to the body key point data.

[0019] In one embodiment, the sequence of images of interest includes a sequence of body images; extracting the sequence of images of interest for characterizing the actions of the subject from the video monitoring data includes: cropping body region images from multiple frames of the body monitoring video data according to the body key point data to obtain a sequence of body images.

[0020] In one embodiment, obtaining the body action information of the subject according to the body key point data includes: determining the body key points undergoing movement and their displacement information according to the body key point data, so as to obtain the body action information of the subject.

[0021] In one embodiment, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; processing the biomedical feature data, the action information, and the sequence of images of interest by using the pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject includes: inputting the biomedical feature data, the action information, and the sequence of images of interest into the abnormal discharge detection model; using the feature processing layer to extract action feature data from the action information, extract image feature data from the sequence of images of interest, and fuse the biomedical feature data, the action feature data, and the image feature data to obtain a fused feature; using the attention layer to characterize the fused feature to obtain an embedded feature; using the classification layer to map the embedded feature to the output space to obtain the abnormal discharge detection result of the subject.

[0022] In one embodiment, before processing the biomedical feature data, the action information, and the sequence of images of interest by using the pre-trained abnormal discharge detection model, the method further includes: performing time alignment on at least one of the action information and the sequence of images of interest with the biomedical feature data.

[0023] In one embodiment, the time alignment of at least one of the action information and the sequence of images of interest with the biomedical feature data includes: detecting one or more pieces of biomedical feature data suspected of abnormal discharges and corresponding one or more first time points from the biomedical feature data; detecting one or more pieces of action data suspected of abnormal discharges and corresponding one or more second time points from the action information; matching the biomedical feature data suspected of abnormal discharges with the action data suspected of abnormal discharges, and determining the corresponding relationship between the first time point and the second time point according to the matching result; determining a time calibration parameter based on the corresponding relationship between the first time point and the second time point, and using the time calibration parameter to perform time alignment on the action information and the biomedical feature data.

[0024] In one embodiment, the matching of the biomedical feature data suspected of abnormal discharges with the action data suspected of abnormal discharges includes: determining a first relative value between the biomedical feature data suspected of abnormal discharges and other biomedical feature data; determining a second relative value between the action data suspected of abnormal discharges and other action data in the action information; obtaining a matching result between the biomedical feature data suspected of abnormal discharges and the action data suspected of abnormal discharges by comparing the first relative value with the second relative value.

[0025] According to a second aspect of the present disclosure, there is provided a human brain abnormal discharge detection device, including: a first acquisition module configured to acquire biomedical feature data of a subject to be measured; a second acquisition module configured to acquire video monitoring data of the subject to be measured; an action information detection module configured to detect action information of the subject to be measured according to the video monitoring data; an image sequence extraction module configured to extract a sequence of images of interest for characterizing the actions of the subject to be measured from the video monitoring data; and a model processing module configured to process the biomedical feature data, the action information, and the sequence of images of interest by using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject to be measured.

[0026] In one embodiment, the acquisition of the biomedical feature data of the subject to be measured includes: acquiring the biomedical monitoring data of the subject to be measured collected by a biomedical monitoring device; preprocessing the biomedical monitoring data; and obtaining the biomedical feature data according to the preprocessed biomedical monitoring data.

[0027] In one embodiment, the biomedical monitoring data includes: multi-channel electroencephalogram (EEG) monitoring data collected from multiple parts of the scalp of the subject to be tested; obtaining the biomedical feature data from the preprocessed biomedical monitoring data includes: calculating the potential difference between each channel and a reference electrode from the preprocessed multi-channel EEG monitoring data to obtain initial EEG signal feature data; extracting the biomedical feature data based on the initial EEG signal feature data.

[0028] In one embodiment, the biomedical feature data includes EEG signal waveform features; extracting the biomedical feature data based on the initial EEG signal feature data includes: using a pre-trained waveform feature extraction model to process the initial EEG signal feature data to extract the EEG signal waveform features.

[0029] In one embodiment, the biomedical feature data includes EEG signal time-frequency features; extracting the biomedical feature data based on the initial EEG signal feature data includes: performing time-frequency transformation on the initial EEG signal feature data to obtain EEG signal time-frequency data corresponding to the initial EEG signal feature data; extracting the EEG signal time-frequency features based on the EEG signal time-frequency data.

[0030] In one embodiment, preprocessing the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical normalization processing.

[0031] In one embodiment, the video monitoring data includes face video monitoring data; detecting the action information of the subject to be tested based on the video monitoring data includes: detecting face key point data from the face video monitoring data; obtaining the face action information of the subject to be tested based on the face key point data.

[0032] In one embodiment, the image sequence of interest includes a face image sequence; extracting the image sequence of interest for characterizing the actions of the subject to be tested from the video monitoring data includes: cropping face region images from multiple frames of the face video monitoring data based on the face key point data to obtain a face image sequence.

[0033] In one embodiment, obtaining the face action information of the subject to be tested based on the face key point data includes: determining the face key points that move and their displacement information based on the face key point data to obtain the face action information of the subject to be tested.

[0034] In one embodiment, the video monitoring data includes body video monitoring data; detecting the action information of the subject according to the video monitoring data includes: detecting body key point data from the body video monitoring data; and obtaining the body action information of the subject according to the body key point data.

[0035] In one embodiment, the image sequence of interest includes a body image sequence; extracting the image sequence of interest for characterizing the actions of the subject from the video monitoring data includes: cropping body region images from multiple frames of the body monitoring video data according to the body key point data to obtain a body image sequence.

[0036] In one embodiment, obtaining the body action information of the subject according to the body key point data includes: determining the body key points where movement occurs and their displacement information according to the body key point data to obtain the body action information of the subject.

[0037] In one embodiment, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; using the pre-trained abnormal discharge detection model to process the biomedical feature data, the action information, and the image sequence of interest to obtain the abnormal discharge detection result of the subject includes: inputting the biomedical feature data, the action information, and the image sequence of interest into the abnormal discharge detection model; using the feature processing layer to extract action feature data from the action information, extract image feature data from the image sequence of interest, and fuse the biomedical feature data, the action feature data, and the image feature data to obtain a fused feature; using the attention layer to characterize the fused feature to obtain an embedded feature; and using the classification layer to map the embedded feature to an output space to obtain the abnormal discharge detection result of the subject.

[0038] In one embodiment, the model processing module is further configured to: before using the pre-trained abnormal discharge detection model to process the biomedical feature data, the action information, and the image sequence of interest, perform time alignment on at least one of the action information and the image sequence of interest with the biomedical feature data.

[0039] In one embodiment, the time alignment of at least one of the action information and the sequence of images of interest with the biomedical feature data includes: detecting one or more pieces of biomedical feature data with suspected abnormal discharges and corresponding one or more first time points from the biomedical feature data; detecting one or more pieces of action data with suspected abnormal discharges and corresponding one or more second time points from the action information; matching the biomedical feature data with suspected abnormal discharges and the action data with suspected abnormal discharges, and determining the corresponding relationship between the first time point and the second time point according to the matching result; determining a time calibration parameter based on the corresponding relationship between the first time point and the second time point, and using the time calibration parameter to perform time alignment on the action information and the biomedical feature data.

[0040] In one embodiment, the matching of the biomedical feature data with suspected abnormal discharges and the action data with suspected abnormal discharges includes: determining a first relative value between the biomedical feature data with suspected abnormal discharges and other biomedical feature data; determining a second relative value between the action data with suspected abnormal discharges and other action data in the action information; and obtaining a matching result between the biomedical feature data with suspected abnormal discharges and the action data with suspected abnormal discharges by comparing the first relative value and the second relative value.

[0041] According to a third aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program, which when executed by a processor, implements the method for detecting abnormal discharges in the human brain and its possible implementation manners in the first aspect described above.

[0042] According to a fourth aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method for detecting abnormal discharges in the human brain and its possible implementation manners in the first aspect described above by executing the executable instructions.

[0043] In the solution of the present disclosure, on the one hand, biomedical feature data and video monitoring data are obtained. The video monitoring data is processed to obtain the action information of the subject under test and the sequence of images of interest. By combining the three types of data, namely biomedical feature data, action information, and the sequence of images of interest, for detecting abnormal brain discharges, the accuracy of the detection result can be improved. Information missing can be mutually compensated among different types of data. For example, the action information or the sequence of images of interest can compensate for the information that cannot be reflected in the biomedical feature data, ensuring the stability of the detection result and reducing the situation of misjudgment. And throughout the process, there is no need for artificial reading and other human processing, realizing the automated detection of abnormal brain discharges. On the other hand, through the processing of the video monitoring data, two different types of data, namely action information and the sequence of images of interest, are obtained, realizing the full excavation of the video monitoring data and enhancing the richness and integrity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1A FIG. shows a schematic diagram of a system architecture in this exemplary embodiment.

[0045] Figure 1B FIG. shows a schematic diagram of another system architecture in this exemplary embodiment.

[0046] Figure 2 FIG. shows a flowchart of a method for detecting abnormal brain discharges in this exemplary embodiment.

[0047] Figure 3 FIG. shows a flowchart of obtaining biomedical feature data in this exemplary embodiment.

[0048] Figure 4 FIG. shows a schematic diagram of multi-channel electroencephalogram signals in this exemplary embodiment.

[0049] Figure 5 FIG. shows a schematic diagram of extracting features from multi-channel electroencephalogram monitoring data in this exemplary embodiment.

[0050] Figure 6 FIG. shows a schematic diagram of face key points and body key points in this exemplary embodiment.

[0051] Figure 7 FIG. shows a schematic diagram of processing video monitoring data in this exemplary embodiment.

[0052] Figure 8 FIG. shows a flowchart of obtaining an abnormal discharge detection result by using an abnormal discharge detection model in this exemplary embodiment.

[0053] Figure 9 FIG. shows a schematic diagram of obtaining a training data set in this exemplary embodiment.

[0054] Figure 10 A schematic diagram showing the training of an abnormal discharge detection model in this exemplary embodiment.

[0055] Figure 11 A schematic diagram showing the detection of abnormal discharges in the human brain in this exemplary embodiment.

[0056] Figure 12 A schematic structural diagram of a device for detecting abnormal discharges in the human brain in this exemplary embodiment.

[0057] Figure 13 A schematic structural diagram of an electronic device in this exemplary embodiment.

[0058] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. Detailed implementation manners

[0059] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and then implement the present disclosure, and do not limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to be able to convey the scope of the present disclosure completely to those skilled in the art.

[0060] Embodiments of the present disclosure can be implemented as a system, a device, an apparatus, a method, or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms: completely hardware, completely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0061] The principles and spirit of the present disclosure will be elaborated in detail below with reference to several representative embodiments of the present disclosure. Summary of the Invention

[0063] The inventors of the present disclosure have found that the accuracy of current detection of abnormal discharges in the human brain needs to be improved. Specifically, in the related art, artificial intelligence technology is used to detect abnormal discharges in the human brain. For example, an intelligent network for identifying epileptiform discharges is trained based on deep learning to detect epileptiform discharges in the human brain. Relevant literature reports that its accuracy rate only reaches 70%, and professional doctors and technicians need to post-process the results read by artificial intelligence.

[0064] In view of the above, the present disclosure provides a method for detecting abnormal brain discharges, a device for detecting abnormal brain discharges, a computer-readable storage medium, and an electronic device. On the one hand, by acquiring biomedical feature data and video monitoring data, processing the video monitoring data to obtain the action information and the sequence of images of interest of the subject under test, and detecting abnormal brain discharges by combining the three types of data, namely, biomedical feature data, action information, and the sequence of images of interest, the accuracy of the detection result can be improved. Information missing among different types of data can be mutually compensated. For example, the action information or the sequence of images of interest can compensate for the information that cannot be reflected in the biomedical feature data, ensuring the stability of the detection result, reducing the situation of misjudgment, and realizing the automatic detection of abnormal brain discharges without manual reading or other human processing throughout the process. On the other hand, by processing the video monitoring data, two different types of data, namely, action information and the sequence of images of interest, are obtained, realizing the full excavation of the video monitoring data and enhancing the richness and integrity of the data.

[0065] After introducing the basic principle of the present disclosure, various non-limiting embodiments of the present disclosure will be specifically introduced below.

[0066] Overview of Application Scenarios

[0067] It should be noted that the following application scenarios are only shown for the convenience of understanding the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0068] The embodiments of the present disclosure can be applied to relevant scenarios of detecting and assisting in the diagnosis and treatment of suspected patients. For example, a certain user is suspected of having epilepsy. During the treatment in the hospital, the doctor needs to know the electroencephalogram state of the user, especially whether there is abnormal discharge in the brain. The method for detecting abnormal brain discharges of the present disclosure can be used for detection. The following is a specific description of the application scenario in combination with the system architecture.

[0069] Figure 1A A schematic diagram of a system architecture for detecting abnormal brain discharges is shown. The system architecture includes a subject under test 101, a biomedical monitoring device 102, a video acquisition device 103, and a processing device 104. When it is necessary to detect abnormal brain discharges of the subject under test 101, a biomedical monitoring device 102 and a video acquisition device 103 can be set for the subject under test 101 to collect corresponding data, and the data is processed by the processing device 104.

[0070] The biomedical monitoring device 102 may include detection electrodes 1021, 1022, 1023 and a main device 1024. The detection electrodes 1021, 1022, 1023 may contact different parts of the subject 101 to collect electrical signals, and the main device 1024 aggregates and processes the electrical signals to obtain biomedical monitoring data or biomedical characteristic data. Exemplarily, the biomedical monitoring device 102 may be an electroencephalogram monitoring device (such as an electroencephalograph), and the detection electrodes 1021, 1022, 1023 may be fixed to various positions on the head of the subject 101 to collect multi-channel electroencephalogram signals. Of course, Figure 1A The shown biomedical monitoring device 102 is only exemplary, and may also include other various components such as helmets, hats, bedding, packaged portable detection electrodes, etc. The present disclosure does not limit this.

[0071] The video acquisition device 103 is used to acquire video monitoring data of the subject 101. For example, it may be a surveillance camera, a mobile phone with a shooting function, etc. It may be set at a position where the camera faces the subject 101 to continuously monitor and shoot to obtain video monitoring data. In one implementation, the video acquisition device 103 may include a dual-channel camera to respectively shoot the face and body of the subject to respectively acquire face video monitoring data and body video monitoring data.

[0072] The biomedical monitoring device 102 and the video acquisition device 103 may be communicatively connected to the processing device 104, for example, connected through a wired or wireless communication link, so that the biomedical monitoring device 102 and the video acquisition device 103 send the collected data to the processing device 104. The processing device 104 may execute the method for detecting abnormal brain discharges in this exemplary embodiment to process the acquired multimodal data and obtain the abnormal discharge detection result of the subject. In one implementation, the processing device 104 may include a display, which may display the abnormal discharge detection result, and may also display one or more of the biomedical monitoring data and the video monitoring data.

[0073] Any two or more of the biomedical monitoring device 102, the video acquisition device 103, and the processing device 104 can be integrated into the same device. Exemplarily, the biomedical monitoring device 102 and the processing device 104 can be integrated into the same device, or more specifically, the main device 1024 of the biomedical monitoring device 102 can serve as the processing device 104. Alternatively, the video acquisition device 103 can be integrated into the biomedical monitoring device 102 or the processing device 104. For example, the biomedical monitoring device 102 can include a head-mounted device, in which detection electrodes 1021, 1022, and 1023 can be provided. When the subject 101 wears the head-mounted device and the head contacts the detection electrodes 1021, 1022, and 1023, a fixing bracket can also be provided on the head-mounted device, and the other end of the fixing bracket is fixedly placed with the video acquisition device 103 (such as a mobile phone can be fixedly placed through a clamp). The video acquisition device 103 faces the face and body of the subject 101 and can capture video monitoring data.

[0074] Figure 1B A schematic diagram showing another system architecture for detecting abnormal brain discharges is shown. The system architecture includes a subject 101, a biomedical monitoring device 102, a video acquisition device 103, a data transceiver device 105, and a server 106. The biomedical monitoring device 102 and the video acquisition device 103 can be communicatively connected to the data transceiver device 105, and the data transceiver device 105 is communicatively connected to the server 106. Thus, after the data transceiver device 105 obtains biomedical monitoring data or biomedical feature data, face video data, and body key point data, it sends them to the server 106. The server 106 can include any form of data processing server such as a cloud server or a distributed server. After the server 106 obtains the multimodal data sent by the data transceiver device 105, by executing the method for detecting abnormal brain discharges in this exemplary embodiment, the abnormal discharge detection result of the subject is obtained. In one embodiment, the server 106 can return the abnormal discharge detection result to the data transceiver device 105 for display on the data transceiver device 105 or the biomedical monitoring device 102.

[0075] Figure 1B The shown system architecture is applicable to portable scenarios. For example, the subject 101 can use the portable biomedical monitoring device 102 and the video acquisition device 103 at any place such as at home or in the office, and the data transceiver device 105 sends the data to the server 106 to implement the detection of abnormal brain discharges. In this way, users can understand the abnormal discharge detection results without leaving home. When encountering problems such as suspected epilepsy, the obtained abnormal discharge detection results can be sent to a doctor to help the doctor judge the condition.

[0076] In one embodiment, Figure 1A or Figure 1B The system architecture shown may further include a motion capture device, which may include one or more sensors bound to key body parts of the subject 101 to sense the movement of each part and collect body key point data of the subject 101. For example, the sensors may be bound to the arm joints, fingers, and palm of the subject 101 (such as the sensors are arranged at the fingers and palm of the motion capture glove, and when the subject 101 wears the motion capture glove, the sensors are located at the fingers and palm of the subject 101). The motion capture device can collect real-time position data of the arm joints, fingers, and palm of the subject 101 to obtain body key point data. Of course, the present disclosure does not limit the number of sensors and the body parts to which they are bound. In addition to the above-mentioned arm joints, fingers, and palm, sensors of the motion capture device can be bound to other key parts of the subject 101 according to specific needs to collect corresponding body key point data.

[0077] Exemplary Method

[0078] An exemplary embodiment of the present disclosure provides a method for detecting abnormal brain discharges. Refer to Figure 2 shown, the method may include steps S210 to S250. Each step in Figure 2 will be specifically described below.

[0079] Refer to Figure 2 , in step S210, biomedical characteristic data of the subject is obtained.

[0080] Among them, the subject refers to a person who needs to detect abnormal brain discharges, such as a possible epilepsy patient. A biomedical signal is a signal generated by the human physiological process and can reflect a person's physiological state or signs. Biomedical characteristic data is characteristic data extracted from biomedical signals. Biomedical signals include, but are not limited to, any one or more of the following signals: electrophysiological signals such as electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), electrooculogram (EOG), and electrogastrogram (EGG), and may also include non-electrophysiological signals such as body temperature, blood pressure, pulse, and respiration. The following is an exemplary description of biomedical signals and their acquisition process.

[0081] The electrocardiogram signal can be an electrocardiogram collected by a multi-channel electrocardiograph. An electrocardiogram (ECG) refers to a graph of various forms of potential changes led out from the body surface by an electrocardiograph during each cardiac cycle, accompanied by the change of bioelectricity in the heart, as the pacemaker, atrium, and ventricle are successively excited. The electrocardiogram is an objective indicator of the occurrence, propagation, and recovery process of cardiac excitation.

[0082] The electroencephalogram (EEG) signal can be the electroencephalogram collected by a multi-channel electroencephalograph. The electroencephalogram (EEG) is a graph obtained by amplifying and recording the spontaneous bioelectric potential of the cerebral cortex of the brain from the scalp using a precision instrument. It is the spontaneous and rhythmic electrical activity of a group of brain cells recorded through electrodes. This electrical activity uses potential as the vertical axis and time as the horizontal axis, thus recording the planar graph of the relationship between potential and time. The frequency (period), amplitude, and phase of the brain waves constitute the basic characteristics of the electroencephalogram.

[0083] Electromyography (EMG) signals are the superposition of motor unit action potentials in time and space among numerous muscle fibers, and can be obtained by pasting electromyography sensors on the skin.

[0084] Electrooculogram (EOG) signals are bioelectric signals caused by the potential difference between the cornea and retina of the eye, and are extremely convenient to collect, which can be completed through a small number of electrodes.

[0085] Electrogastrogram (EGG) signals are the electrical signals generated by the contraction of the stomach muscles, and can be collected using electrodes on the abdominal skin surface of the human body.

[0086] The present disclosure does not limit the specific manner of extracting feature data from biomedical signals, such as may include but not be limited to: preprocessing, data statistics, extracting feature data through neural networks, etc.

[0087] In one implementation, referring to Figure 3 as shown, the above-mentioned acquisition of the biomedical feature data of the subject to be tested may include the following steps S310 to S330:

[0088] Step S310, acquire the biomedical monitoring data of the subject to be tested collected by the biomedical monitoring device.

[0089] Among them, the biomedical monitoring data may be the original biomedical signal. For example, the biomedical monitoring device may be an electrocardiograph or an electroencephalograph, and the biomedical monitoring data may be the electrocardiogram signal or electrocardiogram collected by the electrocardiograph, the electroencephalogram signal or electroencephalogram collected by the electroencephalograph, etc. The electrocardiogram and electroencephalogram are essentially the graph lines drawn by the electrocardiogram signals and electroencephalogram signals at different times, and it can be considered that the electrocardiogram signal is equivalent to the electrocardiogram, and the electroencephalogram signal is equivalent to the electroencephalogram. Figure 4 shows a schematic diagram of multi-channel electroencephalogram signals. The original electroencephalogram monitoring data is plotted as a graph line to obtain Figure 4The waveform shown is the electroencephalogram. The electroencephalograph can have 23 electrodes, and each electrode collects the signals of one channel. Of course, the present disclosure does not limit the number of electrodes of the electroencephalograph, and it can be increased or decreased according to specific situations.

[0090] Step S320: Preprocess the biomedical monitoring data.

[0091] Exemplarily, the preprocessing can include, but is not limited to, one or more of the following processing methods:

[0092] ① Resampling. Resampling can change the sampling rate of the signal and can convert non-uniformly sampled signals into uniformly sampled signals. Exemplarily, the electroencephalogram monitoring data can be resampled at 500 Hz.

[0093] ② Filtering. Exemplarily, the electroencephalogram monitoring data, electrocardiogram monitoring data, and electromyogram monitoring data together include 29 channels, among which the electroencephalogram monitoring data includes 23 channels. Notch filtering at 50 Hz can be performed on all 29 channels to reduce the interference of alternating current signals, and band-pass filtering (such as using a frequency band of 0.1 - 70 Hz) can be performed on the 23 electroencephalogram channels to reduce the interference of signals in non-electroencephalogram signal frequency bands.

[0094] ③ Removing noise data. The noise data may be caused by factors such as poor contact (such as poor contact between the biomedical monitoring device and the part of the subject being measured, poor contact of the device cable, etc.). Removing the noise data can improve the data quality and the accuracy of the detection results. Exemplarily, the channel data with more noise can be removed.

[0095] ④ Numerical normalization processing. Numerical normalization processing can map data of different modalities and different types to the same appropriate numerical range for unified processing. Exemplarily, the dimension of different types of biomedical monitoring data such as electroencephalogram monitoring data, electrocardiogram monitoring data, and electromyogram monitoring data can be removed first, and then normalized to an appropriate numerical range. For example, numerical mapping can be completed by multiplying by corresponding coefficients to complete the numerical normalization processing.

[0096] Step S330: Obtain biomedical feature data according to the preprocessed biomedical monitoring data.

[0097] The preprocessed biomedical monitoring data can be used as the biomedical feature data, or further feature extraction processing can be performed on the preprocessed biomedical monitoring data. For example, the preprocessed biomedical monitoring data can be processed by a pre-trained feature extraction model to obtain the biomedical feature data.

[0098] In one embodiment, the biomedical monitoring data may include: multi-channel electroencephalogram (EEG) monitoring data collected from multiple parts of the scalp of a subject to be tested. Among them, the EEG machine may have multiple electrodes (such as the above-mentioned detection electrodes 1021, 1022, 1023, usually referred to as active electrodes), and each electrode can collect one channel of EEG monitoring data. One or more electrodes can be arranged on each part of the scalp of the subject to be tested, and thus a total of multiple electrodes are used to collect multi-channel EEG monitoring data. Correspondingly, the obtaining of the biomedical feature data from the preprocessed biomedical monitoring data may include the following steps:

[0099] Calculating the potential difference between each channel and the reference electrode from the preprocessed multi-channel EEG monitoring data to obtain the initial EEG signal feature data;

[0100] Extracting the biomedical feature data according to the initial EEG signal feature data.

[0101] Among them, a reference electrode is introduced, and the potential difference is used to ensure the EEG signal, which can reduce the influence of noise. The selection of the reference electrode includes but is not limited to the following methods: Using a unipolar lead, an active electrode and an indifferent electrode are arranged on the subject to be tested. The indifferent electrode can be arranged at positions such as the earlobe, etc. Equivalent to using the indifferent electrode as the reference electrode, the collected multi-channel EEG monitoring data includes the potential difference of each channel relative to the indifferent electrode, and the initial EEG signal feature data is obtained after preprocessing it. Using a bipolar lead method, no indifferent electrode is arranged on the subject to be tested, and each active electrode uses other active electrodes as the reference electrode. Then the collected multi-channel EEG monitoring data includes the potential difference of each channel relative to other active electrodes, and the initial EEG signal feature data is obtained after preprocessing it. Average reference: An active electrode and a ground electrode are arranged on the subject to be tested. Then the collected multi-channel EEG monitoring data includes the potential difference of each channel relative to the ground terminal. One or more of the electrodes in the multi-channel EEG monitoring data can be used as the reference electrode. Calculating the potential difference between each channel and the reference electrode from the preprocessed multi-channel EEG monitoring data to obtain the initial EEG signal feature data. For example, if c electrodes of the EEG machine are arranged on the scalp of the subject to be tested, any one of the c electrodes can be selected as the reference electrode, or multiple of them can be used as the reference electrode. The average value of the preprocessed multi-channel EEG monitoring data is calculated as the reference value, and the difference between the preprocessed multi-channel EEG monitoring data of each channel and this reference value is calculated to obtain the initial EEG signal feature data. In one embodiment, different electrodes can be selected as the reference electrode, and the potential difference of each channel is calculated respectively under different reference electrodes, and then the average value, etc. is calculated as the initial EEG signal feature data.

[0102] After obtaining the initial EEG signal feature data, the initial EEG signal feature data can be used as the biomedical feature data obtained in step S210, or the initial EEG signal feature data can be further processed, such as further extracting the effective information therein to obtain the biomedical feature data.

[0103] In one embodiment, the biomedical feature data may include EEG signal waveform features. The extraction of the biomedical feature data according to the initial EEG signal feature data may include the following steps:

[0104] Use a pre-trained waveform feature extraction model to process the initial EEG signal feature data to extract the EEG signal waveform features.

[0105] Among them, the waveform feature extraction model may be a pre-trained model with a structure such as Transformer, which can extract the temporal features and other aspects of features in the initial EEG signal feature data. Exemplarily, the initial EEG signal feature data can be input into the waveform feature extraction model, and the waveform feature extraction model performs processing such as embedding on the initial EEG signal feature data to obtain the EEG signal waveform features.

[0106] In one embodiment, the biomedical feature data may include EEG signal image features. Image features can be extracted from the EEG signal map that has been preprocessed and / or has the potential difference calculated based on the reference electrode. For example, the EEG signal map can be input into a pre-trained model such as a convolutional neural network to obtain the EEG signal image features.

[0107] In one embodiment, the biomedical feature data may include EEG signal time-frequency features. The extraction of the biomedical feature data according to the initial EEG signal feature data may include the following steps:

[0108] Perform time-frequency transformation on the initial EEG signal feature data to obtain the EEG signal time-frequency data corresponding to the initial EEG signal feature data;

[0109] Extract the EEG signal time-frequency features according to the EEG signal time-frequency data.

[0110] Among them, the initial EEG signal feature data is usually data in the time domain, which expresses the change of the electrical signal over time. Relatively speaking, the information is relatively single. Through time-frequency transformation, the initial EEG signal feature data can be transformed into the time-frequency joint domain to obtain the EEG signal time-frequency data corresponding to the initial EEG signal feature data, which can provide the joint distribution information of the signal in the time domain and the frequency domain. The methods of time-frequency transformation include but are not limited to short-time Fourier transform, wavelet transform, etc. The present disclosure does not limit which specific method is adopted. Further, the EEG signal time-frequency features can be extracted from the EEG signal time-frequency data. For example, image features can be extracted from the time-frequency diagram, or statistical and feature extraction can be performed on the EEG signal time-frequency data to obtain the EEG signal time-frequency features.

[0111] Figure 5 FIG. shows a schematic diagram of extracting feature data from multi-channel EEG monitoring data. Exemplarily, the biomedical monitoring data includes multi-channel EEG monitoring data. The multi-channel EEG monitoring data can be preprocessed first, and then the potential difference between each channel and the reference electrode of the preprocessed multi-channel EEG monitoring data can be calculated to obtain the initial EEG signal feature data. Next, on the one hand, the initial EEG signal feature data is input into a pre-trained waveform feature extraction model, such as Transformer, etc., to obtain the EEG signal waveform features. On the other hand, the initial EEG signal feature data is subjected to short-time Fourier transform to obtain the EEG signal time-frequency data, and then the EEG signal time-frequency data is input into a pre-trained time-frequency feature extraction model, such as EfficientNetv2, to obtain the EEG signal time-frequency features.

[0112] In addition to the above EEG signal waveform features and EEG signal time-frequency features, other aspects of EEG feature data, such as EEG statistical feature data, etc., can also be extracted.

[0113] In addition, the method of extracting EEG feature data can be adopted to extract biomedical feature data from biomedical monitoring data such as electrocardiogram (ECG), electromyogram (EMG), electrooculogram (EOG), electrogastrogram (EGG), etc.

[0114] Exemplarily, the biomedical monitoring data can include multi-channel ECG monitoring data. The potential difference between each channel and the reference electrode of the preprocessed multi-channel ECG monitoring data can be calculated to obtain the initial ECG signal feature data; the biomedical feature data is extracted according to the initial ECG signal feature data. Among them, the biomedical feature data includes the ECG signal waveform features. A pre-trained waveform feature extraction model can be used to process the initial ECG signal feature data to extract the ECG signal waveform features. The biomedical feature data includes the ECG signal time-frequency features. The initial ECG signal feature data can be subjected to time-frequency transformation to obtain the ECG signal time-frequency data corresponding to the ECG signal feature data; the ECG signal time-frequency features are extracted according to the ECG signal time-frequency data.

[0115] Exemplarily, the biomedical monitoring data may include multi-channel electromyography (EMG) monitoring data. The potential difference between each channel and the reference electrode can be calculated from the preprocessed multi-channel EMG monitoring data to obtain the initial EMG signal feature data; the biomedical feature data can be extracted based on the initial EMG signal feature data. Among them, the biomedical feature data includes the waveform features of the EMG signal. The waveform feature extraction model pre-trained can be used to process the initial EMG signal feature data to extract the waveform features of the EMG signal. The biomedical feature data includes the time-frequency features of the EMG signal. Time-frequency transformation can be performed on the initial EMG signal feature data to obtain the time-frequency data of the EMG signal corresponding to the EMG signal feature data; the time-frequency features of the EMG signal can be extracted based on the time-frequency data of the EMG signal.

[0116] In one implementation, after obtaining the original biomedical monitoring data, the biomedical monitoring data can be segmented with a certain time length (such as 4 seconds), for example, obtaining biomedical monitoring data segments in units of 4 seconds, and preprocessing the biomedical monitoring data of each segment respectively, and extracting features from the preprocessed biomedical monitoring data to obtain the biomedical feature data of each segment. Subsequently, taking the segment as a unit, it is detected whether abnormal discharges occur in each segment of the subject under test.

[0117] Continue to refer to Figure 2 , in step S220, video monitoring data of the subject under test is obtained.

[0118] The video monitoring data can be obtained by shooting with a video acquisition device and can be the video of the subject under test taken within a period of time. In one implementation, the video monitoring data may include face video monitoring data and body video monitoring data, which can be respectively collected by the dual-channel cameras of the video acquisition device, or the single-channel camera of the video acquisition device can simultaneously capture the face and body of the subject under test to obtain the video monitoring data, and then the face video monitoring data and the body video monitoring data are separated from the video monitoring data through methods such as picture cropping.

[0119] In this exemplary implementation, after obtaining the video monitoring data of the subject under test, two aspects of processing can be performed. On the one hand, the action information of the subject under test is detected, and on the other hand, the image sequence of interest is extracted. These two aspects of processing are respectively executed in steps S230 and S240.

[0120] Continue to refer to Figure 2 , in step S230, the action information of the subject under test is detected according to the video monitoring data.

[0121] Among them, the action information is used to characterize the part of the test subject where the movement occurs, or to characterize what action the test subject has taken. From the video monitoring data, the dynamic changes of the body parts of the test subject can be detected, and the action information of the test subject can be identified therefrom. Alternatively, the position information or position change information of the key points of the test subject can be used as the action information of the test subject. Exemplarily, the action information of the test subject may include: the key points where the movement occurs in the face and body of the test subject and the displacement information of the key points.

[0122] Continue to refer to Figure 2 , in step S240, an image sequence of interest for characterizing the actions of the test subject is extracted from the video monitoring data.

[0123] Among them, the image sequence of interest may be an image sequence of the region of interest (ROI) in the video monitoring data, or an image sequence of the frames of interest. Exemplarily, the face and body of the user can be recognized from the video monitoring data, and multiple frames of face region images and multiple frames of body region images can be cropped to form an image sequence of interest. Alternatively, the frames of interest where the test subject has dynamic changes can be first recognized from the video monitoring data, and then the body part regions where the test subject has dynamic changes, that is, the regions of interest, can be recognized from the frames of interest, and the images of the regions of interest can be cropped from the frames of interest to form an image sequence of interest.

[0124] In one implementation, the video monitoring data includes face video monitoring data. For example, it can be the video data collected by a dedicated camera for shooting the face in the video acquisition device, or the video data of the face region cropped from the video monitoring data. Correspondingly, the above-mentioned detection of the action information of the test subject according to the video monitoring data may include the following steps:

[0125] Detect face key point data from the face video monitoring data;

[0126] Obtain the face action information of the test subject according to the face key point data.

[0127] Figure 6 Shows a schematic diagram of face key points and body key points. The 33 key points of the face and body can be numbered, and are respectively denoted as key points 0 to 32. Among them, key points 0 to 10 are face key points, and key points 11 to 32 are body key points. Of course, the present disclosure does not limit the number of key points, and can be increased or decreased according to specific situations.

[0128] Facial key point data may include the positions of facial key points at different times. The facial key point data can be detected by a pre-trained face detection model or by using a face detection algorithm. Exemplarily, one or more frames of images can be intercepted from facial video monitoring data, and the images can be analyzed through object detection and key point detection algorithms to detect the positions of facial key points. For example, the positions of facial key points can be determined by detecting the positions of facial parts such as eyes, nose, mouth, ears, etc., thereby obtaining the facial key point data.

[0129] The facial key point data represents the static positions of facial key points at one or more moments, and the position change information of the facial key points can be analyzed therefrom and used as the facial action information of the subject to be tested. Or further identify what actions the face has made, such as identifying actions such as blinking, shaking the head, frowning, opening the mouth, etc., and using the recognition result as the facial action information of the subject to be tested.

[0130] In one implementation, obtaining the facial action information of the subject to be tested based on the facial key point data may include the following steps:

[0131] Based on the facial key point data, determine the facial key points that have moved and their displacement information to obtain the facial action information of the subject to be tested.

[0132] Among them, the facial key point data can represent the positions of facial key points at different times. Based on this, the facial key points that have moved within a period of time can be determined, and the identifiers of these facial key points (such as the key point numbers in Figure 6 can be recorded) and their displacement information to form the facial action information of the subject to be tested.

[0133] In one implementation, the positional relationship between different facial key points can be determined based on the facial key point data to generate the facial action data to be recognized; the facial action data to be recognized is matched with the facial action data of multiple standard facial actions, and the facial action information corresponding to the facial key point data is determined according to the matching result. Among them, the facial action data to be recognized can be the sequence data of the positional relationship of the facial key points. For example, the positional relationship between the facial key points can be characterized as a vector, and different dimensions of the vector represent the distance, orientation, etc. between different key points. According to the facial key point data of each frame, the facial key point positional relationship data of each frame can be correspondingly generated in the form of a vector. The facial key point positional relationship data of a single frame is used as the facial action data to be recognized and matched with the facial action data of the pre-set standard facial actions. The facial action data of the standard facial actions can also be a vector of the facial key point positional data. By calculating the similarity between the vectors, the facial action data of the standard facial action that matches the facial action data to be recognized is obtained. If the similarity between the two reaches the similarity threshold, it is determined that the facial action data to be recognized corresponds to the standard facial action, and thus the facial action information corresponding to the facial key point data is obtained. Or, the facial key point positional relationship data of different frames is formed into a sequence as the facial action data to be recognized. The facial action data of the standard facial actions can also be a sequence, and the facial action information corresponding to the facial key point data is recognized through the matching between the sequences.

[0134] In one implementation, the sequence of interest images includes a sequence of facial images. The above-mentioned extraction of the sequence of interest images for characterizing the actions of the subject from the video monitoring data may include the following steps:

[0135] According to the facial key point data, the facial region images are cropped from multiple frames of the facial video monitoring data to obtain a sequence of facial part images.

[0136] Among them, the facial region image is an image containing the entire face. The facial video monitoring data may contain picture content other than the face, such as the environmental picture around the subject, etc. After obtaining the facial key point data, the position of the facial region can be determined, and the facial region images are cropped from multiple frames of the facial video monitoring data. The facial region images of different frames form a sequence of facial images.

[0137] By cropping the facial region images to obtain a sequence of facial images, the information irrelevant to the face in the facial video monitoring data can be eliminated, and the cropping process is simple, which is beneficial to reducing the calculation amount and improving the accuracy of the subsequent detection results.

[0138] In one embodiment, after obtaining the face video monitoring data, the face video monitoring data can be segmented with a certain time length (such as 4 seconds, which can be the same as the time length of the biomedical monitoring data segmentation), for example, obtaining face video monitoring data segments with 4 seconds as a unit. In the face video monitoring data of each segment, determine one or more key frames, such as the first frame, the midpoint frame, the last frame, etc. Detect the face key points in the key frames to obtain the displacement information of the face key points, thereby forming the face motion information of the subject under test. And determine the position of the face area according to the detected face key points, and intercept the face area image from the face video monitoring data to form a face image sequence. Subsequently, taking the segment as a unit, input the face motion information and the face image sequence together with other information into the abnormal discharge detection model to detect whether abnormal discharge occurs in each segment of the subject under test.

[0139] In one embodiment, the sequence of images of interest includes a sequence of face part images. The above-mentioned extraction of the sequence of images of interest for characterizing the actions of the subject under test from the video monitoring data may include the following steps:

[0140] According to the face key point data, crop the face part region images from multiple frames of the face video monitoring data to obtain a sequence of face part images.

[0141] Among them, the face part region image is an image containing one or more face parts. The face video monitoring data may include the entire face, and may also include picture content other than the face, such as the environmental picture around the subject under test, etc. After obtaining the face key point data, the positions of the face parts can be determined, such as the positions of parts such as eyes, nose, mouth, ears, etc., the region containing all face parts can be determined (such as the entire facial region, excluding hair, etc.), or the region of each face key part can be determined (such as the eye region, nose region, mouth region, ear region), and the face part region images are cropped. The face part region images of different frames form a sequence of face part images.

[0142] By cropping the face part region images to obtain a sequence of face part images, information irrelevant to the face or face motion in the face video monitoring data can be removed, which is beneficial to reducing the calculation amount and improving the accuracy of subsequent detection results.

[0143] In one embodiment, the above-mentioned cropping of the face part region images from multiple frames of the face video monitoring data according to the face key point data may include the following steps:

[0144] According to the face key point data, determine the face parts that are moving, and crop the region images of the face parts that are moving from multiple frames of the face video monitoring data.

[0145] Since the facial key-point data represents the static positions of the facial key points at one or more moments, by combining the facial key-point data of different frames, it is possible to determine which facial parts have moved (i.e., there are dynamic changes in position). Thus, the regional images of the moving facial parts are intercepted from the facial video monitoring data, and the above-mentioned regional images of the facial parts are obtained. For example, according to the facial key-point data, it can be determined that the key points of the eyes and nose parts have dynamic changes (usually position changes), while the key points of other parts such as the mouth and ears have not changed dynamically, indicating that the eyes and nose parts have moved. According to the position information of the key points of the eyes and nose parts in the facial key-point data, the bounding boxes of the eyes and nose parts can be generated from the facial video monitoring data (it can be a bounding box that includes both the eyes and the nose at the same time, or a bounding box that includes the eyes and a bounding box that includes the ears), and the image within the bounding box is cropped. It is also possible to appropriately enlarge the bounding box (such as multiplying the width and height of the bounding box by a scale factor greater than 1, and the scale factor can be 1.1, 1.2, etc., which can be determined according to experience or specific requirements), and intercept the image within the enlarged bounding box to obtain the regional image of the facial part.

[0146] It should be understood that the above-mentioned cropping process can be performed on each frame of the facial video monitoring data. For example, according to the facial key-point data of each frame, the facial parts that move in each frame are determined, and then the corresponding regional images of the facial parts are cropped. Or, the above-mentioned cropping process can be performed only on some frames of the facial video monitoring data, without processing each frame, thereby reducing the number and processing volume of the regional images of the facial parts. Exemplarily, after obtaining the facial video monitoring data, key-frame images can be extracted. For example, one frame is extracted at regular intervals of the number of frames, or the difference between adjacent frames is detected. When the difference reaches a predetermined value, the latter frame of the adjacent frames is extracted as the key-frame image. Facial key-point detection is performed on the key-frame images to obtain facial key-point data. According to the facial key-point data, the facial parts that move in the key-frame images are determined, and the regional images of the facial parts are cropped. Or, for the facial video monitoring data within a certain period of time (usually the unit detection duration, such as the duration can be determined according to the number of images that the abnormal discharge detection model can process), according to the facial key-point data of each frame, the facial parts that move are determined, and then the regional images of the moving facial parts are cropped for each frame or key frame to obtain the regional images of the facial parts.

[0147] Since facial actions are mainly reflected in the moving facial parts, extracting the regional images of the moving facial parts from the facial video monitoring data can focus the subsequent processing on the moving facial parts. Especially in the case of abnormal brain discharges, facial actions are usually relatively subtle expressions. By detecting the regional images of the moving facial parts, it is easier to detect the detailed information therein, further improving the accuracy of the subsequent detection results and reducing the computational amount.

[0148] In one embodiment, the video monitoring data includes body video monitoring data, which can be the video data collected by a dedicated camera for shooting the body in the video acquisition device, or the video data of the body area (excluding the face) cropped from the video monitoring data. Correspondingly, the above method for detecting the motion information of the subject according to the video monitoring data may include the following steps:

[0149] Detect body key point data from the body video monitoring data;

[0150] Obtain the body motion information of the subject according to the body key point data.

[0151] Among them, the body key points can be referred to Figure 6 as shown. The body key point data may include the positions of the body key points at different times. The body key point data can be detected by a pre-trained body detection model or by using a limb detection algorithm. Exemplarily, one or more frames of images can be intercepted from the body video monitoring data, and the images can be analyzed through object detection and key point detection algorithms to detect the positions of the body key points. For example, the positions of the body key points can be determined by detecting the positions of body parts such as the neck, chest, and limbs, thereby obtaining the body key point data.

[0152] The body key point data represents the static positions of the body key points at one or more frames of moments. The position change information of the body key points can be analyzed from it, and used as the body motion information of the subject, or further identify what actions the body has made, such as identifying actions like raising the hand, clapping, shaking, etc., and taking the recognition result as the body motion information of the subject.

[0153] In one embodiment, the above method for obtaining the body motion information of the subject according to the body key point data may include the following steps:

[0154] According to the body key point data, determine the body key points that have moved and their displacement information, and obtain the body motion information of the subject.

[0155] Among them, the body key point data can represent the positions of the body key points at different times. Based on this, the body key points that have moved within a period of time can be determined, and the identifiers of these body key points (such as the key point numbers in Figure 6 ) and their displacement information can be recorded to form the body motion information of the subject.

[0156] In one embodiment, the positional relationship between different body key points can be determined based on the body key point data to generate the body motion data to be recognized. The body motion data to be recognized is then matched with the body motion data of multiple standard body motions, and the body motion information corresponding to the body key point data is determined according to the matching result. Among them, the body motion data to be recognized can be the sequence data of the positional relationship of body key points. For example, the positional relationship between body key points can be characterized as a vector, and different dimensions of the vector represent the distance, orientation, etc. between different key points. According to the body key point data of each frame, the positional relationship data of body key points of each frame can be correspondingly generated in the form of a vector. Taking the positional relationship data of body key points of a single frame as the body motion data to be recognized, it is matched with the body motion data of the preset standard body motions. The body motion data of the standard body motions can also be a vector of body key point position data. By calculating the similarity between vectors, the body motion data of the standard body motion that matches the body motion data to be recognized is obtained. If the similarity between the two reaches the similarity threshold, it is determined that the body motion data to be recognized corresponds to the standard body motion, and thus the body motion information corresponding to the body key point data is obtained. Alternatively, the positional relationship data of body key points of different frames are formed into a sequence as the body motion data to be recognized, and the body motion data of the standard body motions can also be a sequence. The body motion information corresponding to the body key point data is recognized through the matching between sequences.

[0157] In one embodiment, the sequence of images of interest includes the sequence of body images. The above-mentioned extraction of the sequence of images of interest for characterizing the actions of the subject from the video monitoring data may include the following steps:

[0158] According to the body key point data, the body region images are cropped from multiple frames of the body video monitoring data to obtain the sequence of body part images.

[0159] Among them, the body region image is an image containing the entire body. The body video monitoring data may contain picture content other than the body, such as the environmental picture around the subject, etc. After obtaining the body key point data, the position of the body region can be determined, and the body region images are cropped from multiple frames of the body video monitoring data. The body region images of different frames form the sequence of body images.

[0160] By cropping the body region images to obtain the sequence of body images, the information unrelated to the body in the body video monitoring data can be eliminated, and the cropping process is simple, which is beneficial to reducing the calculation amount and improving the accuracy of subsequent detection results.

[0161] In one embodiment, after acquiring the body video monitoring data, the body video monitoring data can be segmented with a certain time length (such as 4 seconds, which can be the same as the time length of the biomedical monitoring data segmentation), for example, obtaining body video monitoring data segments with 4 seconds as a unit. In the body video monitoring data of each segment, determine one or more key frames, such as the first frame, the midpoint frame, the last frame, etc. Detect body key points in the key frames to obtain the displacement information of the body key points, thereby forming the body motion information of the subject under test. And determine the position of the body area according to the detected body key points, and intercept the body area image from the body video monitoring data to form a body image sequence. Subsequently, taking the segment as a unit, input the body motion information and the body image sequence together with other information into the abnormal discharge detection model to detect whether abnormal discharge occurs in each segment of the subject under test.

[0162] In one embodiment, the sequence of images of interest includes the sequence of body part images. The above-mentioned extraction of the sequence of images of interest for characterizing the actions of the subject under test from the video monitoring data may include the following steps:

[0163] According to the body key point data, crop the body part area images from multiple frames of the body monitoring video data to obtain the sequence of body part images.

[0164] Among them, the body part area image is an image containing one or more body parts. The body video monitoring data may include the entire body and may also include the content of the picture outside the body, such as the environmental picture around the subject under test. After obtaining the body key point data, the positions of the body parts can be determined, such as the positions of parts such as the neck, chest, left arm, left hand, right hand, right arm, left leg, left foot, right leg, right foot, etc. The area containing all body parts can be determined, or the area of each body key part (such as the neck area, left arm area, left hand area) can be determined, and the body part area images are cropped. The body part area images of different frames form the sequence of body part images.

[0165] By cropping the body part area images to obtain the sequence of body part images, the information irrelevant to the body or the body motion in the body video monitoring data can be removed, which is beneficial to reducing the calculation amount and improving the accuracy of the subsequent detection results.

[0166] In one embodiment, the above-mentioned cropping of the body part area images from multiple frames of the body monitoring video data according to the body key point data may include the following steps:

[0167] According to the body key point data, determine the body parts that are moving, and crop the area images of the body parts that are moving from multiple frames of the body monitoring video data.

[0168] Since the body key point data represents the static positions of the body key points at one or more moments in a frame, by combining the body key point data of different frames, it can be determined which body parts have moved (i.e., there are dynamic changes in positions). Thus, the regional images of the body parts that have moved are intercepted from the body video monitoring data, that is, the above-mentioned regional images of the body parts are obtained. For example, according to the body key point data, it can be determined that the key points of the eye and nose parts have dynamic changes (generally position changes), while the key points of other parts such as the mouth and ears have not changed dynamically, indicating that the eye and nose parts have moved. According to the position information of the key points of the eye and nose parts in the body key point data, the bounding boxes of the eye and nose parts can be generated from the body video monitoring data (it can be a bounding box that contains both the eye and the nose at the same time, or a bounding box that contains the eye and a bounding box that contains the ear), and the image within the bounding box is cropped. It is also possible to appropriately enlarge the bounding box (such as multiplying the width and height of the bounding box by a scale factor greater than 1, and the scale factor can be 1.1, 1.2, etc., which can be determined according to experience or specific requirements), and intercept the image within the enlarged bounding box to obtain the regional image of the body part.

[0169] It should be understood that the above-mentioned cropping process can be performed on each frame of the body video monitoring data. For example, according to the body key point data of each frame, the body parts that have moved in each frame are determined, and then the corresponding regional images of the body parts are cropped. Or, the above-mentioned cropping process can be performed only on some frames of the body video monitoring data without processing each frame, thereby reducing the number and processing amount of the regional images of the body parts. Exemplarily, after obtaining the body video monitoring data, key frame images can be extracted. For example, one frame is extracted at regular intervals of the number of frames, or the difference between adjacent frames is detected. When the difference reaches a predetermined value, the latter frame of the adjacent frames is extracted as the key frame image. The body key points are detected for the key frame images to obtain the body key point data. According to the body key point data, the body parts that have moved in the key frame images are determined, and the regional images of the body parts are cropped. Or, for the body video monitoring data within a certain period of time (usually the unit detection duration, such as the duration can be determined according to the number of images that the abnormal discharge detection model can process), according to the body key point data of each frame, the body parts that have moved are determined, and then the regional images of the body parts that have moved are cropped for each frame or key frame to obtain the regional images of the body parts.

[0170] Since body movements are mainly reflected in the body parts that have moved, extracting the regional images of the body parts that have moved from the body video monitoring data can focus the subsequent processing on the body parts that have moved. Especially in the case of abnormal discharges in the human brain, body movements are usually relatively subtle. By detecting the regional images of the body parts that have moved, it is easier to detect the detailed information therein, further improving the accuracy of the subsequent detection results and reducing the computational amount.

[0171] Figure 7 A schematic diagram showing the processing of video monitoring data is presented. After acquiring the video monitoring data, the face video monitoring data and the body video monitoring data are separated. The face video monitoring data is input into a pre-trained face detection model to obtain face key point data; the face action information is obtained based on the face key point data; and a face image sequence is cropped from the face video monitoring data according to the face key point data. The body video monitoring data is input into a pre-trained body detection model to obtain body key point data; the body action information is obtained based on the body key point data; and a body image sequence is cropped from the body video monitoring data according to the body key point data.

[0172] Continue to refer to Figure 2 , in step S250, a pre-trained abnormal discharge detection model is used to process the biomedical feature data, the action information, and the image sequence of interest to obtain the abnormal discharge detection result of the subject under test.

[0173] The biomedical feature data, the action information, and the image sequence of interest are data of different modalities. The abnormal discharge detection model can comprehensively process the three types of data to obtain the final abnormal discharge detection result. Information missing can be mutually compensated among different types of data. For example, when the subject under test makes some actions such as blinking or speaking, it may affect the biomedical feature data such as electroencephalogram, resulting in misjudgment. By making up for the information that cannot be reflected in the biomedical feature data through the action information or the image sequence of interest, the situation of misjudgment can be reduced.

[0174] In one implementation, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer. The feature processing layer, the attention layer, and the classification layer are the three main parts of the abnormal discharge detection model, and each part may include one or more intermediate layers.

[0175] Refer to Figure 8 As shown, the above-mentioned use of a pre-trained abnormal discharge detection model to process the biomedical feature data, the action information, and the image sequence of interest to obtain the abnormal discharge detection result of the subject under test may include the following steps S810 to S840:

[0176] Step S810, input the biomedical feature data, the action information, and the image sequence of interest into the abnormal discharge detection model;

[0177] Step S820, use the feature processing layer to extract action feature data from the action information, extract image feature data from the image sequence of interest, and fuse the biomedical feature data, the action feature data, and the image feature data to obtain a fused feature;

[0178] Step S830: Characterize the fused features using an attention layer to obtain embedded features;

[0179] Step S840: Map the embedded features to the output space using a classification layer to obtain the abnormal discharge detection result of the subject under test.

[0180] Among them, the feature processing layer can directly use the action information as action feature data. For example, both the action information and the action feature data can be action recognition results. Alternatively, the action information includes the position information of facial key points and / or body key points. The feature processing layer can further process the action information through fully connected, attention, etc. methods to extract action feature data. The feature processing layer can perform convolution and other processing on the image sequence of interest to extract image feature data. Exemplarily, the feature processing layer can include neural network units such as LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), and CNN (Convolutional Neural Network), which can process the image sequence of interest and extract image feature data. The feature processing layer can also perform feature fusion using MLP (Multi-Layer Perceptron), concatenation, etc. Exemplarily, the feature processing layer can include one or more fully connected layers and a concatenation layer. The biomedical feature data, action feature data, and image feature data are aligned in feature dimensions through the fully connected layer, and then a concatenation operation is performed through the concatenation layer to obtain the fused features.

[0181] In the attention layer, the fused features can be selected. For example, the fused features are re-characterized through attention weights to obtain the embedded features. The embedded features can be dense features, and each dimension of which can fuse information of different modalities.

[0182] The classification layer can include a fully connected layer, etc. By performing a fully connected operation on the embedded features, they are gradually mapped to the input space. Activation functions such as sigmoid (S-shaped function) and softmax (normalized exponential function) can also be used to obtain the probability value of abnormal discharge detection. This probability value can be used as the final abnormal discharge detection result, or it is determined whether there is abnormal discharge according to the probability value to obtain the final abnormal discharge detection result.

[0183] Figure 9The figure shows a schematic diagram of obtaining a training data set. Exemplarily, biomedical monitoring sample data is obtained. For example, electroencephalogram signals can be collected by an electroencephalograph to obtain multi-channel electroencephalogram monitoring sample data. Preprocessing is performed, and then abnormal discharge annotation is carried out. The continuous data can be cut according to a fixed duration, such as cut into segments with a duration of 4 seconds, classified into positive and negative samples, and the time periods of each sample in the original continuous data are recorded. The segments containing abnormal discharge annotation are extracted as positive samples, and the segments without abnormal discharge annotation are extracted as negative samples. Samples labeled as the seizure period or suspicious (doctors are not sure whether it is abnormal discharge) are discarded. Further, medical feature training data is obtained. The result of whether there is abnormal discharge annotation is used as annotation data. Video monitoring sample data in the same time period as the biomedical monitoring sample data is obtained, which can include video monitoring sample data of the face and body. The face and limbs of the video monitoring sample data are analyzed to detect face key points and body key points, and then action sample information and an interesting image sample sequence are obtained. The biomedical feature training data, action sample information, interesting image sample sequence, and corresponding annotation data in the same time period form a set of supervised data, and a large amount of supervised data forms a training data set.

[0184] Figure 10 The figure shows a schematic diagram of training an abnormal discharge detection model. The biomedical feature training data, action sample information, and interesting image sample sequence in the training data set are input into the abnormal discharge detection model to be trained to obtain corresponding abnormal discharge detection sample data. The loss function value is calculated based on the abnormal discharge detection sample data and the annotation data, and the parameters of the abnormal discharge detection model are updated according to the loss function value. The update process is iteratively executed until the abnormal discharge detection model reaches a predetermined training completion condition, such as the accuracy rate on the validation set or the test set reaches the standard, and a trained abnormal discharge detection model is obtained.

[0185] In one implementation manner, before using the pre-trained abnormal discharge detection model to process biomedical feature data, action information, and interesting image sequences, the human brain abnormal discharge detection method may further include the following steps:

[0186] Align at least one of the action information and the interesting image sequence with the biomedical feature data in time.

[0187] Among them, the biomedical feature data, motion information, and image sequence of interest come from different devices, and there may be a time difference between different devices, resulting in time asynchronization among the biomedical feature data, motion information, and image sequence of interest, which affects the detection result of abnormal brain discharges. For example, the time information of the biomedical feature data comes from the time of the biomedical monitoring device, and the time information of the motion information and the image sequence of interest comes from the time of the video acquisition device. Therefore, it can be considered that the time information of the motion information and the image sequence of interest is consistent, and at least one of the motion information and the image sequence of interest is time-aligned with the biomedical feature data, so as to achieve time consistency of the three types of data.

[0188] The following takes the time alignment of the motion information and the biomedical feature data as an example for illustration.

[0189] In one implementation, the time alignment of at least one of the motion information and the image sequence of interest with the biomedical feature data may include the following steps:

[0190] Detect one or more pieces of biomedical feature data suspected of abnormal discharges and corresponding one or more first time points from the biomedical feature data;

[0191] Detect one or more pieces of motion data suspected of abnormal discharges and corresponding one or more second time points from the motion information;

[0192] Match the biomedical feature data suspected of abnormal discharges with the motion data suspected of abnormal discharges, and determine the corresponding relationship between the first time point and the second time point according to the matching result;

[0193] Based on the corresponding relationship between the first time point and the second time point, determine the time calibration parameter, and use the time calibration parameter to time-align the motion information and the biomedical feature data.

[0194] Among them, abnormal discharges may occur at one or more time points, and these times are the above-mentioned first time point or second time point. Or, abnormal discharges may also occur in one or more time periods, and the start time point and end time point of these time periods can be used as the above-mentioned first time point or second time point. For example, in the biomedical feature data, a time period with an abnormally increased signal value is detected, and the biomedical feature data within this time period can be used as the biomedical feature data suspected of abnormal discharges, and the start time point and end time point of this time point are used as the first time point. Based on a similar method, the motion data suspected of abnormal discharges and the second time point can be determined.

[0195] Match the biomedical feature data of suspected abnormal discharge with the action data of suspected abnormal discharge, and then calculate the time difference between the first time point and the second time point with a corresponding relationship to obtain a time calibration parameter. For example, the biomedical feature data of suspected abnormal discharge at each first time point can be obtained, and the first time point and the second time point closest to it are matched into the same set of time calibration data.

[0196] In one implementation, the matching of the biomedical feature data of suspected abnormal discharge with the action data of suspected abnormal discharge may include the following steps:

[0197] Determine a first relative value between the biomedical feature data of suspected abnormal discharge and other biomedical feature data;

[0198] Determine a second relative value between the action data of suspected abnormal discharge and other action data in the action information;

[0199] By comparing the first relative value with the second relative value, obtain the matching result between the biomedical feature data of suspected abnormal discharge and the action data of suspected abnormal discharge.

[0200] Among them, the first relative value represents the relative difference size between the biomedical feature data of suspected abnormal discharge and the biomedical feature data in the normal state, which can be in the form of a percentage. The second relative value represents the relative difference size between the action data of suspected abnormal discharge and the action data in the normal state. The closest first relative value and second relative value can be matched together to obtain the matching result between the biomedical feature data of suspected abnormal discharge and the action data of suspected abnormal discharge, so as to form a set of time calibration data for the corresponding first time point and second time point. This can improve the accuracy of matching.

[0201] Calculate the time difference between the first time point and the second time point in each set of time calibration data to obtain a time calibration parameter. The time calibration parameters of each group can be averaged to obtain the final time calibration parameter. Exemplarily, with the first time point as the reference, the time calibration parameter includes the time difference between the second time point and the first time point. According to the time difference between the second time point and the first time point, adjust the time stamp of the action information, and the time stamp of the image sequence of interest can also be adjusted. Thus, the time alignment of the biomedical feature data, action information, and image sequence of interest is achieved.

[0202] Figure 11A schematic diagram of abnormal discharge detection in the human brain is shown. Multichannel electroencephalogram monitoring data is collected by an electroencephalograph, and after preprocessing and feature extraction, biomedical feature data is obtained; face video monitoring data and body video monitoring data are collected by a video acquisition device. Facial key point detection is performed on the face video monitoring data, and after further processing, facial motion information and a facial image sequence are obtained. Body key point detection is performed on the body video monitoring data, and after further processing, body motion information and a body image sequence are obtained. The biomedical feature data, facial motion information, facial image sequence, body motion information, and body image sequence are input into a trained abnormal discharge detection model, and an abnormal discharge detection result is output.

[0203] Exemplary Apparatus

[0204] An exemplary embodiment of the present disclosure also provides a device for detecting abnormal discharge in the human brain. Refer to Figure 12 As shown, the device 1200 for detecting abnormal discharge in the human brain may include the following program modules: a first acquisition module 1210 configured to acquire biomedical feature data of a subject; a second acquisition module 1220 configured to acquire video monitoring data of the subject; a motion information detection module 1230 configured to detect the motion information of the subject according to the video monitoring data; an image sequence extraction module 1240 configured to extract an interesting image sequence for characterizing the motion of the subject from the video monitoring data; and a model processing module 1250 configured to process the biomedical feature data, motion information, and interesting image sequence by using a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0205] In one embodiment, acquiring the biomedical feature data of the subject includes: acquiring the biomedical monitoring data of the subject collected by a biomedical monitoring device; preprocessing the biomedical monitoring data; and obtaining the biomedical feature data according to the preprocessed biomedical monitoring data.

[0206] In one embodiment, the biomedical monitoring data includes: multichannel electroencephalogram monitoring data collected from multiple parts of the scalp of the subject; obtaining the biomedical feature data according to the preprocessed biomedical monitoring data includes: calculating the potential difference between each channel and a reference electrode from the preprocessed multichannel electroencephalogram monitoring data to obtain initial electroencephalogram signal feature data; and extracting the biomedical feature data according to the initial electroencephalogram signal feature data.

[0207] In one embodiment, the biomedical feature data includes electroencephalogram (EEG) signal waveform features; extracting the biomedical feature data from the initial EEG feature data includes: processing the initial EEG feature data using a pre-trained waveform feature extraction model to extract the EEG signal waveform features.

[0208] In one embodiment, the biomedical feature data includes EEG signal time-frequency features; extracting the biomedical feature data from the initial EEG feature data includes: performing a time-frequency transform on the initial EEG feature data to obtain the EEG signal time-frequency data corresponding to the initial EEG feature data; extracting the EEG signal time-frequency features from the EEG signal time-frequency data.

[0209] In one embodiment, preprocessing the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical normalization processing.

[0210] In one embodiment, the video monitoring data includes face video monitoring data; detecting the action information of the subject according to the video monitoring data includes: detecting the face key point data from the face video monitoring data; obtaining the face action information of the subject according to the face key point data.

[0211] In one embodiment, the image sequence of interest includes a face image sequence; extracting the image sequence of interest for characterizing the actions of the subject from the video monitoring data includes: cropping the face region images from multiple frames of the face video monitoring data according to the face key point data to obtain the face image sequence.

[0212] In one embodiment, obtaining the face action information of the subject according to the face key point data includes: determining the face key points where movement occurs and their displacement information according to the face key point data to obtain the face action information of the subject.

[0213] In one embodiment, the video monitoring data includes body video monitoring data; detecting the action information of the subject according to the video monitoring data includes: detecting the body key point data from the body video monitoring data; obtaining the body action information of the subject according to the body key point data.

[0214] In one embodiment, the image sequence of interest includes a body image sequence; extracting the image sequence of interest for characterizing the actions of the subject from the video monitoring data includes: cropping the body region images from multiple frames of the body monitoring video data according to the body key point data to obtain the body image sequence.

[0215] In one embodiment, body movement information of a subject is obtained based on body key point data, including: determining body key points where movement occurs and their displacement information according to the body key point data, so as to obtain the body movement information of the subject.

[0216] In one embodiment, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; the pre-trained abnormal discharge detection model is used to process biomedical feature data, movement information, and an image sequence of interest to obtain the abnormal discharge detection result of the subject, including: inputting the biomedical feature data, movement information, and image sequence of interest into the abnormal discharge detection model; using the feature processing layer to extract movement feature data from the movement information, extract image feature data from the image sequence of interest, and fuse the biomedical feature data, movement feature data, and image feature data to obtain a fused feature; using the attention layer to represent the fused feature to obtain an embedded feature; using the classification layer to map the embedded feature to the output space to obtain the abnormal discharge detection result of the subject.

[0217] In one embodiment, the model processing module 1250 is further configured to: before using the pre-trained abnormal discharge detection model to process the biomedical feature data, movement information, and image sequence of interest, perform time alignment on at least one of the movement information and the image sequence of interest with the biomedical feature data.

[0218] In one embodiment, performing time alignment on at least one of the movement information and the image sequence of interest with the biomedical feature data includes: detecting one or more biomedical feature data suspected of abnormal discharge and corresponding one or more first time points from the biomedical feature data; detecting one or more movement data suspected of abnormal discharge and corresponding one or more second time points from the movement information; matching the biomedical feature data suspected of abnormal discharge with the movement data suspected of abnormal discharge, and determining the corresponding relationship between the first time point and the second time point according to the matching result; based on the corresponding relationship between the first time point and the second time point, determining a time calibration parameter, and using the time calibration parameter to perform time alignment on the movement information and the biomedical feature data.

[0219] In one embodiment, matching the biomedical feature data suspected of abnormal discharge with the movement data suspected of abnormal discharge includes: determining a first relative value between the biomedical feature data suspected of abnormal discharge and other biomedical feature data; determining a second relative value between the movement data suspected of abnormal discharge and other movement data in the movement information; obtaining the matching result between the biomedical feature data suspected of abnormal discharge and the movement data suspected of abnormal discharge by comparing the first relative value and the second relative value.

[0220] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the embodiments of the above method, and will not be elaborated here.

[0221] Exemplary Storage Medium

[0222] An exemplary embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method of the present disclosure is implemented. The above method can be implemented by a program product. For example, a portable compact disc read-only memory (CD-ROM) can be adopted and includes program code, and can run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0223] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0224] The computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, and the readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0225] The program code included on the readable medium can be transmitted by any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0226] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0227] Exemplary Electronic Device

[0228] Exemplary embodiments of the present disclosure also provide an electronic device, which may be Figure 1A or Figure 1B any device among them. The electronic device includes a processor and a memory for storing executable instructions of the processor. The processor is configured to execute the above-described method of the present disclosure by executing the executable instructions.

[0229] Refer to Figure 13 for a description of the electronic device according to the exemplary embodiments of the present disclosure. Figure 13 The displayed electronic device 1300 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0230] As Figure 13 shown, the electronic device 1300 is presented in the form of a general-purpose computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, and a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310).

[0231] Among them, the storage unit stores program code, which can be executed by the processing unit 1310, so that the processing unit 1310 executes the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the processing unit 1310 may execute method steps as Figure 2 shown, etc.

[0232] The storage unit 1320 may include a volatile storage unit, such as a random access storage unit (RAM) 1321 and / or a cache storage unit 1322, and may further include a read-only storage unit (ROM) 1323.

[0233] The storage unit 1320 may also include a program / utilities 1324 having a set (at least one) of program modules 1325. Such program modules 1325 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0234] The bus 1330 may include a data bus, an address bus, and a control bus.

[0235] The electronic device 1300 may also communicate with one or more external devices 1400 (such as a keyboard, a pointing device, a Bluetooth device, etc.). Such communication may be carried out through the input / output (I / O) interface 1340. The electronic device 1300 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 1350. As shown in the figure, the network adapter 1350 communicates with other modules of the electronic device 1300 through the bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0236] It should be noted that, although several modules or sub-modules of the device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more of the above-described units / modules may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided and embodied by multiple units / modules.

[0237] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0238] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefit. Such a division is only for the convenience of expression. The present disclosure aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for detecting abnormal electrical discharges in the human brain, characterized in that, it includes: Obtain the biomedical characteristic data of the subject; Obtain the video monitoring data of the subject; Detect the action information of the subject according to the video monitoring data; Extract an interesting image sequence from the video monitoring data for characterizing the actions of the subject; Use a pre-trained abnormal discharge detection model to process the biomedical characteristic data, the action information, and the interesting image sequence to obtain the abnormal discharge detection result of the subject; Wherein, before using the pre-trained abnormal discharge detection model to process the biomedical characteristic data, the action information, and the interesting image sequence, the method further includes: Detect one or more biomedical characteristic data suspected of abnormal discharges and corresponding one or more first time points from the biomedical characteristic data; Detect one or more action data suspected of abnormal discharges and corresponding one or more second time points from the action information; Determine the first relative value between the biomedical characteristic data suspected of abnormal discharges and other biomedical characteristic data; determine the second relative value between the action data suspected of abnormal discharges and other action data in the action information; by comparing the first relative value and the second relative value, obtain the matching result between the biomedical characteristic data suspected of abnormal discharges and the action data suspected of abnormal discharges, and determine the corresponding relationship between the first time point and the second time point according to the matching result; Based on the corresponding relationship between the first time point and the second time point, determine the time calibration parameter, and use the time calibration parameter to align the time of the action information and the biomedical characteristic data.

2. The method according to claim 1, characterized in that, The obtaining of the biomedical characteristic data of the subject includes: Obtain the biomedical monitoring data of the subject collected by a biomedical monitoring device; Preprocess the biomedical monitoring data; Obtain the biomedical characteristic data according to the preprocessed biomedical monitoring data.

3. The method according to claim 2, characterized in that, The biomedical monitoring data includes: multi-channel electroencephalogram monitoring data collected from multiple parts of the scalp of the subject; the obtaining of the biomedical characteristic data according to the preprocessed biomedical monitoring data includes: Calculate the potential difference between each channel and the reference electrode for the preprocessed multi-channel electroencephalogram monitoring data to obtain the initial electroencephalogram signal characteristic data; Extract the biomedical characteristic data according to the initial electroencephalogram signal characteristic data.

4. The method according to claim 3, characterized in that, The biomedical characteristic data includes electroencephalogram signal waveform characteristics; the extracting of the biomedical characteristic data according to the initial electroencephalogram signal characteristic data includes: Use a pre-trained waveform characteristic extraction model to process the initial electroencephalogram signal characteristic data to extract the electroencephalogram signal waveform characteristics.

5. The method according to claim 3, wherein, the biomedical feature data includes the time-frequency features of electroencephalogram signals; the extracting of the biomedical feature data from the initial electroencephalogram feature data includes: performing time-frequency transformation on the initial electroencephalogram feature data to obtain the time-frequency data of the electroencephalogram signals corresponding to the initial electroencephalogram feature data; extracting the time-frequency features of the electroencephalogram signals according to the time-frequency data of the electroencephalogram signals.

6. The method according to claim 2, wherein, the preprocessing of the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical normalization processing.

7. The method according to claim 1, wherein, the video monitoring data includes face video monitoring data; the detecting of the action information of the subject according to the video monitoring data includes: detecting face key point data from the face video monitoring data; obtaining the face action information of the subject according to the face key point data.

8. The method according to claim 7, wherein, the image sequence of interest includes a face image sequence; the extracting of the image sequence of interest for characterizing the action of the subject from the video monitoring data includes: cropping face region images from multiple frames of the face video monitoring data according to the face key point data to obtain a face image sequence.

9. The method according to claim 7, wherein, the obtaining of the face action information of the subject according to the face key point data includes: determining the face key points where movement occurs and their displacement information according to the face key point data to obtain the face action information of the subject.

10. The method according to claim 1, wherein, the video monitoring data includes body video monitoring data; the detecting of the action information of the subject according to the video monitoring data includes: detecting body key point data from the body video monitoring data; obtaining the body action information of the subject according to the body key point data.

11. The method according to claim 10, wherein, the image sequence of interest includes a body image sequence; the extracting of the image sequence of interest for characterizing the action of the subject from the video monitoring data includes: cropping body region images from multiple frames of the body video monitoring data according to the body key point data to obtain a body image sequence.

12. The method according to claim 10, wherein, the obtaining of the body action information of the subject according to the body key point data includes: determining the body key points where movement occurs and their displacement information according to the body key point data to obtain the body action information of the subject.

13. The method according to claim 1, wherein, The abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; the method of using the pre-trained abnormal discharge detection model to process the biomedical feature data, the motion information, and the image sequence of interest to obtain the abnormal discharge detection result of the subject includes: Input the biomedical feature data, the motion information, and the image sequence of interest into the abnormal discharge detection model; Use the feature processing layer to extract motion feature data from the motion information, extract image feature data from the image sequence of interest, and fuse the biomedical feature data, the motion feature data, and the image feature data to obtain a fused feature; Use the attention layer to characterize the fused feature to obtain an embedded feature; Use the classification layer to map the embedded feature to the output space to obtain the abnormal discharge detection result of the subject.

14. An abnormal discharge detection device for the human brain, characterized in that, it includes: A first acquisition module configured to acquire biomedical feature data of a subject; A second acquisition module configured to acquire video monitoring data of the subject; A motion information detection module configured to detect the motion information of the subject according to the video monitoring data; An image sequence extraction module configured to extract an image sequence of interest for characterizing the motion of the subject from the video monitoring data; A model processing module configured to use a pre-trained abnormal discharge detection model to process the biomedical feature data, the motion information, and the image sequence of interest to obtain the abnormal discharge detection result of the subject; wherein, the model processing module is further configured to: Before using the pre-trained abnormal discharge detection model to process the biomedical feature data, the motion information, and the image sequence of interest, detect one or more biomedical feature data suspected of abnormal discharge and corresponding one or more first time points from the biomedical feature data; Detect one or more motion data suspected of abnormal discharge and corresponding one or more second time points from the motion information; Determine a first relative value between the biomedical feature data suspected of abnormal discharge and other biomedical feature data; determine a second relative value between the motion data suspected of abnormal discharge and other motion data in the motion information; by comparing the first relative value and the second relative value, obtain a matching result between the biomedical feature data suspected of abnormal discharge and the motion data suspected of abnormal discharge, and determine the corresponding relationship between the first time point and the second time point according to the matching result; Based on the corresponding relationship between the first time point and the second time point, determine a time calibration parameter, and use the time calibration parameter to align the time of the motion information and the biomedical feature data.

15. The device according to claim 14, characterized in that, The obtaining of the biomedical characteristic data of the subject to be tested includes: obtaining the biomedical monitoring data of the subject to be tested collected by a biomedical monitoring device; preprocessing the biomedical monitoring data; and obtaining the biomedical characteristic data according to the preprocessed biomedical monitoring data.

16. The device according to claim 15, wherein, the biomedical monitoring data includes: multi-channel electroencephalogram (EEG) monitoring data collected from multiple parts of the scalp of the subject to be tested; the obtaining of the biomedical characteristic data according to the preprocessed biomedical monitoring data includes: calculating the potential difference between each channel and a reference electrode from the preprocessed multi-channel EEG monitoring data to obtain initial EEG signal characteristic data; and extracting the biomedical characteristic data according to the initial EEG signal characteristic data.

17. The device according to claim 16, wherein, the biomedical characteristic data includes EEG signal waveform characteristics; the extracting of the biomedical characteristic data according to the initial EEG signal characteristic data includes: using a pre-trained waveform characteristic extraction model to process the initial EEG signal characteristic data to extract the EEG signal waveform characteristics.

18. The device according to claim 16, wherein, the biomedical characteristic data includes EEG signal time-frequency characteristics; the extracting of the biomedical characteristic data according to the initial EEG signal characteristic data includes: performing time-frequency transformation on the initial EEG signal characteristic data to obtain EEG signal time-frequency data corresponding to the initial EEG signal characteristic data; and extracting the EEG signal time-frequency characteristics according to the EEG signal time-frequency data.

19. The device according to claim 15, wherein, the preprocessing of the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical normalization processing.

20. The device according to claim 14, wherein, the video monitoring data includes face video monitoring data; the detecting of the action information of the subject to be tested according to the video monitoring data includes: detecting face key point data from the face video monitoring data; and obtaining the face action information of the subject to be tested according to the face key point data.

21. The device according to claim 20, wherein, the interesting image sequence includes a face image sequence; the extracting of the interesting image sequence for characterizing the action of the subject to be tested from the video monitoring data includes: cropping face region images from multiple frames of the face video monitoring data according to the face key point data to obtain a face image sequence.

22. The device according to claim 20, wherein, the obtaining of the face action information of the subject to be tested according to the face key point data includes: determining the face key points where movement occurs and their displacement information according to the face key point data to obtain the face action information of the subject to be tested.

23. The device according to claim 14, wherein, The video monitoring data includes body video monitoring data; detecting the motion information of the subject according to the video monitoring data includes: detecting body key point data from the body video monitoring data; and obtaining the body motion information of the subject according to the body key point data.

24. The apparatus according to claim 23, wherein, the sequence of images of interest includes a sequence of body images; extracting the sequence of images of interest for characterizing the motion of the subject from the video monitoring data includes: cropping body region images from multiple frames of the body video monitoring data according to the body key point data to obtain a sequence of body images.

25. The apparatus according to claim 23, wherein, obtaining the body motion information of the subject according to the body key point data includes: determining the body key points where motion occurs and their displacement information according to the body key point data to obtain the body motion information of the subject.

26. The apparatus according to claim 14, wherein, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; using the pre-trained abnormal discharge detection model to process the biomedical feature data, the motion information, and the sequence of images of interest to obtain the abnormal discharge detection result of the subject includes: inputting the biomedical feature data, the motion information, and the sequence of images of interest into the abnormal discharge detection model; using the feature processing layer to extract motion feature data from the motion information, extract image feature data from the sequence of images of interest, and fuse the biomedical feature data, the motion feature data, and the image feature data to obtain a fused feature; using the attention layer to characterize the fused feature to obtain an embedded feature; and using the classification layer to map the embedded feature to an output space to obtain the abnormal discharge detection result of the subject.

27. A computer-readable storage medium, on which a computer program is stored, wherein, the computer program, when executed by a processor, implements the method according to any one of claims 1 to 13.

28. An electronic device, wherein, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method according to any one of claims 1 to 13 by executing the executable instructions.

Images