Abnormal discharge detection method and device, model training method and device, medium and equipment

Abstract

The embodiment of the invention relates to an abnormal discharge detection method, an abnormal discharge detection device, a training method of an abnormal discharge detection model, a training device of the abnormal discharge detection model, a computer readable storage medium and electronic equipment, and relates to the technical field of artificial intelligence. The abnormal discharge detection method comprises the following steps: determining a to-be-detected biomedical signal and a to-be-detected monitoring video of a detected object; determining motion features of the detected object according to the to-be-detected monitoring video, wherein the motion features comprise at least one of a body motion feature, a head motion feature, an eye motion feature, a mouth motion feature and a nose motion feature; and inputting the to-be-detected biomedical signal and the motion feature into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the detected object. According to the invention, the video action of the detected object and the biomedical signal are combined to carry out abnormal discharge detection, so that the detection accuracy can be improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, the embodiments of the present disclosure relate to an abnormal discharge detection method, a training method for an abnormal discharge detection model, an abnormal discharge detection device, a training device for an abnormal discharge detection model, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.

[0003] Electroencephalogram (EEG) is an important basis for judging whether a patient's brain function is abnormal, and is of great value for clinical diagnosis and treatment, and assessment of the severity of the disease. At present, doctors mainly check EEG waveforms to judge whether a patient's brain is abnormal. Summary of the invention

[0004] However, the above scheme requires a large workload for doctors, and when there are many patients, doctors are likely to make inaccurate judgments due to fatigue and the like.

[0005] Therefore, there is a great need for an abnormal discharge detection method that can quickly obtain abnormal discharge detection results with high accuracy, reduce the workload of doctors, and assist doctors in making clinical diagnoses more accurately.

[0006] In this context, embodiments of the present disclosure are intended to provide an abnormal discharge detection method, an abnormal discharge detection device, a training method for an abnormal discharge detection model, a training device for an abnormal discharge detection model, a computer-readable storage medium, and an electronic device.

[0007] According to a first aspect of an embodiment of the present disclosure, there is provided an abnormal discharge detection method, comprising: determining a biomedical signal to be detected and a monitoring video to be detected of a test object; determining motion features of the test object based on the monitoring video to be detected, the motion features comprising at least one of body motion features, head motion features, eye motion features, mouth motion features, and nose motion features; and inputting the biomedical signal to be detected and the motion features into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the test object.

[0008] In an optional embodiment, determining the biomedical signals to be detected and the monitoring videos to be detected of the subject includes: obtaining the collected biomedical signals and monitoring videos of the subject, wherein the monitoring videos are taken when collecting the biomedical signals of the subject; obtaining the biomedical signals and monitoring videos of the subject in segments according to preset time lengths to obtain biomedical signal segments and monitoring video segments of the same time period; and determining the biomedical signal segments and monitoring video segments of the same time period as the biomedical signals to be detected and the monitoring videos to be detected of the subject.

[0009] In an optional embodiment, determining the biomedical signals to be detected and the monitoring video to be detected of the subject includes: acquiring the collected biomedical signals of the subject; acquiring the biomedical signals of the subject in segments according to a preset time length to determine the biomedical signals to be detected of the subject; when the biomedical signals to be detected indicate abnormal discharge, cutting out a monitoring video segment of the same period as the biomedical signals to be detected from the monitoring video of the subject to determine the monitoring video to be detected of the subject.

[0010] In an optional embodiment, when the motion feature includes a body motion feature, the body motion feature includes body movement and body non-movement, and a method for determining the body motion feature includes: for any monitoring video frame of the monitoring video to be detected, determining the head position of the subject in the monitoring video frame according to a head position detection model, and cropping the monitoring video frame according to the head position to obtain a body video frame corresponding to the monitoring video frame; for any body video frame, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining a frame difference corresponding to the reference video frame according to the sum of pixel value differences at the same position between the body video frame and the reference video frame; determining a target frame difference of the body video frame according to the maximum value of the frame differences corresponding to the reference video frame, and determining the body motion feature of the subject based on the target frame difference.

[0011] In an optional embodiment, when the motion feature includes a head motion feature, the head motion feature includes head movement and no head movement, and the method for determining the head motion feature includes: determining the head position of the object under test in the video frame of the surveillance video to be detected based on a head position detection model; cropping the video frame of the surveillance video to be detected based on the head position to obtain a head monitoring video of the object under test; inputting the head monitoring video into a pre-trained head motion detection model, and when the output result of the head motion detection model indicates head movement, determining the head motion feature as head movement, otherwise, determining the head motion feature as no head movement; wherein the training label of the head motion detection model is determined based on whether the head of the object moves and whether there are facial expression changes in the training video.

[0012] In an optional embodiment, when the motion feature includes a mouth motion feature and / or an eye motion feature, the mouth motion feature includes a mouth motion and a mouth non-motion, the eye motion feature includes a left eye motion feature and a right eye motion feature, the left eye motion feature includes a left eye motion and a left eye non-motion, the right eye motion feature includes a right eye motion and a right eye non-motion, and the method for determining the mouth motion feature and the eye motion feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected according to a head position detection model; detecting the facial key points of the subject based on the head position to obtain the mouth of the subject; The invention discloses a method for detecting a first aspect ratio of a mouth according to a target mouth key point among the mouth key points; determining a second aspect ratio of the left eye according to a target left eye key point among the left eye key points, and determining a third aspect ratio of the right eye according to a target right eye key point among the right eye key points; determining a mouth motion feature according to a first change value of the first aspect ratio corresponding to adjacent frames in the surveillance video to be detected; determining a left eye motion feature according to a second change value of the second aspect ratio corresponding to adjacent frames in the surveillance video to be detected; and determining a right eye motion feature according to a third change value of the third aspect ratio corresponding to adjacent frames in the surveillance video to be detected.

[0013] In an optional embodiment, when the motion feature includes a nose motion feature, the nose motion feature includes nose movement and nose non-movement, and a method for determining the nose motion feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected based on a head position detection model; detecting the facial key points of the subject based on the head position to obtain the nose key point positions of the subject; and determining the nose motion feature based on changes in the nose key point positions indicated by a nose key point position sequence in the video frame of the surveillance video to be detected.

[0014] In an optional embodiment, the step of inputting the biomedical signal to be detected and the motion feature into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the object under test includes: inputting the biomedical signal to be detected into a biomedical signal feature extraction network to obtain the biomedical signal feature corresponding to the biomedical signal to be detected; performing feature enhancement on the motion feature to increase the feature dimension of the motion feature to obtain the target motion feature; splicing the biomedical signal feature and the target motion feature to obtain the spliced ​​feature; and inputting the spliced ​​feature into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the object under test. In an optional embodiment, the biomedical signal includes any one of a biomedical signal numerical sequence and a biomedical signal curve image.

[0015] According to a second aspect of an embodiment of the present disclosure, a training method for an abnormal discharge detection model is provided, comprising: obtaining training samples and sample labels corresponding to the training samples, the training samples comprising biomedical signal samples and monitoring video samples of a test subject, the monitoring video samples being obtained by photographing the test subject when collecting the biosignal samples of the test subject; determining sample motion features corresponding to the monitoring video samples, the sample motion features comprising at least one of sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, and sample nose motion features; inputting the sample motion features corresponding to the biomedical signal samples and the monitoring video samples into an abnormal discharge detection model to be trained to obtain an abnormal discharge detection result of the training samples; determining a loss value of the abnormal discharge detection model to be trained based on a difference between the abnormal discharge detection result of the training samples and the sample labels; iteratively training the abnormal discharge detection model to be trained based on the loss value until a preset condition is met to obtain a pre-trained abnormal discharge detection model.

[0016] In an optional embodiment, the step of inputting the sample motion features corresponding to the biomedical signal samples and the surveillance video samples into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection results of the training samples includes: inputting the biomedical signal samples into a biomedical signal feature extraction network to obtain a first output; inputting the sample motion features into a motion feature enhancement network to obtain a second output; splicing the first output and the second output to obtain a target splicing result; inputting the target splicing result into the abnormal discharge detection model to be trained, and obtaining the abnormal discharge detection results of the training samples according to the output of the abnormal discharge detection model to be trained.

[0017] According to a third aspect of the present disclosure, an abnormal discharge detection device is provided, comprising: a data determination module to be detected, configured to determine a biomedical signal to be detected and a surveillance video to be detected of a subject; a motion feature determination module, configured to determine the motion features of the subject according to the surveillance video to be detected, wherein the motion features include at least one of body motion features, head motion features, eye motion features, mouth motion features, and nose motion features; and a first detection module, configured to input the biomedical signal to be detected and the motion features into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0018] In an optional embodiment, the module for determining the data to be detected is specifically configured to: obtain the collected biomedical signals and monitoring videos of the subject, wherein the monitoring video is taken when the biomedical signals of the subject are collected; obtain the biomedical signals and monitoring videos of the subject in segments according to preset time lengths to obtain biomedical signal segments and monitoring video segments of the same time period; and determine the biomedical signal segments and monitoring video segments of the same time period as the biomedical signals to be detected and monitoring video to be detected of the subject.

[0019] In an optional embodiment, the module for determining the data to be detected is specifically configured to: obtain the collected biomedical signals of the subject; obtain the biomedical signals of the subject in segments according to a preset time length to determine the biomedical signals to be detected of the subject; when the biomedical signals to be detected indicate abnormal discharge, cut out a surveillance video segment of the same period as the biomedical signals to be detected from the surveillance video of the subject to determine the surveillance video of the subject to be detected.

[0020] In an optional embodiment, when the motion feature includes a body motion feature, the body motion feature includes body movement and body non-movement, and a method for determining the body motion feature includes: for any monitoring video frame of the monitoring video to be detected, determining the head position of the subject in the monitoring video frame according to a head position detection model, and cropping the monitoring video frame according to the head position to obtain a body video frame corresponding to the monitoring video frame; for any body video frame, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining a frame difference corresponding to the reference video frame according to the sum of pixel value differences at the same position between the body video frame and the reference video frame; determining a target frame difference of the body video frame according to the maximum value of the frame differences corresponding to the reference video frame, and determining the body motion feature of the subject based on the target frame difference.

[0021] In an optional embodiment, when the motion feature includes a head motion feature, the head motion feature includes head movement and no head movement, and the method for determining the head motion feature includes: determining the head position of the object under test in the video frame of the surveillance video to be detected based on a head position detection model; cropping the video frame of the surveillance video to be detected based on the head position to obtain a head monitoring video of the object under test; inputting the head monitoring video into a pre-trained head motion detection model, and when the output result of the head motion detection model indicates head movement, determining the head motion feature as head movement, otherwise, determining the head motion feature as no head movement; wherein the training label of the head motion detection model is determined based on whether the head of the object moves and whether there are facial expression changes in the training video.

[0022] In an optional embodiment, when the motion feature includes a mouth motion feature and / or an eye motion feature, the mouth motion feature includes a mouth motion and a mouth not moving, the eye motion feature includes a left eye motion feature and a right eye motion feature, the left eye motion feature includes a left eye motion and a left eye not moving, the right eye motion feature includes a right eye motion and a right eye not moving, and the method for determining the mouth motion feature and the eye motion feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected according to a head position detection model; detecting the facial key points of the subject based on the head position to obtain the mouth key points, Left eye key points and right eye key points; determining a first aspect ratio of the mouth according to a target mouth key point among the mouth key points; determining a second aspect ratio of the left eye according to a target left eye key point among the left eye key points, and determining a third aspect ratio of the right eye according to a target right eye key point among the right eye key points; determining the mouth motion feature according to a first change value of the first aspect ratio corresponding to adjacent frames in the surveillance video to be detected; determining the left eye motion feature according to a second change value of the second aspect ratio corresponding to adjacent frames in the surveillance video to be detected; determining the right eye motion feature according to a third change value of the third aspect ratio corresponding to adjacent frames in the surveillance video to be detected.

[0023] In an optional embodiment, when the motion feature includes a nose motion feature, the nose motion feature includes nose movement and nose non-movement, and a method for determining the nose motion feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected based on a head position detection model; detecting the facial key points of the subject based on the head position to obtain the nose key point position of the subject; and determining the nose motion feature based on the change in the nose key point position indicated by the nose key point position sequence in the video frame of the surveillance video to be detected.

[0024] In an optional embodiment, the first detection module is specifically configured to: input the biomedical signal to be detected into a biomedical signal feature extraction network to obtain a biomedical signal feature corresponding to the biomedical signal to be detected; perform feature enhancement on the motion feature to increase the feature dimension of the motion feature to obtain a target motion feature; splice the biomedical signal feature and the target motion feature to obtain a spliced ​​feature; input the spliced ​​feature into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0025] In an optional implementation, the biomedical signal includes any one of a biomedical signal numerical sequence and a biomedical signal curve image.

[0026] According to a fourth aspect of the present disclosure, a training device for an abnormal discharge detection model is provided, comprising: a training sample acquisition module, configured to acquire training samples and sample labels corresponding to the training samples, wherein the training samples include biomedical signal samples and monitoring video samples of the subject, wherein the monitoring video samples are obtained by photographing the subject when collecting the biomedical signal samples of the subject; a sample motion feature determination module, configured to determine sample motion features corresponding to the monitoring video samples, wherein the sample motion features include sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, At least one of the sample nose motion features; a second detection module, configured to input the sample motion features corresponding to the biomedical signal sample and the monitoring video sample into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample; a loss determination module, configured to determine the loss value of the abnormal discharge detection model to be trained according to the difference between the abnormal discharge detection result of the training sample and the sample label; an iterative training module, configured to iteratively train the abnormal discharge detection model to be trained according to the loss value until a preset condition is met to obtain a pre-trained abnormal discharge detection model.

[0027] In an optional embodiment, the second detection module is specifically configured to: input the biomedical signal sample into a biomedical signal feature extraction network to obtain a first output; input the sample motion feature into a motion feature enhancement network to obtain a second output; splice the first output and the second output to obtain a target splicing result; input the target splicing result into an abnormal discharge detection model to be trained, and obtain the abnormal discharge detection result of the training sample according to the output of the abnormal discharge detection model to be trained.

[0028] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the abnormal discharge detection method described in the first aspect above and / or the training method of the abnormal discharge detection model described in the second aspect above are implemented.

[0029] According to the sixth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the abnormal discharge detection method described in the first aspect above, and / or the training method of the abnormal discharge detection model described in the second aspect above, by executing the executable instructions.

[0030] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the abnormal discharge detection method as described in the first aspect above, and / or the training method of the abnormal discharge detection model as described in the second aspect above are implemented.

[0031] According to the abnormal discharge detection method, abnormal discharge detection device, abnormal discharge detection model training method, abnormal discharge detection model training device, computer-readable storage medium and electronic device of the embodiment of the present disclosure, the motion characteristics of the object to be detected are determined through the monitoring video of the object to be detected, and the biomedical signal and motion characteristics of the object to be detected are input into the abnormal discharge detection model to obtain the abnormal discharge detection result. On the one hand, the motion characteristics of the object to be detected are determined through the monitoring video to be detected, and the motion characteristics and biomedical signals are combined to detect abnormal discharge. The motion characteristics can eliminate the interference of the subtle movements of the object to be detected on the abnormal discharge detection result, thereby improving the accuracy of abnormal discharge detection; on the other hand, the abnormal discharge detection result can be obtained without manual reading of the image, which improves the detection efficiency for scenes that require a large number of abnormal discharge detections; on the other hand, since the abnormal discharge detection result determined by the method of the present disclosure is highly accurate, it can provide a high reference value for doctors, assist doctors in making clinical diagnoses quickly and accurately, save doctors' workload, and facilitate wide application in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0033] Figure 1 A schematic flow chart of a method for training an abnormal discharge detection model in an exemplary embodiment of the present disclosure is shown;

[0034] Figure 2 A flowchart diagram showing a method for generating training samples and sample labels of the training samples in an exemplary embodiment of the present disclosure is shown;

[0035] Figure 3 A schematic flow chart showing a method for processing surveillance video samples in an exemplary embodiment of the present disclosure;

[0036] Figure 4 A schematic flow chart showing a method for determining sample body movement characteristics in an exemplary embodiment of the present disclosure;

[0037] Figure 5 A schematic flow chart showing a method for determining head motion characteristics of a sample in an exemplary embodiment of the present disclosure;

[0038] Figure 6 A flowchart showing a method for determining sample mouth movement characteristics in an exemplary embodiment of the present disclosure;

[0039] Figure 7 A schematic flow chart showing a method for determining sample eye movement characteristics in an exemplary embodiment of the present disclosure;

[0040] Figure 8 A schematic flow chart showing a method for determining sample nose motion features in an exemplary embodiment of the present disclosure;

[0041] Fig. 9 A schematic flow chart showing a method for obtaining abnormal discharge detection results of training samples in an exemplary embodiment of the present disclosure;

[0042] Fig.10 A schematic diagram showing a flow chart of another abnormal discharge detection model training method in an exemplary embodiment of the present disclosure;

[0043] Fig.11 A schematic flow chart showing an abnormal discharge detection method in an exemplary embodiment of the present disclosure;

[0044] Fig.12 A schematic flow chart showing a method for determining a biomedical signal to be detected and a surveillance video to be detected in an exemplary embodiment of the present disclosure;

[0045] Fig.13 A flow chart showing another method for determining a biomedical signal to be detected and a surveillance video to be detected in an exemplary embodiment of the present disclosure;

[0046] Fig.14 A schematic flow chart showing a method for obtaining abnormal discharge detection results in an exemplary embodiment of the present disclosure;

[0047] Fig.15A structural diagram showing an abnormal discharge detection system in an exemplary embodiment of the present disclosure;

[0048] Fig.16 A schematic diagram showing an exemplary application scenario of an abnormal discharge detection method in an embodiment of the present disclosure;

[0049] Fig.17 A schematic diagram showing an exemplary application scenario of another abnormal discharge detection method in an embodiment of the present disclosure;

[0050] Fig.18 A schematic diagram showing the composition of an abnormal discharge device in an exemplary embodiment of the present disclosure;

[0051] Fig.19 A schematic diagram showing the composition of a training device for an abnormal discharge detection model according to an embodiment of the present disclosure is shown;

[0052] Fig. 20 A structural diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0053] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0054] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0055] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0056] According to an embodiment of the present disclosure, an abnormal discharge detection method, an abnormal discharge detection device, a training method for an abnormal discharge detection model, a training device for an abnormal discharge detection model, a computer-readable storage medium, and an electronic device are provided.

[0057] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0058] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0060] The inventors have found that the existing method of manually reading EEG to determine whether there is abnormal discharge in the brain is inefficient and its accuracy depends on the doctor's experience. For inexperienced doctors, misjudgment may occur.

[0061] In view of the above content, the basic idea of ​​the present disclosure is to provide an abnormal discharge detection method, an abnormal discharge detection device, a training method for an abnormal discharge detection model, a training device for an abnormal discharge detection model, a computer-readable storage medium and an electronic device, determine the motion characteristics of the object to be detected through the monitoring video to be detected of the object to be detected, input the biomedical signal and the motion characteristics of the object to be detected into the abnormal discharge detection model, and obtain the abnormal discharge detection result. On the one hand, the motion characteristics of the object to be detected are determined through the monitoring video to be detected, and the motion characteristics and the biomedical signal are combined to detect abnormal discharge. The motion characteristics can eliminate the interference of the action of the object to be detected on the abnormal discharge detection result, thereby improving the accuracy of abnormal discharge detection; on the other hand, the abnormal discharge detection result can be obtained without manual reading of the image, which improves the detection efficiency for scenes that require a large number of abnormal discharge detections; on the other hand, since the abnormal discharge detection result determined by the method of the present disclosure is highly accurate, it can provide a high reference value for doctors, assist doctors in making clinical diagnoses quickly and accurately, save doctors' workload, and facilitate wide application in clinical practice.

[0062] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0063] Application Scenario Overview

[0064] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0065] The embodiments of the present disclosure can be applied to abnormal discharge detection scenarios of the human brain. For example, when a patient feels uncomfortable, such as having a headache, the patient's brain electrical signals can be collected and a video of the patient can be taken while the brain electrical signals are collected to obtain video monitoring data when the brain electrical signals are collected. The video monitoring data is then processed to obtain the patient's motion characteristics, and the motion characteristics and the collected EEG signals are input into a pre-trained abnormal discharge detection model to obtain the patient's abnormal discharge detection results, so as to assist the doctor in making a rapid and accurate clinical diagnosis of the patient based on the abnormal discharge detection results combined with other key indicator data, thereby reducing the doctor's workload and improving the doctor's work efficiency.

[0066] Exemplary Methods

[0067] The current method of manually studying EEGs to determine whether there are abnormal discharges in the brain is inefficient, and its accuracy depends on the doctor's personal experience. For inexperienced doctors, misjudgment may occur.

[0068] The exemplary embodiments of the present disclosure first provide a method for training an abnormal discharge detection model to at least overcome all or part of the above-mentioned defects existing in the above-mentioned related art.

[0069] Figure 1 FIG. 1 is a flow chart showing a method for training an abnormal discharge detection model in an embodiment of the present disclosure, referring to FIG. Figure 1 According to an embodiment of the present disclosure, a method for training an abnormal discharge detection model comprises the following steps:

[0070] Step S110, obtaining training samples and sample labels corresponding to the training samples;

[0071] Step S120, determining the sample motion feature corresponding to the surveillance video sample;

[0072] Step S130, inputting the sample motion features corresponding to the biomedical signal sample and the monitoring video sample into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample;

[0073] Step S140, determining a loss value of the abnormal discharge detection model to be trained according to a difference between the abnormal discharge detection result of the training sample and the sample label;

[0074] Step S150, iteratively training the abnormal discharge detection model to be trained according to the loss value until a preset condition is met, thereby obtaining a pre-trained abnormal discharge detection model.

[0075] Next, Figure 1 The specific implementation of step S110 in is described in detail.

[0076] In an optional implementation, the training samples in step S110 include biomedical signal samples and monitoring video samples of the subject.

[0077] In an optional embodiment, the biomedical signal samples are biomedical signals collected from a large number of different subjects. Biomedical signals are active signals spontaneously generated by the physiological process of a living body, which can more accurately reflect the physiological state or physical state of a person. Therefore, the biomedical signal can be used to determine whether the living body is abnormal. The living body in the present disclosure may include any living body, such as a human body, an animal body, etc., and this exemplary embodiment does not specifically limit this.

[0078] Exemplarily, the above-mentioned biomedical signals may include any one or more of the following: physiological signals such as electrocardiogram signals, electroencephalogram signals, electromyography signals, electrooculography signals and electrogastric signals, or may also include non-electrical signals such as body temperature, blood pressure, pulse, respiration, etc., which can be set according to actual conditions, and the present disclosure does not make any special limitations on this.

[0079] Among them, electrocardiogram (ECG) refers to a series of very coordinated electrical stimulation pulses generated inside the heart, which make the heart muscle cells relax and contract rhythmically. These signals are transmitted to different parts of the human body surface to form different potential differences. Multi-channel electrocardiogram equipment can detect these weak potential difference signals from the human body surface to obtain ECG signals.

[0080] Electroencephalogram (EEG) is the overall reflection of the electrophysiological activity of brain nerve cells on the cerebral cortex or scalp surface. It can be understood as the changing activity data of electrical signals when the brain is active. The spontaneous biopotential of the cerebral cortex can be amplified from the scalp through precise instruments to collect EEG signals. EEG signals are the spontaneous and rhythmic electrical activities of brain cell groups recorded by electrodes.

[0081] Electromyography (EMG) is the temporal and spatial superposition of action potentials of motor units in numerous muscle fibers, which can be obtained by sticking electromyography sensors on the skin.

[0082] Electrooculogram (EOG) is a bioelectric signal caused by the potential difference between the cornea and the retina of the eye. It is very easy to collect and can be completed with a small number of electrodes.

[0083] Electrogastrogram (EGG) is an electrical signal generated by the contraction of stomach muscles, which can be collected on the surface of the human abdominal skin using electrodes.

[0084] In an optional implementation, the biomedical signal includes any one of a biomedical signal numerical sequence and a biomedical signal curve image.

[0085] For example, in the present disclosure, the biomedical signal may be a digital sequence of collected biomedical signals. For example, the EEG used for model training may include a standard 10-20 system electrode, two electrocardiogram electrodes, and four electromyogram electrodes. A multi-channel electrocardiograph, a multi-channel electroencephalogram, or a multi-channel electromyogram may be used to simultaneously collect corresponding biomedical electrical signals at a preset collection frequency, such as 500 Hz, and the collected biomedical electrical signals may be discretized to obtain a corresponding biomedical signal numerical sequence.

[0086] Taking EEG signals as an example, the EEG signals collected by the EEG machine are discretized to obtain the EEG signal sequence shown in the following formula (1).

[0087] x t =(x t1 ,x t2 ,x t3 ,x t4 ,…,x tn ) (1)

[0088] In formula (1), x t is the numerical sequence of EEG signals. t1 ,x t2 ,x t3 ,x t4 ,…,x tn Represents the signal value of the EEG signal at different sampling times, such as x t1 Represents the EEG signal x t The signal value at the first sampling moment, x tn Represents the EEG signal x t The signal value at the nth sampling moment in .

[0089] In the present disclosure, the biomedical signal may also be a biomedical signal image, such as an EEG signal image, an ECG signal image, etc. That is, a biomedical signal image may be generated according to a signal curve of the collected biomedical signal, and an abnormal discharge detection model may be trained based on the biomedical signal image.

[0090] In the case where the signal collected by the electroencephalograph is a digital EEG signal, an EEG signal curve can be drawn with the sampling time of the digital EEG signal as the horizontal axis and the sampling value of the digital EEG signal as the vertical axis, and the drawn EEG signal curve is used as an EEG signal image. Then, the EEG signal image is input into the EEG signal feature extraction model to obtain the EEG signal feature, and then the extracted EEG signal feature is combined with the motion feature obtained from the monitoring video and input into the abnormal discharge detection model to be trained to train the abnormal discharge detection model of the brain.

[0091] Taking FP1 (left frontal pole), FP2 (right frontal pole), F3 (left frontal pole), F4 (right frontal pole), C3 (left central pole), and C4 (right central pole) of the above-mentioned 10-20 standard electrodes as an example, the EEG signals collected by the EEG machine are stored in the form of digital signals as shown in Table 1 below.

[0092] Table 1 EEG digital signals

[0093] FP1 (left frontal pole) 52 55 42 41 2 … FP2 (right frontal pole) 32 14 42 12 24 … F3 (left forehead) 6 41 23 24 87 … F4 (right forehead) 56 2 42 12 2 … C3 (center left) 44 55 42 21 86 … C4(center right) 33 53 34 2 51 …

[0094] In other words, taking EEG signals as an example, the original EEG signals collected by the EEG machine are digital signals, and what doctors read when making a diagnosis is the EEG waveform, not the EEG digital signal. Therefore, the EEG machine will also draw the collected EEG digital signal into an EEG waveform (that is, the above-mentioned EEG signal curve) for output.

[0095] For example, when doctors are making a diagnosis, they pay more attention to the EEG waveform, such as whether the peak of the EEG waveform is sharp enough and whether it is obvious compared with the background. That is, the EEG signal curve image is more intuitive for doctors. When generating labels for training samples, doctors also watch monitoring videos and EEG signal curve images to determine whether the training samples are abnormal discharges. Therefore, using EEG signal curve images as input to the model can more accurately simulate the doctor's judgment method for abnormal discharge detection, thereby improving the accuracy of abnormal discharge detection.

[0096] In the present disclosure, the digital signal collected by the EEG machine can be directly input into the biomedical signal feature extraction model to perform biomedical signal feature extraction, or the EEG waveform image output by the EEG machine can be input into the biomedical signal feature extraction model to perform biomedical signal feature extraction. This exemplary embodiment does not make any special limitations on this.

[0097] In an exemplary embodiment, when the biomedical signal is a biomedical signal image, for a multi-channel biomedical signal, the multi-channel biomedical signal curves can be drawn in the same image, and the biomedical signal feature extraction model can use a 2D (two-dimensional) convolutional neural network as a backbone network, such as EfficientNet-V2, to extract biomedical signal features.

[0098] In an optional embodiment, when the biomedical signal is a biomedical signal image, for a multi-channel biomedical signal, the biomedical signal curve of each channel can also be drawn separately to obtain multiple biomedical signal images corresponding to the multi-channel biomedical signal. The biomedical signal curve images corresponding to the biomedical signals of different channels can be used as inputs of different channels of the biomedical signal feature extraction model. At this time, the biomedical signal feature extraction model can use a 3D (three-dimensional) convolutional neural network as a backbone network to extract biomedical signal features.

[0099] In an optional embodiment, the monitoring video sample is obtained by photographing the subject when collecting the bio-signal sample of the subject. For example, when collecting biomedical signals from a large number of different subjects, for each subject, a video of the subject can be recorded at the same time to obtain the video monitoring data of the subject when collecting the biomedical signal of the subject.

[0100] In an optional implementation, for the same subject, the biomedical signal samples and monitoring video samples in the training samples are collected in the same time period, that is, the training samples include a large number of biomedical signals and monitoring videos collected from the same subject in the same time period.

[0101] For example, for each sampled object, a long period of biomedical signals and surveillance videos can be collected, such as 30 minutes of biomedical signals and surveillance videos. Then, the biomedical signals and surveillance videos are segmented according to the same start time and the same preset duration. For example, if the preset duration is 1 minute, the collected biomedical signals and surveillance videos within 30 minutes are divided into 30 pairs of biomedical signals and surveillance video samples of the same period with a duration of 1 minute. The biomedical signals and surveillance video sample pairs are used as training samples. That is, each training sample is a pair of biomedical signals and surveillance videos of the same sampled object in the same period.

[0102] In an optional implementation, the sample label corresponding to the training sample may include whether the sampled object in the training sample has abnormal discharge, that is, 1 indicates the presence of abnormal discharge, and 0 indicates the absence of abnormal discharge. If the sampled object is judged to have abnormal discharge based on the biomedical signal sample and the monitoring video sample pair of the same period, the sample label of the biomedical signal sample and the monitoring video sample of the same period is 1; if the sampled object is judged to have no abnormal discharge based on the biomedical signal sample and the monitoring video sample pair of the same period, the sample label of the biomedical signal sample and the monitoring video sample of the same period is 0.

[0103] For example, sample labels can be added to training samples by manual annotation. For example, doctors can watch surveillance videos and refer to biomedical signal curve images at the same time as the surveillance videos to determine whether the biomedical signals actually have abnormal discharges, thereby generating labels for training samples.

[0104] Below, reference Figure 2 , taking brain abnormality detection as an example, a flow chart of a method for generating training samples and labels of training samples in an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, the method may include steps S210 to S240. In step S210, the electroencephalograph collects multi-channel EEG signals of the sampled object to obtain multi-channel EEG signals; in step S220, the camera collects facial and torso videos of the sampled object to obtain monitoring videos; in step S230, the multi-channel EEG signals and monitoring videos are segmented to obtain multi-channel EEG signal samples and monitoring video samples of the same period; in step S240, each segment of the multi-channel EEG signals is manually marked to determine whether there is abnormal discharge, so as to obtain training sample labels.

[0105] For example, after obtaining the multi-channel EEG signal samples and monitoring video samples of the same period, the doctor can watch the monitoring video samples to determine whether the multi-channel EEG signal samples of the same period really have abnormal discharges. For example, after watching the monitoring video samples, if the doctor believes that the multi-channel EEG signal samples at time T1 are autonomously abnormally discharged without external interference, the above mark 1 can be added at time T1, indicating that there is abnormal discharge at time T1. For the multi-channel EEG signal samples and monitoring video samples of the same period, as long as there is any moment in the multi-channel biomedical signal samples in the period that is added as mark 1, the sample label of the training samples composed of the multi-channel EEG signal samples and monitoring video samples of the same period is that there is abnormal discharge. Conversely, if there is no moment in the channel biomedical signal samples in the period that is marked as 1, the sample label of the training samples composed of the multi-channel EEG signal samples and monitoring video samples of the same period is that there is no abnormal discharge.

[0106] Next, Figure 1 The specific implementation of step S120 in is described in detail.

[0107] Taking EEG signals as an example, the automatic recognition of EEG signal anomalies requires a large number of training samples to learn the characteristics of EEG signals, and the interference of artifacts is also an important reason. EEG signal anomaly detection needs to be performed in a conscious state with eyes closed or in a state of impaired consciousness, but EEG signals are weak and will inevitably be affected by artifacts such as electromyography, electrocardiography, and movement.

[0108] In an optional implementation, electromyographic and electrocardiographic artifacts may be determined by collecting electrocardiographic signals and electromyographic signals.

[0109] In an optional implementation, motion artifacts can be identified through recorded video, so that the influence of motion artifacts on abnormal discharge detection can be eliminated, and the accuracy of automatic detection and identification of abnormal discharges can be improved, so that it can be better applied in clinical practice.

[0110] For example, as mentioned above, the detection of abnormal EEG signals needs to be performed in a conscious state with eyes closed or in a state of impaired consciousness. Therefore, in addition to large-scale body movements that affect the detection results, small-scale movements will also affect the results. In other words, in the identification of motion artifacts, in addition to large-scale body movements that need to be identified, small-scale body movements, such as blinking, small-scale chewing, small-scale talking, and other small movements that are difficult to be observed by the human eye, also need to be identified.

[0111] Based on this, in an optional implementation, the motion features corresponding to the surveillance video samples include at least one of sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, and sample nose motion features.

[0112] For example, it is possible to determine based on surveillance video samples whether the subject's body, head, eyes, mouth, nose, etc. are moving when collecting biomedical signals at the same time as the surveillance video samples, so as to obtain the motion characteristics of the subject, and then combine the motion characteristics of the subject with the biomedical signal characteristics to determine whether the subject has abnormal discharge. In this way, the influence of motion artifacts on abnormal discharge detection can be avoided, thereby improving the accuracy of abnormal discharge detection.

[0113] For example, for each surveillance video sample obtained by segmentation, Figure 3 The framework shown processes the surveillance video sample to determine the body parts and head of the subject in the video frame of the surveillance video sample, so as to determine the motion characteristics of the subject in the time period corresponding to the surveillance video sample according to the positions of the body parts and head of the subject in the video frame of the surveillance video sample. Figure 3, video frames can be collected in the monitoring video sample according to the preset frame rate to obtain monitoring video frame sequence samples. The monitoring video frame sequence samples are input into the patient detection model to obtain the position of the patient in each video frame in the monitoring video frame sequence samples. According to the position of the patient in each video frame of the monitoring video frame sequence samples, the video frames in the monitoring video frame sequence samples are cropped to obtain patient video frame sequence samples. The patient video frame sequence samples are input into the face detection model to obtain the position of the patient's head in each video frame in the patient video frame sequence samples. According to the position of the patient's head in each video frame in the patient video frame sequence samples, each video frame in the patient video frame sequence samples is cropped to obtain patient head video frame sequence samples and patient body video frame sequence samples. According to the facial key point detection method, the positions of the patient's eyes, mouth and nose are determined in each video frame in the patient's head video frame sequence samples.

[0114] In an optional implementation, the patient detection model may include any machine learning model that can be used for target detection, such as the YOLOv5 (You Only Look Once) target detection model or any other deep learning model. The patient head detection model may also include any face detection model, such as the YOLOv5 Face face detection model.

[0115] For example, monitoring video samples can be first collected based on a preset frame rate, such as 15 frames per second, to obtain monitoring video frame sequence samples. Then, the patient is detected in each video frame in the monitoring video frame sequence samples based on the YOLOv5 model, and the video frames in the monitoring video frame sequence samples are cropped according to the detected patient positions to obtain patient video frame sequence samples. The patient's face is detected in each video frame of the patient video frame sequence samples based on the YOLOv5 Face face detection model, and the patient in the patient video frame sequence samples is cropped according to the detected face position to obtain patient head video frame sequence samples and patient body video frame sequence samples. In this way, the patient's body motion features can be determined based on the patient's body video frame sequence samples, and the patient's head motion features, eye motion features, mouth motion features, and nose motion features can be determined based on the patient's head video frame sequence samples.

[0116] Next, combine Figures 4 to 8 , further explains the specific implementation method of determining the sample body movement characteristics, head movement characteristics, eye movement characteristics, mouth movement characteristics, and nose movement characteristics.

[0117] In an optional embodiment, the body movement feature includes body movement and body non-movement. Figure 4A flow chart showing a method for determining sample body movement characteristics in an exemplary embodiment of the present disclosure is shown. Figure 4 , the method may include steps S410 to S440. Wherein:

[0118] In step S410, for any surveillance video frame of the surveillance video sample, the head position of the subject in the surveillance video frame is determined according to the head position detection model, and the surveillance video frame is cropped according to the head position to obtain a body video frame corresponding to the surveillance video frame.

[0119] In an optional implementation, the head position detection model may include the above-mentioned face detection model, and the face position detected by the face detection model may be understood as the head position.

[0120] For example, as mentioned above, surveillance video samples can be collected according to a preset frame rate, such as 15 frames per second, to obtain surveillance video frames. For each surveillance video frame, the position of the sampled object can be detected in the surveillance video frame according to the target detection model, and then the area outside the sampled object in the surveillance video frame can be cut off to obtain the sampled object video frame. In this way, other background areas in the surveillance video frame that are not related to the sampled object can be avoided from interfering with the motion detection of the sampled object, thereby improving the accuracy and efficiency of the motion feature determination.

[0121] The target detection model usually uses a rectangular frame to represent the position of the detected target in the image frame. Therefore, the monitoring video frame can be cropped according to the boundary of the rectangular frame of the detected position of the captured object, and the area inside the rectangular frame is retained in the monitoring video frame, and the area outside the rectangular frame is cropped to obtain the captured object video frame. After obtaining the captured object video frame, the captured object video frame can be input into the face detection model, and the face position in the captured object video frame is detected by the face detection model, so as to obtain the head position of the captured object. Under normal circumstances, the detected face position of the captured object can also be framed in the captured object video frame by a rectangular frame, so as to segment the captured object video frame according to the rectangular frame to obtain the head video frame of the captured object and the body video frame of the captured object. That is, the head video frame only includes the head of the captured object, and the body video frame includes the torso and limbs except the head of the captured object.

[0122] Next, in step S420, for any body video frame, a reference video frame of the body video frame is determined according to N other body video frames adjacent to the body video frame.

[0123] In an optional implementation, N may be an integer greater than or equal to 1. The value of N may be customized based on experience or user needs, such as 3, 5, etc. The value of N should not be set too large or too small. If the value of N is set too large, the amount of subsequent calculations will increase and the processing efficiency will be reduced. If the value of N is set too small, the result of the determined body movement characteristics may be inaccurate. Therefore, when setting N, a balance needs to be struck between the amount of calculation and the accuracy.

[0124] Taking the value of N as 3 as an example, for any body video frame, the three video frames immediately before and after it can be used as reference video frames for the body video frame, that is, for the body video frames of the 4th frame and thereafter, there are reference video frames. There are 5 reference video frames for the 3rd frame, 4 reference video frames for the 2nd frame, and the reference video frames for the 1st frame are only the 3 consecutive video frames immediately after it, that is, the 2nd frame, the 3rd frame, and the 4th frame are the reference video frames for the 1st frame.

[0125] In step S430, a frame difference value corresponding to the reference video frame is determined according to the sum of the pixel value differences at the same position between the body video frame and the reference video frame.

[0126] For example, for each reference video frame, the frame difference corresponding to the reference video frame can be determined according to the sum of the differences of the pixel values ​​at the same pixel coordinates. n (x, y) represents the pixel value at the position (x, y) of the nth frame. Taking the nth frame as the current body video frame and the n+1th frame as a reference frame of the body video frame of the nth frame as an example, the difference between the pixel values ​​at the position with the pixel coordinates (x, y) in the nth frame and the pixel coordinates (x, y) in the n+1th frame can be expressed as D n (x,y)=|f n (x,y)-f n+1 (x,y)|. Taking the case that there are N pixels in the body video frame, the sum of the pixel value differences at the same position in the reference video frame (n+1) and the body video frame (n) can be expressed as D nN (x nN ,y nN )=|f n (x1,y1)-f n+1 (x1,y1)|+|f n (x2,y 21 )-f n+1 (x2,y2)|+…+|f n (x N ,y N )-f n+1 (x N ,yN )|.

[0127] In step S440, a target frame difference value of the body video frame is determined according to the maximum value of the frame difference values ​​corresponding to the reference video frame.

[0128] For example, the frame difference values ​​corresponding to each reference video frame may be sorted in descending order or ascending order to determine the maximum value among the frame difference values ​​corresponding to the reference video frames, and the maximum frame difference value may be determined as the target frame difference value.

[0129] In step S440, the sample body motion features are determined according to the target frame difference.

[0130] Optionally, an exemplary implementation of step S440 may be: when the target frame difference is greater than a first preset value, the sample body motion feature may be determined as body motion; otherwise, the sample body motion feature may be determined as body non-motion.

[0131] In an optional implementation, when the sample body motion feature is determined to be body motion according to the target frame difference corresponding to any body video frame, the sample body motion feature corresponding to the monitoring video sample is determined to be the body motion feature, and when the target frame difference corresponding to all body video frames in the monitoring video sample determines that the sample body motion feature is body non-motion, the sample body motion feature corresponding to the monitoring video sample is determined to be body non-motion. That is, for the monitoring video sample, as long as any body video frame corresponding to the monitoring video sample determines that the sample body motion feature is body motion, the sample body motion feature of the monitoring video sample is body motion, otherwise, the sample body motion feature of the monitoring video sample is determined to be body non-motion.

[0132] In another optional implementation, when the sample body motion feature is determined to be body motion according to M consecutive surveillance video frames in the surveillance video sample, the sample body motion feature corresponding to the surveillance video sample can be determined to be body motion, otherwise, the sample body motion feature corresponding to the surveillance video sample is determined to be body non-motion. M can be customized according to needs, such as M is 4, M is 5, etc. In this way, the accuracy of body motion feature determination can be improved.

[0133] Through the above-mentioned steps S410 to S440, the subtle movements of the body can be detected through the pixel difference between the current frame and the reference frame, thereby improving the accuracy of the detection of body movement features. For example, when the limbs of the sampled object are covered with a quilt, the key points of the limbs of the human body cannot be detected by the human body key point detection method, and it is also impossible to determine whether the sampled object has moved by the change in the coordinate position of the human body key points. However, even if the quilt is covered, as long as the limbs move, the pictures of adjacent frames will change, and the pixel values ​​at the same position will also change, so it is still possible to detect whether the limbs of the sampled object have moved by the target frame difference. In other words, the method for determining the body movement features in the present disclosure can detect subtle body movements to avoid the influence of subtle movements on the abnormal discharge detection structure, thereby helping to improve the accuracy of abnormal discharge detection.

[0134] In an optional embodiment, the head movement feature includes head movement and head no movement. Figure 5 A flow chart showing a method for determining head movement characteristics in an exemplary embodiment of the present disclosure is shown. Figure 5 , the method may include steps S510 to S530.

[0135] In step S510, the head position of the sampled object in the surveillance video frame of the surveillance video sample is determined according to the head position detection model.

[0136] Exemplarily, the specific implementation of step S510 may refer to the above-mentioned step S410. The above-mentioned step S410 has already described in detail how to determine the head position of the sampled object in the surveillance video frame of the surveillance video sample, which will not be repeated here.

[0137] In step S520, the surveillance video frames of the surveillance video sample are cropped based on the head position to obtain a head surveillance video of the sampled subject.

[0138] Exemplarily, the specific implementation of step S520 may also refer to the relevant content in the above-mentioned step S410, that is, the surveillance video frames in the surveillance video sample are cropped according to the rectangular frame representing the head position to obtain the head surveillance video of the sampled object.

[0139] In step S530, the head monitoring video is input into a pre-trained head motion detection model. When the output result of the head motion detection model indicates head motion, the sample head motion feature is determined to be head motion; otherwise, the sample head motion feature is determined to be no head motion.

[0140] In an optional implementation, head training samples may be collected first, and then the head training samples may be labeled to generate labels for the head training samples. Then, supervised learning training may be performed on the head movement detection model to be trained based on the head training samples and the labels for the head training samples, thereby obtaining a pre-trained head movement detection model.

[0141] For example, the head video frame corresponding to the surveillance video frame in the surveillance video sample can be determined according to the above-mentioned step S410, and then the head video sample corresponding to the surveillance video sample can be determined according to the head video frame corresponding to the surveillance video frame in the surveillance video sample, and the head video sample can be used as the head training sample.

[0142] In an optional implementation, the training label of the head motion detection model is determined based on whether the head of the subject in the training video moves and whether there are any facial expression changes.

[0143] For example, the labels of head training samples include two types: one type is that the head of the subject does not move and the facial expression does not change during the time period corresponding to the head training sample, such as when there are no subtle facial movements such as blinking or opening the mouth, the training label of the head training sample is that the head does not move; the other type is that the head of the subject moves and / or the face of the subject changes during the time period corresponding to the head training sample, such as changes in facial expressions, subtle facial movements such as blinking, etc., the training label of the head training sample is head movement. In other words, the head movement detection model is a binary classification model, which can convert the head movement detection task into a binary classification task of whether the head moves.

[0144] In an optional embodiment, the head movement detection model can be any machine learning model capable of classification, such as a deep convolutional neural network model, etc., and this exemplary embodiment does not specifically limit this.

[0145] In an optional embodiment, after the head motion detection model is trained, the head video frame sequence corresponding to the head training sample can be input into the pre-trained head motion detection model, and the head video frame sequence can be classified into two categories by the model to determine the classification result of the head video frame sequence. When the classification result is head movement, the sample head motion feature of the head training sample is determined to be head movement. When the classification result is no head movement, the sample head motion feature of the head training sample is determined to be no head movement.

[0146] In the present disclosure, the head movement detection model not only learns the movement characteristics of the head itself, such as large movement characteristics such as rotation, but also learns subtle movement characteristics such as changes in facial expressions during the training process. Therefore, the head movement detection model in the present disclosure can automatically identify both large and subtle movements of the head to determine whether there is any movement of the head of the subject within the monitoring video time period, thereby improving the accuracy of head movement detection.

[0147] In an optional embodiment, the mouth movement feature includes mouth movement and mouth no movement. Figure 6 A flow chart showing a method for determining mouth movement characteristics in an exemplary embodiment of the present disclosure is shown. Figure 6 , the method may include steps S610 to S640. Wherein:

[0148] In step S610, the head position of the sampled object in the surveillance video frame of the surveillance video sample is determined according to the head position detection model.

[0149] Exemplarily, the specific implementation of step S610 may refer to the relevant content in the above-mentioned step S410, which will not be described in detail here.

[0150] In step S620, the facial key points of the subject are detected based on the head position to obtain the mouth key points of the subject.

[0151] In an optional implementation, after the head position of the sampled object is determined in the monitoring video frame of the monitoring video sample, the monitoring video frame can be cut according to the head position to obtain a head video sequence. Then, in the head video frame of the head video sequence, the facial key points of the sampled object are detected according to the facial key point detection method, and the mouth key points of the sampled object are determined from the facial key points.

[0152] In an optional implementation, any open source facial key point detection method may be used to detect facial key points in the head video frames of the head video sequence, and then determine the mouth key points from the detected facial key points.

[0153] In step S630, a first aspect ratio of the mouth is determined according to the target mouth key points among the mouth key points.

[0154] In an optional implementation, the target mouth key points may include a top key point, a bottom key point, a leftmost key point, and a rightmost key point among the mouth key points.

[0155] For example, the height of the mouth can be determined by the distance between the top key point and the bottom key point of the mouth key points, the width of the mouth can be determined by the distance between the leftmost key point and the rightmost key point of the mouth key points, and then the first aspect ratio of the mouth can be determined according to the ratio of the width of the mouth to the length of the mouth. In other words, the first aspect ratio of the mouth can be determined by the following formula (2):

[0156]

[0157] In formula (2), AR1 (Aspect Ratio) represents the first aspect ratio, width m Indicates the width of the mouth determined by the distance between the leftmost key point of the mouth and the rightmost key point of the mouth, height m Indicates the height of the mouth determined by the mouth's vertex keypoint (i.e. the highest point of the mouth) and the mouth's bottom keypoint (i.e. the lowest point of the mouth).

[0158] In step S640, the mouth movement feature of the sample is determined according to the first change value of the first aspect ratio corresponding to the adjacent surveillance video frames in the surveillance video sample.

[0159] In an optional embodiment, when the difference between the first aspect ratios corresponding to any adjacent surveillance video frames in the surveillance video sample is greater than a second preset value, the sample mouth movement feature can be determined to be mouth movement. When the difference between the first aspect ratios corresponding to any adjacent surveillance video frames in the surveillance video sample is less than or equal to the second preset value, the sample mouth movement feature can be determined to be mouth non-movement.

[0160] In an optional embodiment, the eye movement feature includes a left eye movement feature and a right eye movement feature, the left eye movement feature includes left eye movement and left eye no movement, and the right eye movement feature includes right eye movement and right eye no movement. For example, Figure 7 A flowchart of a method for determining eye movement characteristics in an exemplary embodiment of the present disclosure is shown. Figure 7 , the method may include steps S710 to S740.

[0161] In step S710, the head position of the sampled object in the surveillance video frame in the surveillance video sample is determined according to the head position detection model.

[0162] Exemplarily, the specific implementation of step S710 may also refer to the relevant content in the above-mentioned step S410, which will not be described again here.

[0163] In step S720, facial key points of the sampled object are detected based on the head position to obtain left eye key points and right eye key points of the sampled object.

[0164] For example, the facial key points of the subject can be detected in the surveillance video frame based on the head position according to the facial key point detection algorithm, and the left eye key points and right eye key points of the subject can be determined from the detected facial key points.

[0165] When performing facial key point detection, each key point has a corresponding label, and the facial organ represented by the key point can be determined based on the label. For example, the 1st to 6th key points represent the left eye, the 7th to 12th key points represent the right eye, etc. In other words, the left eye key point, right eye key point and the above-mentioned mouth key point of the subject can be determined based on the key point label.

[0166] In step S730, a second aspect ratio of the left eye in the monitoring video frame is determined according to the target left eye key point in the left eye key points, and a left eye motion feature is determined based on a second change value of the second aspect ratio of the left eye in adjacent monitoring video frames.

[0167] In an optional implementation, the target left-eye key point may include a top key point of the left eye, a bottom key point of the left eye, a leftmost key point of the left eye, and a rightmost key point of the left eye.

[0168] For example, the width of the left eye can be determined according to the distance between the leftmost key point of the left eye and the rightmost key point of the left eye, the height of the left eye can be determined according to the distance between the top key point of the left eye (i.e., the highest key point of the left eye) and the bottom key point of the left eye (i.e., the lowest key point of the left eye), and then the second aspect ratio can be determined according to the ratio between the width of the left eye and the height of the left eye.

[0169] In an optional embodiment, when the difference between the second aspect ratios of the left eye in any adjacent surveillance video frames in the surveillance video sample is greater than a third preset value, the left eye movement feature can be determined to be left eye movement. When the difference between the second aspect ratios of the left eye in any adjacent surveillance video frames in the surveillance video sample is less than or equal to the third preset value, the sample left eye movement feature can be determined to be left eye non-movement. In other words, as long as the difference between the second aspect ratios of the left eye in two adjacent surveillance video frames in the surveillance video sample is greater than the third preset value, the left eye movement feature is determined to be left eye movement, and when the difference between the second aspect ratios of the left eye in all adjacent surveillance video frames in the surveillance video sample is less than or equal to the third preset value, the left eye movement feature is determined to be left eye non-movement.

[0170] In step S740, a third aspect ratio of the right eye in the monitoring video frame is determined according to the target right eye key point in the right eye key points, and a right eye motion feature is determined based on a third change value of the third aspect ratio of the right eye in adjacent monitoring video frames in the monitoring video sample.

[0171] In an optional implementation, the target right eye key point may include a top key point of the right eye, a bottom key point of the right eye, a leftmost key point of the right eye, and a rightmost key point of the right eye.

[0172] For example, the width of the right eye can be determined according to the distance between the leftmost key point of the right eye and the rightmost key point of the right eye, the height of the right eye can be determined according to the distance between the top key point of the right eye (i.e., the highest key point of the right eye) and the bottom key point of the right eye (i.e., the lowest key point of the right eye), and then the third aspect ratio can be determined according to the ratio between the width of the right eye and the height of the right eye.

[0173] In an optional embodiment, when the difference between the third aspect ratios of the right eye in any adjacent surveillance video frames in the surveillance video sample is greater than a fourth preset value, the right eye movement feature can be determined to be right eye movement. When the difference between the third aspect ratios of the right eye in any adjacent surveillance video frames in the surveillance video sample is less than or equal to the fourth preset value, the sample right eye movement feature can be determined to be right eye non-movement. In other words, as long as there are two adjacent surveillance video frames in the surveillance video sample where the difference between the third aspect ratios of the right eye is greater than the fourth preset value, the right eye movement feature is determined to be right eye movement, and when the difference between the third aspect ratios of the right eye in all adjacent surveillance video frames in the surveillance video sample is less than or equal to the fourth preset value, the right eye movement feature is determined to be right eye non-movement.

[0174] In an optional embodiment, the nose movement feature includes nose movement and nose non-movement. Figure 8 A flowchart of a method for determining nose movement characteristics in an exemplary embodiment of the present disclosure is shown, with reference to Figure 8 , the party

[0175] The method may include steps S810 to S830.

[0176] In step S810, the head position of the sampled object in the surveillance video frame of the surveillance video sample is determined according to the head position detection model.

[0177] Exemplarily, the specific implementation of step S810 may also refer to the relevant content in the above-mentioned step S410, which will not be described again here.

[0178] In step S820, the facial key points of the sampled object are detected based on the head position to obtain a nose key point position sequence of the sampled object in the monitoring video sample.

[0179] Exemplarily, as mentioned above, the facial key points of the sampled object can be detected in the surveillance video frame based on the head position of the sampled object in the surveillance video frame according to any facial key point detection algorithm, and then the nose key points in the surveillance video frame can be determined according to the facial key point identifiers. According to the nose key points in the surveillance video frame in the surveillance video sample, the nose key point position sequence of the sampled object in the surveillance video sample is obtained.

[0180] In step S830, the nose movement feature is determined according to the position change of the nose key points indicated by the nose key point position sequence.

[0181] In an optional implementation, the position difference of the nose in adjacent surveillance video frames can be determined based on the nose key point position sequence, such as whether the position distance is greater than a fifth preset value, to determine the nose motion feature. For example, if the distance between the positions of the nose in any adjacent surveillance video frames in the surveillance video sample is greater than the fifth preset value, the nose motion feature is determined to be nose motion, and if the distance between the positions of the nose in any adjacent surveillance video frames in the surveillance video sample is less than or equal to the fifth preset value, the nose motion feature is determined to be nose non-motion.

[0182] For example, each nose key point detected in each surveillance video frame has nose key point coordinates, and the moving distance between adjacent video frames of the nose key point can be determined based on the coordinates of the nose key points with the same identification in adjacent surveillance video frames, such as the coordinates of the third face key point (belonging to the nose key point) in the adjacent surveillance video frames. In the case where the moving distance of any nose key point between adjacent video frames is greater than the fifth preset value, the nose motion feature corresponding to the adjacent video frame can be determined to be nose motion. In the case where the moving distance of each nose key point between adjacent video frames is less than or equal to the fifth preset value, the nose motion feature corresponding to the adjacent video frame can be determined to be nose non-motion. In the case where the nose motion feature corresponding to any adjacent surveillance video frame in the surveillance video sample is nose motion, the nose motion feature corresponding to the surveillance video sample is determined to be nose motion. In the case where the nose motion features corresponding to all adjacent surveillance video frames in the surveillance video sample are nose non-motion, the nose motion feature of the surveillance video sample is determined to be nose non-motion.

[0183] Since the nose is different from the mouth or eyes, which can move by itself, the movement of the nose follows the head. For example, when the head deflects, the nose follows the head deflection, and the nose will not be blocked by other objects such as quilts. In other words, the nose key points can be detected in each monitoring video frame. Therefore, in the present disclosure, the nose key point position sequence can be used to detect whether the nose moves. At the same time, the nose motion feature can also reflect the head movement, improving the accuracy and comprehensiveness of motion feature detection.

[0184] In an optional implementation, when labeling the training samples, since subtle movements such as blinking and slight lip movement are not easily observed manually, the motion features of the surveillance video samples can be automatically determined according to the above-mentioned related methods, such as determining whether the body, eyes, mouth, nose, and other motion features of the subject in the video surveillance sample are moving. Then, based on the motion features of the surveillance video samples, the doctor adds a label indicating whether the biomedical signal sample has abnormal discharge to the biomedical signal sample that belongs to the same period as the surveillance video sample. In this way, it can be avoided that when the doctor manually watches the surveillance video, he fails to notice subtle movements such as blinking of the subject, which leads to incorrect labeling, thereby improving the accuracy of the training sample labeling, and further assisting in improving the detection accuracy of the abnormal discharge detection model obtained by training.

[0185] Next, Figure 1 The specific implementation of step S130 in is described in detail.

[0186] In an optional embodiment, a sample motion feature vector can be determined based on the sample motion feature determined in step S120. For example, the dimension of the sample motion feature vector can be determined based on the motion feature type included in the sample motion feature, and the vector value of the corresponding position in the vector can be determined based on the motion feature of the motion feature type corresponding to the position. For example, if the motion feature of the motion feature type corresponding to a certain position is motion, then the vector value of the position is 1; if the motion feature of the motion feature type corresponding to a certain position is non-motion, then the vector value of the position is 0.

[0187] In an optional implementation, the motion feature types include the above-mentioned body motion features, head motion features, left eye motion features, right eye motion features, mouth motion features, and nose motion features, a total of 6 types. The sample motion feature includes at least one of the above-mentioned 6 motion types. If the sample motion feature includes 3 motion feature types, the sample motion feature vector is a 3D vector. If the sample motion feature includes 1 motion feature type, the sample motion feature is a 1D vector, and so on.

[0188] Taking the sample motion features including body motion features, head motion features, left eye motion features, right eye motion features, mouth motion features, and nose motion features as an example, the dimension of the sample motion feature vector is 6 dimensions. Taking the 6-dimensional vector as a row vector as an example, the motion feature types represented by different columns in the 6-dimensional row vector can be pre-set. For example, the first column represents body motion features, the second column represents head motion features, the third column represents left eye motion features, the fourth column represents right eye motion features, the fifth column represents mouth motion features, and the sixth column represents nose motion features. If the body motion feature is body motion, the head motion feature is that the head does not move, the left eye motion feature is that the left eye does not move, the right eye motion feature is that the right eye moves, the mouth motion feature is that the mouth moves, and the nose motion feature is that the nose moves, then the vector value of the sample motion feature vector is [1, 0, 0, 1, 1, 1]. Of course, the 6-dimensional vector can also be a column vector, and this exemplary embodiment does not make any special limitation on this.

[0189] Next, combine Fig. 9 An exemplary specific implementation of step S130 is described. Exemplarily, Fig. 9 A flow chart showing a method for obtaining abnormal discharge detection results of training samples in an exemplary embodiment of the present disclosure is shown. Fig. 9 The method may include steps S910 to S940. Wherein:

[0190] In step S910, a biomedical signal sample is input into a biomedical signal feature extraction network to obtain a first output.

[0191] In an exemplary embodiment, the biomedical signal samples may be preprocessed, STFT (Scale-invariant feature transform) transformed, and feature extracted to obtain processed biomedical signal samples, which are then used to perform model training.

[0192] For example, in preprocessing, a 50 Hz notch filter can be used to filter the collected electrical signals to reduce the pollution of alternating current. A bandpass filter with a frequency range of 0.1-70 Hz is applied to 21 EEG electrodes and 2 earlobe reference electrodes for filtering, and signal space projection (SSP) is used to remove heart field artifacts in the EEG signals.

[0193] The electrical signals sampled from the electrodes have time correlation, and Fourier transform can help display the signals in the frequency field. Short-time Fourier transform is good at processing short signal fragments. Therefore, STFT can be used to convert EEG data into a two-dimensional matrix consisting of frequency and time information. After STFT transformation, a total of 77 matrices transformed from EEG signals are achieved. Convolutional neural networks (CNNs) are used to extract features from these matrices to obtain low-dimensional features that are convenient for abnormal discharge detection models to use.

[0194] Exemplarily, EfficientNetV2 (a deep learning neural network) can be used for feature extraction of biomedical signals. EfficientNetV2 has superior feature extraction capabilities, is easy to train, and has low latency. In order to achieve a balance between feature extraction capabilities and time consumption, EfficientNetV2-S can be used. EfficientNetV2-S consists of a fused MBConv (Mobile Inverted Bottleneck Convolution) block and an MBConv block. The fused MBConv block has a large number of parameters, and many lightweight networks use MBConv to replace the fused MBConv block and obtain good performance. However, EfficientNetV2-S proves that the MBConv block may have an adverse effect on performance, and that a mixture of the two types of networks can balance performance, network size, and latency. After feature extraction by EfficientNetV2-S, a 1024-dimensional feature vector can be obtained, which contains the features of the biomedical signal and is combined with the motion feature vector corresponding to the monitoring video sample for abnormal discharge detection.

[0195] Of course, other deep learning models can also be used to extract features of biomedical signals. For example, VGG (Visual Geometry Group)-16, ResNet18 (Residual Networks), etc. are used, and this exemplary embodiment does not specifically limit this. However, after experimental evaluation and comparison, it was found that the single-modal model of EfficientNetV2-S, that is, only using biomedical signals for abnormal discharge detection based on EfficientNetV2-S, the AUPRC (Area Under the Precision-Recall Curve, area under the precision and recall curve) evaluation index value is 0.8316, which greatly exceeds the performance of ResNet18 and VGG-16. The multimodal model that combines biomedical signal features with motion features obtained from the video has a higher AUPRC of 0.8623 for biomedical signal feature extraction using the EfficientNetV2-S model. In addition, when the sensitivity is set to 0.8, the accuracy of EfficientNetV2-S is 76.6%, which is better than the accuracy of ResNet18 (69.2%) and VGG-16 (65.4%). Therefore, it is more recommended to use EfficientNetV2-S for feature extraction of biomedical signals for abnormal discharge detection.

[0196] In step S920, the sample motion feature is input into the motion feature enhancement network to obtain a second output.

[0197] As mentioned above, the dimension of the sample motion feature vector is less than or equal to 6. Compared with the biomedical signal sample feature vector, the vector length of the sample motion feature is too short. If the two are directly concatenated and input into the abnormal discharge detection model to be trained, the sample motion features may be ignored to a certain extent, affecting the training effect of the model.

[0198] In an optional implementation, the sample motion features may be enhanced to increase the dimension of the sample motion features. For example, taking the vector dimension of the extracted biomedical signal sample features as 1024 as an example, the sample motion features may be increased to 128 dimensions, 256 dimensions, 512 dimensions or 1024 dimensions, etc., to reduce the dimensionality gap between the sample motion features and the biomedical signal features, thereby avoiding the situation where the sample motion features are ignored due to the large gap between the two.

[0199] In an optional implementation, the motion feature enhancement network may include a multi-layer perceptron. Of course, the motion feature enhancement network may also include other networks that can convert low-dimensional features into high-dimensional features. This exemplary implementation does not specifically limit this.

[0200] In step S930, the first output and the second output are spliced ​​to obtain a target splicing result.

[0201] In an optional implementation, the first output and the second output may be spliced ​​to obtain a target splicing result, such as directly splicing the vector of the first output and the vector of the second output to obtain a higher-dimensional vector, which is used as the target splicing result.

[0202] In step S940, the target splicing result is input into the abnormal discharge detection model to be trained, and the abnormal discharge detection result of the training sample is obtained according to the output of the abnormal discharge detection model to be trained.

[0203] In an exemplary embodiment, the target splicing result can be used as the input of the abnormal discharge detection model to be trained, and then the abnormal discharge detection model to be trained can perform feature learning on the input target splicing result, and output a predicted abnormal discharge detection result according to the feature learning result.

[0204] In an optional implementation, the abnormal discharge detection model to be trained may be any machine learning model capable of classification, such as a classifier model based on a multi-layer perceptron, a classifier model based on a convolutional neural network, etc. That is, in the present disclosure, the abnormal discharge detection task may be converted into a binary classification task, and the classification results include two types: the presence of abnormal discharge and the absence of abnormal discharge.

[0205] Next, Figure 1 The specific implementation of step S140 in is described in detail.

[0206] In an optional implementation, after obtaining the abnormal discharge detection result of the training sample, the difference between the predicted abnormal discharge detection result of the training sample and the sample label of the training sample can be calculated, and the difference can be used as the training loss value of the abnormal discharge detection model to be trained.

[0207] Among them, the above difference can be specifically measured by mean absolute error, mean square error, Euclidean distance, etc., and can be set according to actual conditions. The present disclosure does not make any special limitation on this.

[0208] Next, Figure 1 The specific implementation of step S150 in is described in detail.

[0209] In an optional implementation, the preset condition may include that the number of iterative training times reaches a preset number or the loss value satisfies a preset convergence condition, such as being less than a preset loss value.

[0210] For example, the model parameters of the abnormal discharge detection model to be trained can be continuously updated according to the direction of loss value reduction until the loss value is less than a preset loss value threshold, or until the number of iterations reaches a preset number, and a trained abnormal discharge detection model can be obtained.

[0211] In an optional embodiment, the biomedical signal feature extraction network, the motion feature enhancement network and the abnormal discharge detection model to be trained are jointly trained. In other words, after obtaining the training samples and the sample labels of the training samples, the biomedical signal samples in the training samples can be input into the biomedical signal feature extraction network to obtain the first output, and then the monitoring video samples in the training samples are simultaneously input into the patient detection model and the face detection model to obtain the body position and head position of the sampled object, and then based on the head position and body position of the sampled object in the monitoring video frame, the sample motion feature corresponding to the training sample is determined according to the above-mentioned motion feature determination method, and then the sample motion feature is input into the motion feature enhancement network to obtain the second output, and then the first output and the second output are spliced ​​and input into the abnormal discharge detection model to be trained to predict the abnormal discharge detection result of the training sample. When adjusting the network parameters according to the loss value, the network parameters of the biomedical signal feature extraction network, the motion feature enhancement network and the abnormal discharge detection model to be trained can be adjusted so that the loss value of the latter training is less than the loss value of the previous training, that is, iterative learning is performed in the direction of reducing the loss value.

[0212] For example, Fig.10 A flow chart showing another method for training an abnormal discharge detection model in an exemplary embodiment of the present disclosure is shown. Fig.10 , the method may include steps S1010 to S1070. Wherein: in step S1010, a training sample and a sample label are obtained; in step S1020, a biomedical signal sample in the training sample is input into a biomedical signal feature extraction network to obtain a first output; in step S1020, a monitoring video sample is input into a patient detection model, and a patient position video sample is obtained according to the output result; in step S1030, a patient position video sample is input into a face detection model, and a head video sample and a body video sample are obtained according to the output result; in step S1040, a sample motion feature is determined according to the head video sample and the body video sample; in step S1050, the sample motion feature is input into a motion feature enhancement network to obtain a second output; in step S1060, the first output and the second output are spliced ​​and input into an abnormal discharge detection model to be trained to obtain a predicted abnormal discharge detection result; in step S1070, a model loss value is calculated according to the predicted abnormal discharge detection result and the sample label, and the model parameters are updated until a trained abnormal discharge detection model is obtained.

[0213] In step S1070, updating the model parameters can be understood as updating the parameters of at least one of the biomedical signal feature extraction network, the motion feature enhancement network, and the abnormal discharge detection model.

[0214] After the above abnormal discharge detection model is trained, abnormal discharge detection can be performed based on the trained abnormal discharge detection model. Fig.11 , Fig.11 A flow chart of an abnormal discharge detection method in an exemplary embodiment of the present disclosure is shown, and the method may include steps S1110 to S1130. In which: step S1110, determining the biomedical signal to be detected and the monitoring video to be detected of the object under test; step S1120, determining the motion characteristics of the object under test according to the monitoring video to be detected; step S1130, inputting the biomedical signal to be detected and the motion characteristics into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the object under test.

[0215] Next, combine Fig.12 and Fig.13 An exemplary implementation of step S1110 is described.

[0216] For example, Fig.12 A flow chart showing a method for determining a biomedical signal to be detected and a surveillance video to be detected in an exemplary embodiment of the present disclosure is shown, with reference to Fig.12 , the method may include steps S1210 to S1230. Wherein:

[0217] In step S1210, the collected biomedical signals and monitoring videos of the subject are obtained.

[0218] In an optional implementation, the monitoring video is captured when collecting biomedical signals from the subject.

[0219] In an optional embodiment, the test subject may be a patient who has epilepsy or epilepsy-like symptoms, or a person who seeks medical treatment for other brain discomfort, such as brain pain, etc. This exemplary embodiment does not specifically limit this.

[0220] For example, a biomedical signal acquisition device can be used to acquire biomedical signals, such as an electroencephalograph to acquire the EEG signals of the subject, and then a video of the subject is recorded while the biomedical signals are being acquired, so as to obtain a monitoring video of the subject when the biomedical signals of the subject are being acquired. The acquisition time and the recording time can be determined according to the needs, such as acquiring 30 minutes of biomedical signals and recording the subject within these 30 minutes to obtain a monitoring video of 30 minutes.

[0221] In step S1220, the biomedical signal and the monitoring video of the subject are acquired in segments according to a preset time length to obtain biomedical signal segments and monitoring video segments of the same time period.

[0222] Exemplarily, according to the relevant implementation in the above-mentioned step S110, the biomedical signal and the monitoring video can be segmented according to the same starting time and the same preset duration to obtain biomedical signal segments and monitoring video segments of the same time period.

[0223] In step S1230, the biomedical signal segments and monitoring video segments in the same period are determined as the biomedical signals to be detected and the monitoring videos to be detected of the subject.

[0224] For example, the surveillance video segment obtained in the above step S1220 may be used as the surveillance video to be detected, and the biomedical signal segment obtained in the above step S1220 may be used as the biomedical signal to be detected.

[0225] For example, Fig.13 A flow chart showing another method for determining a biomedical signal to be detected and a surveillance video to be detected in an exemplary embodiment of the present disclosure is shown, with reference to Fig.13 , the method may include steps S1310 to S1330. Wherein:

[0226] In step S1310, the collected biomedical signals of the subject are acquired.

[0227] For example, the explanation of the subject can refer to the above-mentioned step S1210, and the method of collecting biomedical signals can also refer to the above-mentioned step S1210, which will not be repeated here.

[0228] In step S1320, the biomedical signals of the subject are acquired in segments according to a preset time length to determine the biomedical signals to be detected of the subject.

[0229] For example, the biomedical signal may be segmented according to a preset duration to obtain biomedical signal segments of the subject, and the biomedical signal segments are used as the biomedical signal to be detected.

[0230] In step S1330, when the biomedical signal to be detected indicates abnormal discharge, a surveillance video segment of the same period as the biomedical signal to be detected is extracted from the surveillance video of the subject to be detected to determine the surveillance video of the subject to be detected.

[0231] In an optional embodiment, a first abnormal discharge detection model can be pre-trained, the input of the first abnormal discharge detection model is a biomedical signal, and the output is whether the biomedical signal has abnormal discharge. For example, the biomedical signal to be detected can be input into the pre-trained first abnormal discharge detection model to obtain a prediction result of whether the biomedical signal to be detected has abnormal discharge. In the case where the prediction result of the biomedical signal to be detected is the presence of abnormal discharge, a monitoring segment of the same period as the biomedical signal to be detected is cut out from the monitoring video corresponding to the biomedical signal collected in step S1310, and the monitoring video segment is used as the monitoring video to be detected. Then, the biomedical signal to be detected and the monitoring video to be detected are combined to detect abnormal discharge.

[0232] In other words, when performing abnormal discharge detection, the present disclosure can obtain the collected biomedical signal and the monitoring video, first segment the biomedical signal, and use each segmented biomedical signal segment as the biomedical signal to be detected, and then perform abnormal discharge detection on the biomedical signal to be detected, that is, first input the biomedical signal to be detected into the above-mentioned first abnormal discharge detection model to obtain the abnormal discharge prediction result of the biomedical signal to be detected, and when the abnormal discharge prediction result indicates that the biomedical signal to be detected is discharged normally, it can be directly determined that the object under test is discharged normally in the time period corresponding to the biomedical signal to be detected; when the abnormal discharge prediction result indicates that the biomedical signal to be detected is discharged abnormally, a monitoring video segment of the same time period as the biomedical signal with detection having abnormal discharge can be intercepted from the acquired monitoring video, and the monitoring video segment is determined as the monitoring video to be detected, and then the biomedical signal to be detected and the monitoring video with detection determined according to the biomedical signal to be detected are combined, and according to the subsequent steps S1120 and S1130, it is determined whether the biomedical signal to be detected really has abnormal discharge.

[0233] If the subsequent step S1130 determines that the abnormal discharge detection result is abnormal discharge, it is determined that the subject has abnormal discharge in the time period corresponding to the biomedical signal to be detected. If the subsequent step S1130 determines that the abnormal discharge detection result is normal discharge, that is, there is no abnormal discharge, it is determined that the subject has no abnormal discharge in the time period corresponding to the biomedical signal to be detected. If the subject has no abnormal discharge in the time period corresponding to each biomedical signal to be detected, it is determined that the abnormal discharge detection result of the subject is normal. If the subject has abnormal discharge in the time period corresponding to any biomedical signal to be detected, it is determined that the abnormal discharge detection result of the subject is abnormal discharge, and the biomedical signal segment with abnormal discharge and its corresponding monitoring video segment are output to facilitate the doctor to reconfirm the detection result.

[0234] Through the above steps S1310 to S1330, it is possible to first determine whether abnormal discharge exists based on the biomedical signal to be detected in a single mode. When abnormal discharge exists in the biomedical signal to be detected, the biomedical signal to be detected and the corresponding monitoring video to be detected are combined to determine whether abnormal discharge really exists based on multi-modal input. Since the processing efficiency of single-modal input is greater than the processing efficiency of multi-modal input, and the prediction accuracy of multi-modal input is higher than the prediction accuracy of multi-modal input, it is possible to improve the detection efficiency of abnormal discharge and ensure the accuracy of abnormal discharge detection.

[0235] The specific implementation of step S1120 is described in detail below.

[0236] In an optional embodiment, the motion characteristics of the subject may include at least one of body motion characteristics, head motion characteristics, eye motion characteristics, mouth motion characteristics, and nose motion characteristics.

[0237] In an optional embodiment, body movement features include body movement and body non-movement, and a method for determining the body movement features includes: for any video frame of the surveillance video to be detected, determining the head position of the subject in the video frame according to a head position detection model, and cropping the video frame according to the head position to obtain a body video frame corresponding to the video frame; for any body video frame, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining the sum of pixel value differences at the same position between the body video frame and the reference video frame, and determining the frame difference of the body video frame according to the maximum value of the sum of the pixel value differences; and determining the body movement features of the subject according to the frame difference.

[0238] For example, the method for determining the body movement characteristics of the subject can refer to the above Figure 4 The method for determining the sample body movement characteristics shown will not be described in detail here.

[0239] In an optional embodiment, the head movement feature includes head movement and no head movement, and the method for determining the head movement feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected based on a head position detection model; cropping the video frame of the surveillance video to be detected based on the head position to obtain a head monitoring video of the subject; inputting the head monitoring video into a pre-trained head movement detection model, and determining the head movement feature as head movement when the output result of the head movement detection model indicates head movement, otherwise, determining the head movement feature as no head movement; wherein the training label of the head movement detection model is determined based on whether the head of the object in the training video moves and whether there are changes in facial expression.

[0240] For example, the method for determining the head movement characteristics of the subject can refer to the above Figure 5 The method for determining the head movement characteristics shown will not be described in detail here.

[0241] In an optional embodiment, the mouth movement feature includes mouth movement and mouth non-movement, the eye movement feature includes left eye movement feature and right eye movement feature, the left eye movement feature includes left eye movement and left eye non-movement, the right eye movement feature includes right eye movement and right eye non-movement, and the method for determining the mouth movement feature and the eye movement feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected according to the head position detection model; detecting the facial key points of the subject based on the head position to obtain the mouth key points, left eye key points and right eye key points of the subject; and determining the mouth movement feature and the eye movement feature according to the head position detection model. The target mouth key point among the mouth key points determines the first aspect ratio of the mouth; the second aspect ratio of the left eye is determined according to the target left eye key point among the left eye key points, and the third aspect ratio of the right eye is determined according to the target right eye key point among the right eye key points; the mouth motion feature is determined according to a first change value of the first aspect ratio corresponding to adjacent frames in the surveillance video to be detected; the left eye motion feature is determined according to a second change value of the second aspect ratio corresponding to adjacent frames in the surveillance video to be detected; and the right eye motion feature is determined according to a third change value of the third aspect ratio corresponding to adjacent frames in the surveillance video to be detected.

[0242] For example, the method for determining the mouth movement characteristics and the method for determining the eye movement characteristics of the subject can refer to the above Figure 6 The sample mouth movement features shown are determined in the manner and Figure 7The method for determining the sample mouth movement characteristics shown will not be described in detail here.

[0243] In an optional embodiment, the nose movement feature includes nose movement and nose non-movement, and the method for determining the nose movement feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected based on a head position detection model; detecting the facial key points of the subject based on the head position to obtain a nose key point position sequence of the subject in the surveillance video to be detected; and determining the nose movement feature based on the change in the nose key point position indicated by the nose key point position sequence.

[0244] For example, the method for determining the nose movement characteristics of the subject can refer to the above Figure 8 The method for determining the sample nose movement features shown will not be described in detail here.

[0245] Next, combine Fig.14 The specific implementation of step S1230 is described in detail.

[0246] For example, Fig.14 A flow chart showing a method for obtaining abnormal discharge detection results in an exemplary embodiment of the present disclosure is shown. Fig.14 , the method may include steps S1410 to S1440. Wherein: in step S1410, the biomedical signal to be detected is input into the biomedical signal feature extraction network to obtain the biomedical signal feature corresponding to the biomedical signal to be detected; in step S1420, the motion feature is enhanced to increase the feature dimension of the motion feature to obtain the target motion feature; in step S1430, the biomedical signal feature and the target motion feature are spliced ​​to obtain the spliced ​​feature; in step S1440, the spliced ​​feature is input into the pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the object under test.

[0247] For example, the specific implementation of step S1410 to step S1440 may refer to the specific implementation of step S910 to step S940 described above, and will not be described in detail here.

[0248] In an optional implementation, when the abnormal discharge detection result predicted by the abnormal discharge detection model is that there is an abnormality, it is determined that the subject has abnormal discharge in the time period corresponding to the biomedical signal to be detected, which means that the subject has abnormal discharge. When the abnormal discharge detection result predicted by the abnormal discharge detection model is that there is no abnormality, it is determined that the subject has no abnormal discharge in the time period corresponding to the biomedical signal to be detected.

[0249] For example, Fig.15 The structure diagram of an abnormal discharge detection system in an exemplary embodiment of the present disclosure is shown, which may include a data acquisition module 1510 , a biomedical signal and vital sign extraction module 1520 , a motion feature determination module 1530 , a feature fusion module 1540 and an abnormality detection classification result determination module 1550 .

[0250] In an optional embodiment, the data in the present disclosure is from multimodal electrical data (such as electroencephalogram, electrocardiogram (ECG) and electromyogram (EMG)) and video data. That is, the data acquisition module 1510 acquires two types of data, namely, biomedical signal electrical data and monitoring video data. Then, the data acquisition module 1510 sends the acquired biomedical signal electrical data to the biomedical signal vital sign extraction module 1520, and sends the monitoring video data to the motion feature determination module 1530. Then, the biomedical signal feature extraction module 1510 transforms the electrical data by short-time Fourier transform (STFT), and then uses Efficien tNetV2-S is used to extract meaningful electrical signal features. The motion feature determination module 1540 decomposes the video into image frames, uses a detection model based on YOLOv5 to detect the patient and the face, and obtains the patient's facial video sequence and torso video sequence, and determines the patient's motion features based on the patient's facial video sequence and torso video sequence according to the above-mentioned motion feature method. Finally, after the feature fusion module 1540 enhances the motion features, the biomedical features and the enhanced motion features are spliced ​​and fused, and then input into the classifier in the abnormal detection classification result determination module 1550, and the classifier can output the abnormal discharge detection result according to the fused features.

[0251] The abnormal discharge detection in the brain is taken as an example below to illustrate the abnormal discharge detection method in the present disclosure in combination with specific application scenarios.

[0252] refer to Fig.16 , Fig.16 A schematic diagram showing an exemplary application scenario of the abnormal discharge detection method according to an embodiment of the present disclosure is shown. Fig.16 As shown: 1601 is the object under test, 1603a, 1603b and 1603c are detection electrodes, 1607 is an EEG signal acquisition device, 1605 is an interface of the EEG signal acquisition device 1607, 1609 is an abnormal discharge detection device developed based on the abnormal brain discharge detection method in the present disclosure, 1611 is a display (which may include the mobile phone, computer, etc. of the object under test, and the present disclosure does not make special limitations on this), and 1616 is a camera.

[0253] For example, the detection electrodes 1603a, 1603b, and 1603c connected to the EEG signal acquisition device 1607 can be fixed at various positions on the head of the subject 1601, so as to acquire multi-channel EEG signals of the subject 1601 through the EEG signal acquisition device 1607. The EEG signal acquisition device 1607 can also include devices in various forms such as helmets, hats, bedding, and packaged portable detection electrodes, which are not particularly limited in the present disclosure.

[0254] While collecting the above-mentioned multi-channel EEG signals, the subject 1601 can be synchronously monitored through the camera 1616 to capture video monitoring data including the face and torso of the subject 1601 to record the movement status of the subject.

[0255] After obtaining the multi-channel EEG signals and video monitoring data of the subject, the multi-channel EEG signals and video monitoring data can be transmitted to the abnormal discharge detection device 1609, and then the abnormal discharge detection device 1609 can output the abnormal discharge detection result for the subject based on the abnormal brain discharge detection method disclosed in the present invention. After obtaining the abnormal discharge detection result, the abnormal discharge detection result can be displayed on the display 1611 so that the doctor or the subject can understand the specific detection result.

[0256] It should be noted that the EEG signal acquisition device 1607 in the present disclosure can also be coupled with the abnormal discharge detection device 1609, that is, they can be integrated into the same device, and the present disclosure does not make any special limitation on this.

[0257] refer to Fig.17 , Fig.17 A schematic diagram showing another exemplary application scenario of the abnormal discharge detection method in the embodiment of the present disclosure is shown. Fig.17 As shown: the EEG signal acquisition device 1607 in this scene can be a portable device, and the camera 1616 can be a camera carried by a portable terminal such as a mobile phone or PAD, so that the subject can collect EEG signals and video monitoring data anytime and anywhere at home, office, etc. 1610 is a data transceiver, and 1615 is a server. The server 1615 can include any form of data processing server such as a cloud server and a distributed server, and the present disclosure does not make any special restrictions on this.

[0258] After the EEG signal acquisition device 1607 acquires the multi-channel EEG signals of the subject and the camera 1616 acquires the video surveillance data of the subject, the multi-channel EEG signals and video surveillance data can be transmitted to the data transceiver 1610, and then the data transceiver 1610 can upload the multi-channel EEG signals and video surveillance data to the server 1615.

[0259] Furthermore, the server 1615 can output abnormal discharge detection results for the subject based on the abnormal brain discharge detection method disclosed in the present invention. After outputting the abnormal discharge detection results, the abnormal discharge detection results can be displayed on the display 1611 so that the doctor or the subject can understand the specific detection results.

[0260] Through the above scenarios, the subjects can learn about the abnormal discharge detection results without leaving home. When encountering problems such as suspected epilepsy, the abnormal discharge detection results can be sent to the doctor to facilitate the doctor to determine the diagnosis results.

[0261] Based on the above technical solutions, the present disclosure can at least achieve the following technical effects:

[0262] On the one hand, the motion characteristics of the object to be tested are determined through the monitoring video to be tested, and the motion characteristics are combined with biomedical signals to detect abnormal discharges. The motion characteristics can eliminate the interference of the movement of the object to be tested on the abnormal discharge detection results, thereby improving the accuracy of abnormal discharge detection. On the other hand, the abnormal discharge detection results can be obtained without manual image reading, which improves the detection efficiency for scenarios that require a large number of abnormal discharge detections. On another hand, since the abnormal discharge detection results determined by the method of the present invention are highly accurate, it can provide a high reference value for doctors, assist doctors in making clinical diagnoses quickly and accurately, save doctors' workload, and facilitate widespread application in clinical practice.

[0263] Exemplary Devices

[0264] After introducing the abnormal discharge detection method according to the exemplary embodiment of the present disclosure, next, referring to Fig.18 An abnormal discharge detection device in an exemplary embodiment of the present disclosure will be described.

[0265] Fig.18 An abnormal discharge detection device 1800 in an exemplary embodiment of the present disclosure is shown, including: a module 1810 for determining data to be detected, configured to determine a biomedical signal to be detected and a surveillance video to be detected of a subject; a motion feature determination module 1820, configured to determine the motion feature of the subject according to the surveillance video to be detected, wherein the motion feature includes at least one of a body motion feature, a head motion feature, an eye motion feature, a mouth motion feature, and a nose motion feature; a first detection module 1830, configured to input the biomedical signal to be detected and the motion feature into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0266] In an optional embodiment, the module 1810 for determining the data to be detected may be specifically configured to: obtain the collected biomedical signals and monitoring videos of the subject, wherein the monitoring videos are taken when the biomedical signals of the subject are collected; obtain the biomedical signals and monitoring videos of the subject in segments according to preset time lengths to obtain biomedical signal segments and monitoring video segments of the same time period; and determine the biomedical signal segments and monitoring video segments of the same time period as the biomedical signals to be detected and the monitoring video to be detected of the subject.

[0267] In an optional embodiment, the module 1810 for determining the data to be detected may also be specifically configured as follows: acquiring the collected biomedical signals of the subject; acquiring the biomedical signals of the subject in segments according to a preset time length to determine the biomedical signals to be detected of the subject; when the biomedical signals to be detected indicate abnormal discharge, extracting a surveillance video segment of the same period as the biomedical signals to be detected from the surveillance video of the subject to determine the surveillance video of the subject to be detected.

[0268] In an optional embodiment, when the motion feature includes a body motion feature, the body motion feature includes body movement and body non-movement, and a method for determining the body motion feature includes: for any monitoring video frame of the monitoring video to be detected, determining the head position of the subject in the monitoring video frame according to a head position detection model, and cropping the monitoring video frame according to the head position to obtain a body video frame corresponding to the monitoring video frame; for any body video frame, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining a frame difference corresponding to the reference video frame according to the sum of pixel value differences at the same position between the body video frame and the reference video frame; determining a target frame difference of the body video frame according to the maximum value of the frame difference corresponding to the reference video frame, and determining the body motion feature of the subject based on the target frame difference.

[0269] In an optional embodiment, when the motion feature includes a head motion feature, the head motion feature includes head movement and no head movement, and the method for determining the head motion feature includes: determining the head position of the object under test in the video frame of the surveillance video to be detected based on a head position detection model; cropping the video frame of the surveillance video to be detected based on the head position to obtain a head surveillance video of the object under test; inputting the head surveillance video into a pre-trained head motion detection model, and when the output result of the head motion detection model indicates head movement, determining the head motion feature as head movement, otherwise, determining the head motion feature as no head movement; wherein the training label of the head motion detection model is determined based on whether the head of the object moves and whether there are facial expression changes in the training video.

[0270] In an optional embodiment, when the motion feature includes a mouth motion feature and / or an eye motion feature, the mouth motion feature includes a mouth motion and a mouth not moving, the eye motion feature includes a left eye motion feature and a right eye motion feature, the left eye motion feature includes a left eye motion and a left eye not moving, the right eye motion feature includes a right eye motion and a right eye not moving, and the method for determining the mouth motion feature and the eye motion feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected according to a head position detection model; detecting the facial key points of the subject based on the head position to obtain the mouth of the subject key points, left eye key points and right eye key points; determining a first aspect ratio of the mouth according to a target mouth key point in the mouth key points; determining a second aspect ratio of the left eye according to a target left eye key point in the left eye key points, and determining a third aspect ratio of the right eye according to a target right eye key point in the right eye key points; determining the mouth motion feature according to a first change value of the first aspect ratio corresponding to adjacent frames in the surveillance video to be detected; determining the left eye motion feature according to a second change value of the second aspect ratio corresponding to adjacent frames in the surveillance video to be detected; determining the right eye motion feature according to a third change value of the third aspect ratio corresponding to adjacent frames in the surveillance video to be detected.

[0271] In an optional embodiment, when the motion feature includes a nose motion feature, the nose motion feature includes nose movement and nose non-movement, and a method for determining the nose motion feature includes: determining the head position of the subject in the video frame of the surveillance video to be detected based on a head position detection model; detecting the facial key points of the subject based on the head position to obtain the nose key point position of the subject; and determining the nose motion feature based on the change in the nose key point position indicated by the nose key point position sequence in the video frame of the surveillance video to be detected.

[0272] In an optional embodiment, the first detection module 1830 can be specifically configured to: input the biomedical signal to be detected into a biomedical signal feature extraction network to obtain a biomedical signal feature corresponding to the biomedical signal to be detected; perform feature enhancement on the motion feature to increase the feature dimension of the motion feature to obtain a target motion feature; splice the biomedical signal feature and the target motion feature to obtain a spliced ​​feature; input the spliced ​​feature into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject.

[0273] In an optional implementation, the biomedical signal includes any one of a biomedical signal numerical sequence and a biomedical signal curve image.

[0274] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the above-mentioned abnormal discharge detection method invention implementation method, and will not be repeated here.

[0275] Next, refer to Fig.19 A training device for an abnormal discharge detection model in an exemplary embodiment of the present disclosure is described. Fig.19 The training device 1900 of the abnormal discharge detection model in an exemplary embodiment of the present disclosure is shown, including: a training sample acquisition module 1910, configured to acquire training samples and sample labels corresponding to the training samples, wherein the training samples include biomedical signal samples and monitoring video samples of the subject, wherein the monitoring video samples are obtained by photographing the subject when collecting the biosignal data samples of the subject; a sample motion feature determination module 1920, configured to determine the sample motion features corresponding to the monitoring video samples, wherein the sample motion features include sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features features, sample nose motion features; a second detection module 1930, configured to input the sample motion features corresponding to the biomedical signal sample and the monitoring video sample into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample; a loss determination module 1940, configured to determine the loss value of the abnormal discharge detection model to be trained according to the difference between the abnormal discharge detection result of the training sample and the sample label; an iterative training module 1950, configured to iteratively train the abnormal discharge detection model to be trained according to the loss value until a preset condition is met to obtain a pre-trained abnormal discharge detection model.

[0276] In an optional embodiment, the second detection module 1930 can be specifically configured as follows: inputting the biomedical signal sample into a biomedical signal feature extraction network to obtain a first output; inputting the sample motion feature into a motion feature enhancement network to obtain a second output; splicing the first output and the second output to obtain a target splicing result; inputting the target splicing result into an abnormal discharge detection model to be trained, and obtaining the abnormal discharge detection result of the training sample according to the output of the abnormal discharge detection model to be trained.

[0277] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the inventive implementation of the above method and will not be repeated here.

[0278] Exemplary Storage Media

[0279] The storage medium according to the exemplary embodiment of the present disclosure is described below.

[0280] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus, or a device.

[0281] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0282] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0283] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0284] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (FAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0285] Exemplary Electronic Devices

[0286] refer to Fig. 20 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0287] Fig. 20 The electronic device 2000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0288] like Fig. 20 As shown, the electronic device 2000 is in the form of a general computing device. The components of the electronic device 2000 may include, but are not limited to: at least one processing unit 2010, at least one storage unit 2020, a bus 2030 connecting different system components (including the storage unit 2020 and the processing unit 2010), and a display unit 2040.

[0289] The storage unit stores program codes, which can be executed by the processing unit 2010, so that the processing unit 2010 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 2010 can perform the following steps: Figure 1 The method steps shown and / or Fig.11 The steps of the method shown, etc.

[0290] The storage unit 2020 may include a volatile storage unit, such as a random access storage unit (RAM) 2021 and / or a cache storage unit 2022 , and may further include a read-only storage unit (ROM) 2023 .

[0291] The storage unit 2020 may also include a program / utility 2024 having a set (at least one) of program modules 2025, such program modules 2025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0292] The bus 2030 may include a data bus, an address bus, and a control bus.

[0293] The electronic device 2000 can also communicate with one or more external devices 2100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and this communication can be performed through an input / output (I / O) interface 2050. The electronic device 2000 also includes a display unit 2040, which is connected to the input / output (I / O) interface 2050 for display. In addition, the electronic device 2000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 2060. As shown in the figure, the network adapter 2060 communicates with other modules of the electronic device 2000 through a bus 2030. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 2000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0294] It should be noted that, although several modules or submodules of the device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0295] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0296] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. An abnormal discharge detection method, characterized in that: include: Determine the biomedical signals to be detected and the monitoring videos to be detected of the subject; Determine the motion characteristics of the detected object according to the surveillance video to be detected, wherein the motion characteristics include at least one of body motion characteristics, head motion characteristics, eye motion characteristics, mouth motion characteristics, and nose motion characteristics; The biomedical signal to be detected and the motion feature are input into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the detected object.

2. The abnormal discharge detection method according to claim 1, characterized in that: Determining the biomedical signal to be detected and the monitoring video to be detected of the object to be detected includes: Acquiring the collected biomedical signals and monitoring video of the subject, wherein the monitoring video is captured when the biomedical signals of the subject are collected; Acquiring the biomedical signal and monitoring video of the subject in segments according to a preset time length to obtain biomedical signal segments and monitoring video segments of the same time period; The biomedical signal segments and monitoring video segments in the same period are determined as the biomedical signals to be detected and the monitoring videos to be detected of the subject.

3. The abnormal discharge detection method according to claim 1, characterized in that: Determining the biomedical signal to be detected and the monitoring video to be detected of the object to be detected includes: Acquiring biomedical signals of the subject under test; Acquire the biomedical signal of the subject in segments according to a preset time length to determine the biomedical signal to be detected of the subject; In the case that the biomedical signal to be detected indicates abnormal discharge, a monitoring video segment of the same period as the biomedical signal to be detected is intercepted from the monitoring video of the object to be detected to determine the monitoring video of the object to be detected.

4. The abnormal discharge detection method according to claim 1, characterized in that: In the case where the motion feature includes a body motion feature, the body motion feature includes body motion and body non-motion, and the body motion feature is determined in the following manner: For any surveillance video frame of the surveillance video to be detected, determine the head position of the subject in the surveillance video frame according to the head position detection model, and crop the surveillance video frame according to the head position to obtain a body video frame corresponding to the surveillance video frame; For any of the body video frames, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; For each reference video frame, a frame difference corresponding to the reference video frame is determined according to the sum of pixel value differences at the same position between the body video frame and the reference video frame; A target frame difference of the body video frame is determined according to the maximum value of the frame difference corresponding to the reference video frame, and the body movement characteristics of the subject are determined based on the target frame difference.

5. The abnormal discharge detection method according to claim 1, characterized in that: In the case where the motion feature includes a head motion feature, the head motion feature includes head motion and head non-motion, and the manner of determining the head motion feature includes: Determine the head position of the subject in the video frame of the surveillance video to be detected according to the head position detection model; Based on the head position, the video frame of the monitoring video to be detected is cut to obtain a head monitoring video of the object under test; Inputting the head monitoring video into a pre-trained head movement detection model, and determining the head movement feature as head movement when an output result of the head movement detection model indicates head movement, otherwise determining the head movement feature as no head movement; The training label of the head motion detection model is determined according to whether the head of the object in the training video moves and whether there are facial expression changes.

6. A training method for an abnormal discharge detection model, characterized in that: include: Acquire training samples and sample labels corresponding to the training samples, wherein the training samples include biomedical signal samples and monitoring video samples of the subject, wherein the monitoring video samples are obtained by photographing the subject when collecting the biomedical signal samples of the subject; Determine a sample motion feature corresponding to the surveillance video sample, wherein the sample motion feature includes at least one of a sample body motion feature, a sample head motion feature, a sample eye motion feature, a sample mouth motion feature, and a sample nose motion feature; Inputting the sample motion features corresponding to the biomedical signal sample and the monitoring video sample into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample; Determining a loss value of the abnormal discharge detection model to be trained according to a difference between the abnormal discharge detection result of the training sample and the sample label; The abnormal discharge detection model to be trained is iteratively trained according to the loss value until a preset condition is met, thereby obtaining a pre-trained abnormal discharge detection model.

7. An abnormal discharge detection device, characterized in that: include: A to-be-detected data determination module is configured to determine a to-be-detected biomedical signal and a to-be-detected surveillance video of a tested object; A motion feature determination module is configured to determine the motion feature of the detected object according to the surveillance video to be detected, wherein the motion feature includes at least one of a body motion feature, a head motion feature, an eye motion feature, a mouth motion feature, and a nose motion feature; The first detection module is configured to input the biomedical signal to be detected and the motion feature into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the detected object.

8. A training device for an abnormal discharge detection model, characterized in that: include: A training sample acquisition module is configured to acquire training samples and sample labels corresponding to the training samples, wherein the training samples include biomedical signal samples and monitoring video samples of the subject, wherein the monitoring video samples are obtained by photographing the subject when collecting the biomedical signal samples of the subject; A sample motion feature determination module is configured to determine a sample motion feature corresponding to the surveillance video sample, wherein the sample motion feature includes at least one of a sample body motion feature, a sample head motion feature, a sample eye motion feature, a sample mouth motion feature, and a sample nose motion feature; A second detection module is configured to input the sample motion features corresponding to the biomedical signal sample and the monitoring video sample into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample; A loss determination module, configured to determine a loss value of the abnormal discharge detection model to be trained according to a difference between the abnormal discharge detection result of the training sample and the sample label; The iterative training module is configured to iteratively train the abnormal discharge detection model to be trained according to the loss value until a preset condition is met to obtain a pre-trained abnormal discharge detection model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 6.

Images