Abnormal Discharge Detection Method, Model Training Method, Device, Medium and Equipment

Abstract

The embodiments of the present disclosure relate to an abnormal discharge detection method, an abnormal discharge detection device, a training method for an abnormal discharge detection model, a training device for an abnormal discharge detection model, a computer-readable storage medium, and an electronic device, and relate to the field of artificial intelligence technology. Among them, the above abnormal discharge detection method includes: acquiring the biomedical signal and the monitoring video of the object under test, where the monitoring video is obtained by photographing the object under test when collecting the biomedical signal of the object under test; obtaining a biomedical signal image according to the signal curve of the biomedical signal; inputting the biomedical signal image and the monitoring video into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the object under test. The present disclosure combines the video actions of the object under test and the biomedical signal image to detect abnormal discharges, which can improve the accuracy of detection.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, embodiments of the present disclosure relate to an abnormal discharge detection method, a training method for an abnormal discharge detection model, an abnormal discharge detection device, a training device for an abnormal discharge detection model, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section aims to provide background or context for the embodiments of the present disclosure stated in the claims. The description herein is not admitted to be prior art merely because it is included in this section.

[0003] Electroencephalogram is an important basis for judging whether the brain function of a patient is abnormal and has important value for clinical diagnosis and assessment of the severity of the condition. Currently, doctors mainly view the electroencephalogram waveform to determine whether there is an abnormality in the patient's brain. Summary of the Invention

[0004] However, for doctors, the above solution has a large workload, and in the case of a large number of patients, doctors will inevitably make inaccurate judgments due to fatigue and the like.

[0005] Therefore, there is a great need for an abnormal discharge detection method that can quickly obtain highly accurate abnormal discharge detection results, reduce the workload of doctors, and assist doctors in making more accurate clinical diagnoses.

[0006] In this context, embodiments of the present disclosure are expected to provide an abnormal discharge detection method, an abnormal discharge detection device, a training method for an abnormal discharge detection model, a training device for an abnormal discharge detection model, a computer-readable storage medium, and an electronic device.

[0007] According to a first aspect of the embodiments of the present disclosure, there is provided an abnormal discharge detection method, including: obtaining a biomedical signal and a monitoring video of a subject under test, where the monitoring video is obtained by photographing the subject under test when the biomedical signal of the subject under test is collected; obtaining a biomedical signal image according to the signal curve of the biomedical signal; and inputting the biomedical signal image and the monitoring video into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject under test.

[0008] In an alternative embodiment, the step of inputting the biomedical signal image and the monitoring video into the abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested includes: slicing the biomedical signal image according to a preset duration to obtain sub-images of the biomedical signal; segmenting the monitoring video according to the preset duration to obtain a monitoring video segment that belongs to the same time period as the sub-images of the biomedical signal; and inputting the sub-images of the biomedical signal and the monitoring video segment into the abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested.

[0009] In an alternative embodiment, the step of inputting the biomedical signal image and the monitoring video into the abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested includes: slicing the biomedical signal image according to a preset duration to obtain sub-images of the biomedical signal; when the sub-images of the biomedical signal indicate that the subject to be tested has abnormal discharges, extracting from the monitoring video a monitoring video segment that belongs to the same time period as the sub-images of the biomedical signal; and inputting the sub-images of the biomedical signal and the monitoring video segment into the abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested.

[0010] In an alternative embodiment, the biomedical signal includes multi-channel biomedical signals. The step of obtaining a biomedical signal image according to the signal curves of the biomedical signals includes: obtaining one biomedical signal image according to the signal curves of the multi-channel biomedical signals; or obtaining multiple biomedical signal images according to the signal curves of the multi-channel biomedical signals.

[0011] In an alternative embodiment, the step of inputting the biomedical signal image and the monitoring video into the abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested includes: inputting the biomedical signal image into a biomedical signal feature extraction network to obtain biomedical signal features; determining the motion features of the subject to be tested according to the monitoring video, where the motion features include at least one of body motion features, head motion features, eye motion features, mouth motion features, and nose motion features; and inputting the biomedical signal features and the motion features into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested.

[0012] In an alternative embodiment, the step of inputting the biomedical signal feature and the motion feature into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested includes: enhancing the feature of the motion feature to increase the feature dimension of the motion feature to obtain a target motion feature; splicing the biomedical signal feature and the target motion feature to obtain a spliced feature; and inputting the spliced feature into the pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested.

[0013] In an alternative embodiment, when the motion feature includes a body motion feature, the body motion feature includes body motion and body non-motion, and the determination method of the body motion feature includes: for any monitoring video frame of the monitoring video to be detected, determining the head position of the subject to be tested in the monitoring video frame according to a head position detection model, and cropping the monitoring video frame according to the head position to obtain a body video frame corresponding to the monitoring video frame; for any one of the body video frames, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining a frame difference corresponding to the reference video frame according to the sum of pixel value differences at the same position between the body video frame and the reference video frame; determining a target frame difference of the body video frame according to the maximum value among the frame differences corresponding to the reference video frames, and determining the body motion feature of the subject to be tested based on the target frame difference.

[0014] In an alternative embodiment, when the motion feature includes a head motion feature, the head motion feature includes head motion and head non-motion, and the determination method of the head motion feature includes: determining the head position of the subject to be tested in the video frame of the monitoring video to be detected according to a head position detection model; cropping the video frame of the monitoring video to be detected based on the head position to obtain a head monitoring video of the subject to be tested; inputting the head monitoring video into a pre-trained head motion detection model, and when the output result of the head motion detection model indicates head motion, determining the head motion feature as head motion, otherwise, determining the head motion feature as head non-motion; wherein, the training label of the head motion detection model is determined according to whether the head of the object in the training video moves and whether there are facial expression changes.

[0015] In an alternative embodiment, when the motion features include mouth motion features and / or eye motion features, the mouth motion features include mouth motion and no mouth motion, the eye motion features include left-eye motion features and right-eye motion features, the left-eye motion features include left-eye motion and no left-eye motion, the right-eye motion features include right-eye motion and no right-eye motion, and the determination methods for the mouth motion features and the eye motion features are as follows: Determine the head position of the subject in the video frame of the to-be-detected surveillance video according to the head position detection model; Detect the facial key points of the subject based on the head position to obtain the mouth key points, left-eye key points, and right-eye key points of the subject; Determine the first aspect ratio of the mouth according to the target mouth key points among the mouth key points; Determine the second aspect ratio of the left eye according to the target left-eye key points among the left-eye key points, and determine the third aspect ratio of the right eye according to the target right-eye key points among the right-eye key points; Determine the mouth motion features according to the first change value of the first aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; Determine the left-eye motion features according to the second change value of the second aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; Determine the right-eye motion features according to the third change value of the third aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video.

[0016] In an alternative embodiment, when the motion features include nose motion features, the nose motion features include nose motion and no nose motion, and the determination method for the nose motion features is as follows: Determine the head position of the subject in the video frame of the to-be-detected surveillance video according to the head position detection model; Detect the facial key points of the subject based on the head position to obtain the nose key point position of the subject; Determine the nose motion features according to the change situation of the nose key point positions indicated by the nose key point position sequence in the video frame of the to-be-detected surveillance video.

[0017] According to a second aspect of the embodiments of the present disclosure, a method for training an abnormal discharge detection model is provided, including: obtaining training samples and sample labels corresponding to the training samples, where the training samples include biomedical signal samples and monitoring video samples of a subject to be tested, and the monitoring video samples are obtained by photographing the subject to be tested when collecting the biomedical signal samples of the subject to be tested; obtaining biomedical signal image samples according to the signal curves of the biomedical signal samples; inputting the biomedical signal image samples and the monitoring video samples into an abnormal discharge detection model to be trained to obtain the abnormal discharge detection results of the training samples; determining the loss value of the abnormal discharge detection model to be trained according to the difference between the abnormal discharge detection results of the training samples and the sample labels; and performing iterative training on the abnormal discharge detection model to be trained according to the loss value until a preset condition is met, to obtain a pre-trained abnormal discharge detection model.

[0018] In an alternative embodiment, the step of inputting the biomedical signal samples and the sample motion features corresponding to the monitoring video samples into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection results of the training samples includes: the step of inputting the biomedical signal image samples and the monitoring video samples into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection results of the training samples includes: determining the sample motion features corresponding to the monitoring video samples, where the sample motion features include at least one of sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, and sample nose motion features; inputting the biomedical signal image samples into a biomedical signal feature extraction network to obtain a first output; inputting the sample motion features into a motion feature enhancement network to obtain a second output; splicing the first output and the second output to obtain a target splicing result; and inputting the target splicing result into the abnormal discharge detection model to be trained, and obtaining the abnormal discharge detection results of the training samples according to the output of the abnormal discharge detection model to be trained.

[0019] According to a third aspect of the present disclosure, an abnormal discharge detection device is provided, including: a data acquisition module configured to acquire the biomedical signals and monitoring videos of a subject to be tested, where the monitoring videos are obtained by photographing the subject to be tested when collecting the biomedical signals of the subject to be tested; an image generation module configured to obtain biomedical signal images according to the signal curves of the biomedical signals; and a first detection module configured to input the biomedical signal images and the monitoring videos into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection results of the subject to be tested.

[0020] In an alternative embodiment, the first detection module is specifically configured to: slice the biomedical signal image according to a preset duration to obtain a sub-image of the biomedical signal; segment the monitoring video according to the preset duration to obtain a monitoring video segment that belongs to the same time period as the sub-image of the biomedical signal; and input the sub-image of the biomedical signal and the monitoring video segment into an abnormal discharge detection model to obtain an abnormal discharge detection result of the subject under test.

[0021] In an alternative embodiment, the first detection module is specifically configured to: slice the biomedical signal image according to a preset duration to obtain a sub-image of the biomedical signal; when the sub-image of the biomedical signal indicates that the subject under test has abnormal discharge, extract a monitoring video segment that belongs to the same time period as the sub-image of the biomedical signal from the monitoring video; and input the sub-image of the biomedical signal and the monitoring video segment into an abnormal discharge detection model to obtain an abnormal discharge detection result of the subject under test.

[0022] In an alternative embodiment, the biomedical signal includes multi-channel biomedical signals, and the image generation module is configured to: obtain a biomedical signal image according to the signal curves of the multi-channel biomedical signals; or obtain multiple biomedical signal images according to the signal curves of the multi-channel biomedical signals.

[0023] In an alternative embodiment, the first detection module is configured to: input the biomedical signal image into a biomedical signal feature extraction network to obtain biomedical signal features; determine the motion features of the subject under test according to the monitoring video, where the motion features include at least one of body motion features, head motion features, eye motion features, mouth motion features, and nose motion features; and input the biomedical signal features and the motion features into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject under test.

[0024] In an alternative embodiment, the step of inputting the biomedical signal features and the motion features into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject under test includes: enhancing the features of the motion features to increase the feature dimension of the motion features to obtain target motion features; concatenating the biomedical signal features and the target motion features to obtain concatenated features; and inputting the concatenated features into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject under test.

[0025] In an alternative embodiment, when the motion feature includes a body motion feature, the body motion feature includes body motion and no body motion. The determination method of the body motion feature includes: for any monitoring video frame of the to-be-detected monitoring video, determining the head position of the subject in the monitoring video frame according to the head position detection model, and cropping the monitoring video frame according to the head position to obtain a body video frame corresponding to the monitoring video frame; for any one of the body video frames, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining the frame difference corresponding to the reference video frame according to the sum of the pixel value differences at the same position between the body video frame and the reference video frame; determining the target frame difference of the body video frame according to the maximum value among the frame differences corresponding to the reference video frames, and determining the body motion feature of the subject based on the target frame difference.

[0026] In an alternative embodiment, when the motion feature includes a head motion feature, the head motion feature includes head motion and no head motion. The determination method of the head motion feature includes: determining the head position of the subject in the video frame of the to-be-detected monitoring video according to the head position detection model; cropping the video frame of the to-be-detected monitoring video based on the head position to obtain a head monitoring video of the subject; inputting the head monitoring video into a pre-trained head motion detection model, and when the output result of the head motion detection model indicates head motion, determining that the head motion feature is head motion, otherwise, determining that the head motion feature is no head motion; wherein, the training label of the head motion detection model is determined according to whether the head of the object in the training video moves and whether there are facial expression changes.

[0027] In an alternative embodiment, when the motion features include mouth motion features and / or eye motion features, the mouth motion features include mouth motion and non-mouth motion, the eye motion features include left-eye motion features and right-eye motion features, the left-eye motion features include left-eye motion and non-left-eye motion, the right-eye motion features include right-eye motion and non-right-eye motion, and the determination methods for the mouth motion features and the eye motion features are as follows: Determine the head position of the subject in the video frame of the to-be-detected surveillance video according to the head position detection model; Detect the facial key points of the subject based on the head position to obtain the mouth key points, left-eye key points, and right-eye key points of the subject; Determine the first aspect ratio of the mouth according to the target mouth key points among the mouth key points; Determine the second aspect ratio of the left eye according to the target left-eye key points among the left-eye key points, and determine the third aspect ratio of the right eye according to the target right-eye key points among the right-eye key points; Determine the mouth motion features according to the first change value of the first aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; Determine the left-eye motion features according to the second change value of the second aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; Determine the right-eye motion features according to the third change value of the third aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video.

[0028] In an alternative embodiment, when the motion features include nose motion features, the nose motion features include nose motion and non-nose motion, and the determination method for the nose motion features is as follows: Determine the head position of the subject in the video frame of the to-be-detected surveillance video according to the head position detection model; Detect the facial key points of the subject based on the head position to obtain the nose key point position of the subject; Determine the nose motion features according to the change situation of the nose key point positions indicated by the nose key point position sequence in the video frame of the to-be-detected surveillance video.

[0029] According to a fourth aspect of the present disclosure, there is provided a training device for an abnormal discharge detection model, including: a training sample acquisition module configured to acquire training samples and sample labels corresponding to the training samples, where the training samples include biomedical signal samples of a subject under test and monitoring video samples, and the monitoring video samples are obtained by photographing the subject under test when collecting the biomedical signal samples of the subject under test; an image sample generation module configured to obtain biomedical signal image samples according to the signal curves of the biomedical signal samples; a second detection module configured to input the biomedical signal image samples and the monitoring video samples into an abnormal discharge detection model to be trained to obtain abnormal discharge detection results of the training samples; a loss determination module configured to determine a loss value of the abnormal discharge detection model to be trained according to the difference between the abnormal discharge detection results of the training samples and the sample labels; and an iterative training module configured to perform iterative training on the abnormal discharge detection model to be trained according to the loss value until a preset condition is satisfied to obtain a pre-trained abnormal discharge detection model.

[0030] In an optional implementation manner, the second detection module may be specifically configured to: determine sample motion features corresponding to the monitoring video samples, where the sample motion features include at least one of sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, and sample nose motion features; input the biomedical signal image samples into a biomedical signal feature extraction network to obtain a first output; input the sample motion features into a motion feature enhancement network to obtain a second output; splice the first output and the second output to obtain a target splicing result; input the target splicing result into the abnormal discharge detection model to be trained, and obtain the abnormal discharge detection results of the training samples according to the output of the abnormal discharge detection model to be trained.

[0031] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements the abnormal discharge detection method described in the first aspect above, and / or the training method of the abnormal discharge detection model described in the second aspect above.

[0032] According to a sixth aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the abnormal discharge detection method described in the first aspect above, and / or the training method of the abnormal discharge detection model described in the second aspect above by executing the executable instructions.

[0033] According to a seventh aspect of the embodiments of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements the abnormal discharge detection method described in the first aspect above, and / or the training method of the abnormal discharge detection model described in the second aspect above.

[0034] According to the abnormal discharge detection method, abnormal discharge detection device, training method of abnormal discharge detection model, training device of abnormal discharge detection model, computer-readable storage medium and electronic device of the embodiments of the present disclosure, a biomedical signal image is obtained from the signal curve of the biomedical signal, and then the biomedical signal image and the monitoring video are jointly input into the abnormal discharge detection model to automatically obtain the abnormal discharge detection result. On the one hand, the abnormal discharge detection result can be obtained without manual reading of the image, improving the efficiency of abnormal discharge recognition; on the other hand, the training label of the abnormal discharge detection model of the present disclosure is marked by manually checking the biomedical signal image and the corresponding monitoring video. Therefore, using the biomedical signal image as the input of the abnormal discharge detection model can make the input data format of the model consistent with the data format during training label marking, so that the model can better simulate the scenario of a doctor reading the image to judge abnormal discharge and improve the accuracy of abnormal discharge detection. On the other hand, since the abnormal discharge detection result determined by the method of the present disclosure is relatively accurate, it can provide high reference value for doctors, assist doctors in quickly and accurately performing clinical diagnosis, save the workload of doctors, and is convenient for wide application in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the drawings, several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0036] Figure 1 A flowchart showing a method for training an abnormal discharge detection model in an exemplary embodiment of the present disclosure;

[0037] Figure 2 A flowchart showing a method for generating training samples and sample labels of the training samples in an exemplary embodiment of the present disclosure;

[0038] Figure 3A A schematic diagram showing a biomedical signal curve image in an exemplary embodiment of the present disclosure;

[0039] Figure 3B A schematic diagram showing another biomedical signal curve image in an exemplary embodiment of the present disclosure;

[0040] Figure 3Schematic flowchart of a method for processing surveillance video samples in an exemplary embodiment of the present disclosure;

[0041] Figure 4 Schematic flowchart of a method for determining the body movement characteristics of a sample in an exemplary embodiment of the present disclosure;

[0042] Figure 5 Schematic flowchart of a method for determining the head movement characteristics of a sample in an exemplary embodiment of the present disclosure;

[0043] Figure 6 Schematic flowchart of a method for determining the mouth movement characteristics of a sample in an exemplary embodiment of the present disclosure;

[0044] Figure 7 Schematic flowchart of a method for determining the eye movement characteristics of a sample in an exemplary embodiment of the present disclosure;

[0045] Figure 8 Schematic flowchart of a method for determining the nose movement characteristics of a sample in an exemplary embodiment of the present disclosure;

[0046] Figure 9 Schematic flowchart of a method for obtaining the abnormal discharge detection result of a training sample in an exemplary embodiment of the present disclosure;

[0047] Figure 10 Schematic flowchart of another method for training an abnormal discharge detection model in an exemplary embodiment of the present disclosure;

[0048] Figure 11 Schematic flowchart of a method for abnormal discharge detection in an exemplary embodiment of the present disclosure;

[0049] Figure 12 Schematic flowchart of a method for obtaining the abnormal discharge detection result in an exemplary embodiment of the present disclosure;

[0050] Figure 13 Schematic flowchart of another method for obtaining the abnormal discharge detection result in an exemplary embodiment of the present disclosure;

[0051] Figure 14 Schematic flowchart of yet another method for obtaining the abnormal discharge detection result in an exemplary embodiment of the present disclosure;

[0052] Figure 15 Structure diagram of an abnormal discharge detection system in an exemplary embodiment of the present disclosure;

[0053] Figure 16 Schematic diagram of an exemplary application scenario of an abnormal discharge detection method in an embodiment of the present disclosure;

[0054] Figure 17 A schematic diagram showing an exemplary application scenario of another abnormal discharge detection method in an embodiment of the present disclosure;

[0055] Figure 18 A schematic diagram showing the composition of an abnormal discharge device in an exemplary embodiment of the present disclosure;

[0056] Figure 19 A schematic diagram showing the composition of a training device for an abnormal discharge detection model according to an embodiment of the present disclosure;

[0057] Figure 20 A structural diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0058] In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Embodiments

[0059] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and then implement the present disclosure, rather than to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to be able to fully convey the scope of the present disclosure to those skilled in the art.

[0060] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, a device, an apparatus, a method, or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms: completely hardware, completely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0061] According to an embodiment of the present disclosure, there is provided an abnormal discharge detection method, an abnormal discharge detection device, a training method for an abnormal discharge detection model, a training device for an abnormal discharge detection model, a computer-readable storage medium, and an electronic device.

[0062] In this article, the number of any element in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0063] The principles and spirit of the present disclosure will be elaborated in detail below with reference to several representative embodiments of the present disclosure. Summary of the Invention

[0065] The inventor of the present disclosure has found that the existing method of manually studying electroencephalograms to determine whether there is abnormal discharge in the brain is inefficient, and the accuracy depends on the experience of doctors. For doctors with insufficient experience, misjudgment may occur.

[0066] In view of the above content, the basic idea of the present disclosure is to provide an abnormal discharge detection method, an abnormal discharge detection device, a training method for an abnormal discharge detection model, a training device for an abnormal discharge detection model, a computer-readable storage medium and an electronic device, wherein a biomedical signal image is obtained through a signal curve of a biomedical signal, and then the biomedical signal image and a monitoring video are input into the abnormal discharge detection model to automatically obtain an abnormal discharge detection result. On the one hand, the abnormal discharge detection result can be obtained without manual reading of the image, thereby improving the efficiency of abnormal discharge recognition; on the other hand, the training label of the abnormal discharge detection model of the present disclosure is annotated by manually referring to the biomedical signal image and the corresponding monitoring video, so that the biomedical signal image is used as the input of the abnormal discharge detection model, so that the input data format of the model and the data format when the training label is annotated can be consistent, so that the model can better simulate the scene of doctors reading the image to judge abnormal discharge, thereby improving the accuracy of abnormal discharge detection. On the other hand, since the abnormal discharge detection result determined by the method of the present disclosure is highly accurate, it can provide a high reference value for doctors, assist doctors in making clinical diagnoses quickly and accurately, save doctors' workload, and facilitate wide application in clinical practice.

[0067] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0068] Overview of Application Scenarios

[0069] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0070] The embodiments of the present disclosure can be applied to the abnormal discharge detection scenario of the human brain. For example, when the patient feels uncomfortable, such as having a headache, the brain electrical signal of the patient can be collected and the patient can be video-shot while collecting the brain electrical signal of the patient to obtain the video monitoring data when the brain electrical signal is collected, and then the brain electrical signal curve is drawn according to the collected brain electrical signal data to obtain the brain electrical signal image, and then the brain electrical signal image and the monitoring video are input into the abnormal discharge detection model to obtain the abnormal discharge detection result of the patient. Then, the doctor can make a quick and accurate clinical diagnosis of the patient based on the abnormal discharge detection result combined with other key indicator data, reduce the doctor's workload, and improve the doctor's work efficiency.

[0071] Exemplary Method

[0072] The current method of manually studying electroencephalograms to determine whether there is abnormal electrical discharge in the brain is inefficient, and its accuracy depends on the personal experience of doctors. For doctors with insufficient experience, there is a possibility of misjudgment.

[0073] The exemplary embodiments of the present disclosure first provide a training method for an abnormal electrical discharge detection model to at least overcome all or part of the above-mentioned defects existing in the related art.

[0074] Figure 1 The flowchart of the training method for the abnormal electrical discharge detection model in the embodiments of the present disclosure is shown. Refer to Figure 1 , according to an embodiment of the present disclosure, the training method for the abnormal electrical discharge detection model includes the following steps:

[0075] Step S110, obtaining a training sample and a sample label corresponding to the training sample;

[0076] Step S120, obtaining a biomedical signal image sample according to the signal curve of the biomedical signal sample;

[0077] Step S130, inputting the biomedical signal image sample and the monitoring video sample into the abnormal electrical discharge detection model to be trained to obtain the abnormal electrical discharge detection result of the training sample;

[0078] Step S140, determining the loss value of the abnormal electrical discharge detection model to be trained according to the difference between the abnormal electrical discharge detection result of the training sample and the sample label;

[0079] Step S150, performing iterative training on the abnormal electrical discharge detection model to be trained according to the loss value until a preset condition is met, and obtaining a pre-trained abnormal electrical discharge detection model.

[0080] Next, Figure 1 the specific implementation of step S110 in

[0081] In an optional embodiment, the training sample in step S110 includes a biomedical signal sample and a monitoring video sample of the subject to be tested.

[0082] In an optional embodiment, the biomedical signal sample is a biomedical signal collected from the bodies of a large number of different subjects. The biomedical signal is an active signal spontaneously generated by the physiological process of a living body, and can relatively accurately reflect the physiological state or physical sign state of a person. Therefore, whether there is an abnormality in the living body can be determined through this biomedical signal. The living body in the present disclosure may include any living body, such as a human body, an animal body, etc., and the present exemplary embodiment does not make special limitations on this.

[0083] Exemplarily, the above-mentioned biomedical signals may include any one or more of the following: physiological signals such as electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), electrooculogram (EOG), and electrogastrogram (EGG), or may also include non-electrophysiological signals such as body temperature, blood pressure, pulse, and respiration. It can be set according to the actual situation, and the present disclosure does not make special limitations thereon.

[0084] Among them, the electrocardiogram (ECG) refers to a series of very coordinated electrical stimulation pulses generated inside the heart, which enable the cardiac muscle cells to relax and contract rhythmically. These signals are transmitted to different parts of the human body surface to form different potential differences. These weak potential difference signals can be detected from the human body surface through a multi-channel electrocardiograph device, thereby obtaining the electrocardiogram signal.

[0085] The electroencephalogram (EEG) is the overall reflection of the electrophysiological activities of brain nerve cells on the cerebral cortex or the scalp surface. It can be understood as the change activity data of the electrical signals during brain activity. The electroencephalogram signal can be collected by amplifying the spontaneous bioelectric potential of the cerebral cortex from the scalp through a precise instrument. The electroencephalogram signal is the spontaneous and rhythmic electrical activity of a group of brain cells recorded by electrodes.

[0086] The electromyogram (EMG) is the superposition of the action potentials of motor units in numerous muscle fibers in time and space, and can be obtained by pasting electromyography sensors on the skin.

[0087] The electrooculogram (EOG) is a bioelectric signal caused by the potential difference between the cornea and the retina of the eye, and is extremely convenient to collect, and can be completed through a small number of electrodes.

[0088] The electrogastrogram (EGG) is the electrical signal generated by the contraction of the stomach muscles, and can be collected using electrodes on the abdominal skin surface of the human body.

[0089] In an alternative embodiment, the monitored video sample is obtained by photographing the subject when collecting the biological signal sample of the subject. For example, when collecting biomedical signals from the bodies of a large number of different subjects, for each subject being collected, the subject can be simultaneously video-recorded to obtain the video monitoring data of the subject when collecting the biomedical signal of the subject being collected.

[0090] In an alternative embodiment, for the same subject to be sampled, the acquisition time periods of the biomedical signal samples and the monitoring video samples in the training samples are the same. That is to say, the training samples include a large number of biomedical signals and monitoring videos collected from the same subject to be sampled during the same time period.

[0091] For example, for each subject to be sampled, long-time biomedical signals and monitoring videos can be collected. For instance, 30 minutes of biomedical signals and monitoring videos are collected. Then, the biomedical signals and monitoring videos are segmented according to the same start time and the same preset duration. If the preset duration is 1 minute, the collected 30-minute biomedical signals and monitoring videos are divided into 30 pairs of biomedical signal and monitoring video samples of the same time period with a duration of 1 minute each. These pairs of biomedical signal and monitoring video samples are used as training samples. That is to say, each training sample is a pair of biomedical signal and monitoring video of the same subject to be sampled during the same time period.

[0092] In an alternative embodiment, the sample labels corresponding to the training samples may include whether there is abnormal discharge in the subject to be sampled in the training samples. That is to say, for example, 1 is used to indicate the presence of abnormal discharge, and 0 indicates the absence of abnormal discharge. If it is determined that the subject to be sampled has abnormal discharge based on the pair of biomedical signal samples and monitoring video samples of the same time period, the sample labels of the biomedical signal samples and monitoring video samples of this time period are 1. If it is determined that the subject to be sampled does not have abnormal discharge based on the pair of biomedical signal samples and monitoring video samples of the same time period, the sample labels of the biomedical signal samples and monitoring video samples of this time period are 0.

[0093] For example, the sample labels can be added to the training samples by manual annotation. For example, doctors can watch the monitoring videos and refer to the biomedical signal curve images of the same time period as the monitoring videos to determine whether there is actually abnormal discharge in the biomedical signals, thereby generating the labels of the training samples.

[0094] Next, with reference to Figure 2 , taking brain abnormality detection as an example, a schematic flowchart of a method for generating training samples and labels of the training samples in an exemplary embodiment of the present disclosure is shown. As Figure 2As shown, the method may include steps S210 to S240. Among them: in step S210, an electroencephalograph collects multi-channel electroencephalogram signals of the object to be collected, obtains multi-channel electroencephalogram signals, and produces an electroencephalogram signal image according to the multi-channel electroencephalogram signals; in step S220, a camera collects the facial and torso videos of the object to be collected to obtain a surveillance video; in step S230, the multi-channel electroencephalogram signal image is sliced and the surveillance video is segmented to obtain multi-channel electroencephalogram signal sub-image samples and surveillance video samples in the same time period; in step S240, it is manually marked whether there is abnormal discharge in each multi-channel electroencephalogram signal sub-image sample to obtain a training sample label.

[0095] For example, after obtaining the multi-channel electroencephalogram signal samples and surveillance video samples in the same time period, a doctor can watch the surveillance video samples to judge whether there is really abnormal discharge in the multi-channel electroencephalogram signal samples in the same time period. For example, after watching the surveillance video samples, if the doctor believes that the multi-channel electroencephalogram signal samples are autonomously abnormally discharged without external interference at time T1, then the above-mentioned mark 1 can be added at time T1, that is, it indicates that there is abnormal discharge at time T1. For the multi-channel electroencephalogram signal samples and surveillance video samples in the same time period, as long as there is any moment when the multi-channel biomedical signal samples in this time period are added with mark 1, the sample label of the training sample composed of the multi-channel electroencephalogram signal samples and surveillance video samples in the same time period is that there is abnormal discharge. On the contrary, if there is no moment marked as 1 in the channel biomedical signal samples in this time period, the sample label of the training sample composed of the multi-channel electroencephalogram signal samples and surveillance video samples in the same time period is that there is no abnormal discharge.

[0096] Next, the specific implementation manner of obtaining the biomedical signal image sample according to the signal curve of the biomedical signal sample in step S120 will be described.

[0097] For example, what the biomedical signal acquisition device directly collects is a biomedical signal digital sequence. For example, a multi-channel electrocardiograph, a multi-channel electroencephalograph, and a multi-channel electromyograph can be used to simultaneously collect the corresponding biomedical electrical signals at a preset acquisition frequency, such as 500HZ, and what is obtained is a biomedical signal numerical sequence.

[0098] Taking the electroencephalogram signal as an example, the electroencephalogram signal collected by the electroencephalograph can be the electroencephalogram signal numerical sequence shown in the following formula (1).

[0099] x t =(x t1 ,x t2 ,x t3 ,x t4 ,…,x tn ) (1)

[0100] In formula (1), x t is a numerical sequence of EEG signals. x t1 , x t2 , x t3 , x t4 , …, x tn represent the signal values of the EEG signal at different sampling times. For example, x t1 represents the signal value of the EEG signal x t at the first sampling time, and x tn represents the signal value of the EEG signal x t at the nth sampling time.

[0101] In the present disclosure, the numerical sequence of biomedical signals collected by a biomedical signal device is converted into a biomedical signal image, such as an EEG signal image, an ECG signal image, etc., and then abnormal discharge detection is performed based on the biomedical signal image. In other words, in the present disclosure, a biomedical signal image can be generated according to the signal curve of the collected biomedical digital signal, and an abnormal discharge detection model is trained based on the biomedical signal image.

[0102] In an alternative embodiment, the sampling time of the biomedical digital signal can be used as the abscissa, and the sampling value of the biomedical digital signal can be used as the ordinate to plot a biomedical signal curve, and the plotted biomedical signal curve is used as the biomedical signal image. Then, in subsequent steps, the biomedical signal image is input into a biomedical signal feature extraction model to obtain biomedical signal features, and then the extracted biomedical signal features are combined with the motion features obtained from the monitoring video and input into the abnormal discharge detection model to be trained for training the abnormal discharge detection model of the brain.

[0103] Taking FP1 (left frontal pole), FP2 (right frontal pole), F3 (left frontal), F4 (right frontal), C3 (left central), and C4 (right central) in the 10-20 standard electrodes in EEG signal acquisition as an example, the EEG signals collected by an electroencephalograph are stored in the form of digital signals as shown in Table 1 below.

[0104] Table 1 EEG digital signals

[0105] FP1 (Left Frontopolar) 52 55 42 41 2 … FP2 (Right Frontopolar) 32 14 42 12 24 … F3 (Left Frontal) 6 41 23 24 87 … F4 (Right Frontal) 56 2 42 12 2 … C3 (Left Central) 44 55 42 21 86 … C4 (Right Central) 33 53 34 2 51 …

[0106] In other words, taking the electroencephalogram (EEG) signal as an example, the original EEG signal collected by an EEG machine is a digital signal. However, when a doctor makes a diagnosis, what the doctor reads is not the EEG digital signal, but the EEG waveform, such as whether the peak of the EEG waveform is sharp enough and whether the contrast with the background is obvious. That is, the EEG signal curve image is more intuitive for doctors. As mentioned above, when generating the labels of training samples, it is also determined whether the training sample is abnormal discharge by a doctor watching the monitoring video and the EEG signal curve image. Therefore, using the EEG signal curve image as the input of the model can make the format of the input data of the model consistent with the data format used when adding the training labels, so as to more accurately simulate the doctor's judgment method for abnormal discharge detection, thereby improving the accuracy of abnormal discharge detection.

[0107] In an alternative embodiment, the biomedical signals in the present disclosure include multi-channel biomedical signals. A specific implementation of step S120 may include: obtaining a biomedical signal image according to the signal curve of the multi-channel biomedical signal; or obtaining a plurality of biomedical signal images according to the signal curve of the multi-channel biomedical signal.

[0108] For example, when the biomedical signal is a biomedical signal image, the multi-channel biomedical signal curves can be drawn on the same image. As Figure 3A shown, the 6 biomedical signal curves in Table 1 are drawn on the same graph. In this way, the subsequent biomedical signal feature extraction model can use a 2D (two-dimensional) convolutional neural network as the backbone network, such as efficientnet-v2, to extract the features of the biomedical signal.

[0109] When the biomedical signal is a biomedical signal image, the biomedical signal curves of each channel can also be drawn separately to obtain a plurality of biomedical signal images corresponding to the multi-channel biomedical signal. As Figure 3B shown, 6 biomedical signal images can be obtained from the 6 biomedical signal curves in Table 1. In this way, during the subsequent processing, the biomedical signal curve images corresponding to the biomedical signals of different channels can be used as the inputs of different channels of the biomedical signal feature extraction model. At this time, the biomedical signal feature extraction model can use a 3D (three-dimensional) convolutional neural network as the backbone network to extract the features of the biomedical signal.

[0110] Next, the specific implementation of step S130 in Figure 1 will be described in detail.

[0111] Exemplarily, a specific implementation of step S130 may include: inputting a biomedical signal image sample into a biomedical signal feature extraction network to obtain biomedical signal features; determining the sample motion features corresponding to the monitoring video sample, where the sample motion features include at least one of sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, and sample nose motion features; and inputting the biomedical signal features and the sample motion features into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample.

[0112] In an exemplary implementation, the biomedical signal digital sample can be preprocessed, STFT (Scale-invariant feature transform) transformed, and feature extracted to obtain the processed biomedical signal digital sample. Based on the biomedical signal curve corresponding to the processed biomedical signal digital sample, a biomedical signal image sample for model training can be obtained.

[0113] For example, in the preprocessing, a 50HZ notch filter can be used to filter the collected electrical signal to reduce the contamination of alternating current. A band-pass filter with a frequency range of 0.1 - 70Hz is applied for filtering on 21 electroencephalogram electrodes and 2 earlobe reference electrodes, and signal space projection (SSP) is used to remove the cardiac field artifacts in the electroencephalogram signal.

[0114] The electrical signal sampled from the electrode has temporal correlation, and the Fourier transform can help display the signal in the frequency domain. The short-time Fourier transform is good at processing short signal segments. Therefore, STFT can be used to transform the EEG data into a two-dimensional matrix composed of frequency and time information. After the STFT transformation, a total of 77 matrices are obtained from the EEG signal transformation. Convolutional Neural Networks (CNN) can be used to extract features from these matrices to obtain low-dimensional features convenient for the abnormal discharge detection model to use.

[0115] Exemplarily, EfficientNetV2 (a deep learning neural network) can be used as a biomedical signal feature extraction network for extracting features of biomedical signals. EfficientNetV2 has superior feature extraction capabilities, is easy to train, and has low latency. To achieve a balance between feature extraction capabilities and the time consumed, EfficientNetV2-S can be used. EfficientNetV2-S consists of fused MBConv (Mobile Inverted Bottleneck Convolution) blocks and MBConv blocks. The fused MBConv blocks have a large number of parameters. Many lightweight networks use MBConv to replace the fused MBConv blocks and achieve good performance. However, EfficientNetV2-S proves that the MBConv blocks may have an adverse impact on performance, and the hybrid of the two types of networks can balance performance, network size, and latency. After feature extraction by EfficientNetV2-S, a 1024-dimensional feature vector can be obtained, which contains the features of the biomedical signal and will be combined with the motion feature vector corresponding to the surveillance video sample for abnormal discharge detection.

[0116] Of course, other deep learning models can also be used for feature extraction of biomedical signals. Such as using VGG (Visual Geometry Group)-16, ResNet18 (Residual Networks), etc. This exemplary embodiment does not make special limitations on this. However, after experimental evaluation and comparison, it is found that for the single-modal model using EfficientNetV2-S, that is, only using biomedical signals for abnormal discharge detection based on EfficientNetV2-S, the value of the AUPRC (Area Under the Precision-Recall Curve) evaluation index is 0.8316, which greatly exceeds the performance of ResNet18 and VGG-16. For the multi-modal model that combines biomedical signal features and motion features obtained from the video, the AUPRC for feature extraction of biomedical signals using the EfficientNetV2-S model is even higher, which is 0.8623. In addition, when the sensitivity is set to 0.8, the accuracy of EfficientNetV2-S is 76.6%, which is better than the accuracies of ResNet18 (69.2%) and VGG-16 (65.4%). Therefore, it is more recommended to use EfficientNetV2-S for feature extraction of biomedical signals for abnormal discharge detection.

[0117] Exemplarily, taking electroencephalogram (EEG) signals as an example, for the automatic recognition of abnormal EEG signals, in addition to the need for a large number of training samples to learn the inherent characteristics of EEG signals, the interference of artifacts is also an important reason. The abnormal detection of EEG signals needs to be carried out in the state of being awake with eyes closed or in a state of consciousness disorder. Since EEG signals are weak, they will inevitably be affected by artifacts such as electromyogram (EMG), electrocardiogram (ECG), and movements.

[0118] In an alternative embodiment, artifacts such as EMG and ECG can be judged by collecting ECG signals and EMG signals.

[0119] In an alternative embodiment, movement artifacts can be identified by recording videos. In this way, the influence of movement artifacts on the detection of abnormal discharges can be excluded, and the accuracy of automatic detection and recognition of abnormal discharges can be improved, so as to better apply it to clinical practice.

[0120] Exemplarily, as mentioned above, the detection of abnormal EEG signals needs to be carried out in the state of being awake with eyes closed or in a state of consciousness disorder. Therefore, in addition to large-scale limb movements affecting the detection results, small-scale subtle movements will also have an impact on the results. That is to say, in the identification of movement artifacts, in addition to large-scale limb movements that need to be identified, small-scale limb movements, such as blinking, small-scale chewing, small-scale speaking, etc., which are difficult to be observed by the human eye, also need to be identified.

[0121] Based on this, in an alternative embodiment, the sample motion features corresponding to the monitored video samples include at least one of sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, and sample nose motion features.

[0122] For example, it can be determined whether the body, head, eyes, mouth, nose, etc. of the object being sampled are moving when collecting biomedical signals in the same time period as the monitored video sample according to the monitored video sample, so as to obtain the motion features of the object being sampled. Then, the motion features of the object being sampled and the biomedical signal features are combined to jointly determine whether there is abnormal discharge in the object being sampled. In this way, the influence of movement artifacts on the detection of abnormal discharges can be avoided, and the accuracy of abnormal discharge detection can be improved.

[0123] Exemplarily, for each monitored video sample obtained by segmentation, it can be processed according to the Figure 3 shown framework to determine the body parts and head of the object being sampled in the video frames of the monitored video sample. Thus, the motion features of the object being sampled in the time period corresponding to the monitored video sample can be determined according to the positions of the body parts and head of the object being sampled in the video frames of the monitored video sample. Refer to Figure 3, video frames can be collected from the surveillance video sample according to a preset frame rate to obtain a surveillance video frame sequence sample. The surveillance video frame sequence sample is input into a patient detection model to obtain the position of the patient in each video frame of the surveillance video frame sequence sample. According to the position of the patient in each video frame of the surveillance video frame sequence sample, the video frames in the surveillance video frame sequence sample are cropped to obtain a patient video frame sequence sample. The patient video frame sequence sample is input into a face detection model to obtain the position of the patient's head in each video frame of the patient video frame sequence sample. According to the position of the patient's head in each video frame of the patient video frame sequence sample, each video frame in the patient video frame sequence sample is cropped to obtain a patient head video frame sequence sample and a patient body video frame sequence sample. According to the face key point detection method, the positions of the patient's eyes, mouth, and nose are determined in each video frame of the patient head video frame sequence sample.

[0124] In an alternative embodiment, the patient detection model may include any machine learning model capable of target detection, such as the YOLOv5 (You Only Look Once) target detection model or any other deep learning model. The patient head detection model may also include any face detection model, such as the YOLOv5 Face face detection model, etc.

[0125] For example, the surveillance video sample can be collected first based on a preset frame rate, such as a rate of 15 frames per second, to obtain a surveillance video frame sequence sample. Then, the patient is detected in each video frame of the surveillance video frame sequence sample based on the YOLOv5 model, and the video frames in the surveillance video frame sequence sample are cropped according to the detected patient position to obtain a patient video frame sequence sample. The patient face is detected in each video frame of the patient video frame sequence sample based on the YOLOv5 Face face detection model, and the patient in the patient video frame sequence sample is cropped according to the detected face position to obtain a patient head video frame sequence sample and a patient body video frame sequence sample. In this way, the body movement characteristics of the patient can be determined based on the patient body video frame sequence sample, and the head movement characteristics, eye movement characteristics, mouth movement characteristics, and nose movement characteristics of the patient can be determined based on the patient head video frame sequence sample.

[0126] Next, in combination with Figures 4 to 8 , the specific implementation methods for determining the sample body movement characteristics, head movement characteristics, eye movement characteristics, mouth movement characteristics, and nose movement characteristics will be further described.

[0127] In an alternative embodiment, the body movement characteristics include body movement and no body movement. Exemplarily, Figure 4The flowchart shows a method for determining the body movement characteristics of a sample in an exemplary embodiment of the present disclosure. Refer to Figure 4 , the method may include steps S410 to S440. Among them:

[0128] In step S410, for any monitoring video frame of the monitoring video sample, the head position of the object to be captured in the monitoring video frame is determined according to the head position detection model, and the monitoring video frame is cropped according to the head position to obtain the body video frame corresponding to the monitoring video frame.

[0129] In an alternative embodiment, the head position detection model may include the above-mentioned face detection model, and the face position detected by the face detection model can be understood as the head position.

[0130] For example, as described above, the monitoring video sample can be collected according to a preset frame rate, such as 15 frames per second, to obtain the monitoring video frames. For each monitoring video frame, the position of the object to be captured in the monitoring video frame can be detected first according to the object detection model, and then the area outside the object to be captured in the monitoring video frame can be cropped to obtain the video frame of the object to be captured. In this way, the interference of other background areas unrelated to the object to be captured in the monitoring video frame on the movement detection of the object to be captured can be avoided, and the accuracy and efficiency of the determination of the movement characteristics can be improved.

[0131] The object detection model usually represents the position of the detected object in the image frame with a rectangular box. Therefore, the monitoring video frame can be cropped according to the boundary of the rectangular box of the detected position of the object to be captured, and the area inside the rectangular box is retained in the monitoring video frame, and the area outside the rectangular box is cropped to obtain the video frame of the object to be captured. After obtaining the video frame of the object to be captured, the video frame of the object to be captured can be input into the face detection model, and the face position is detected in the video frame of the object to be captured through the face detection model, so as to obtain the head position of the object to be captured. Usually, the detected face position of the object to be captured can also be framed in the video frame of the object to be captured with a rectangular box, and then the video frame of the object to be captured is segmented according to the rectangular box to obtain the head video frame of the object to be captured and the body video frame of the object to be captured. That is, the head video frame only includes the head of the object to be captured, and the body video frame includes the torso part and limbs except the head of the object to be captured.

[0132] Next, in step S420, for any body video frame, the reference video frame of the body video frame is determined according to N other body video frames adjacent to the body video frame.

[0133] In an alternative embodiment, N can be an integer greater than or equal to 1. The value of N can be customized according to experience or user requirements. For example, the value of N can be 3, 5, etc. However, the value of N should not be set too large or too small. Setting the value of N too large will increase the subsequent computational amount and reduce the processing efficiency. Setting the value of N too small may result in inaccurate results for the determined body movement characteristics. Therefore, when setting N, a balance needs to be struck between the computational amount and accuracy.

[0134] Taking the value of N as 3 as an example, for any body video frame, its three adjacent video frames before and after can be used as the reference video frames of this body video frame. That is, for the 4th frame and subsequent body video frames, it has a certain number of reference video frames. For the 3rd frame, there are 5 reference video frames, for the 2nd frame, there are 4 reference video frames, and for the 1st frame, there are only 3 consecutive video frames after it that are adjacent to it, namely the 2nd frame, the 3rd frame, and the 4th frame, which are the reference video frames of the 1st frame.

[0135] In step S430, according to the sum of the pixel value differences at the same position between the body video frame and the reference video frames, the frame difference corresponding to the reference video frame is determined.

[0136] For example, for each reference video frame, the frame difference corresponding to the reference video frame can be determined according to the sum of the pixel value differences at the same pixel coordinate. For example, if (x, y) represents the position of a pixel point, f n (x, y) represents the pixel value at the (x, y) position of the nth frame. Taking the nth frame as the current body video frame and the (n + 1)th frame as a reference frame of the nth frame's body video frame as an example, the difference between the pixel values at the position with pixel coordinates (x, y) in the nth frame and the position with pixel coordinates (x, y) in the (n + 1)th frame can be expressed as D n (x, y) = |f n (x, y) - f n+1 (x, y)|. Taking the body video frame having N pixel points as an example, the sum of the pixel value differences at the same position between the reference video frame (n + 1)th frame and the body video frame nth frame can be expressed as D nN (x nN , y nN ) = |f n (x1, y1) - f n+1 (x1, y1)| + |f n (x2, y 21 ) - f n+1 (x2, y2)| + … + |f n (x N , y N ) - f n+1 (x N , yN )|.

[0137] In step S440, according to the maximum value in the frame differences corresponding to the reference video frames, the target frame difference of the body video frames is determined.

[0138] For example, the frame differences corresponding to each reference video frame can be sorted in descending order or ascending order to determine the maximum value in the frame differences corresponding to the reference video frames, and the maximum frame difference is determined as the target frame difference.

[0139] In step S440, according to the target frame difference, the sample body motion features are determined.

[0140] Optionally, an exemplary implementation of step S440 may be: when the target frame difference is greater than a first preset value, it can be determined that the sample body motion feature is body motion; otherwise, it is determined that the sample body motion feature is no body motion.

[0141] In an alternative implementation, when, based on the target frame difference corresponding to any body video frame, it is determined that the sample body motion feature is body motion, then it is determined that the sample body motion feature corresponding to this surveillance video sample is body motion feature; in the case where the sample body motion features determined by the target frame differences corresponding to all body video frames in the surveillance video sample are all no body motion, then it is determined that the sample body motion feature corresponding to this surveillance video sample is no body motion. That is, for a surveillance video sample, as long as it is determined that the sample body motion feature corresponding to any body video frame of this surveillance video sample is body motion, then the sample body motion feature of this surveillance video sample is body motion; otherwise, it is determined that the sample body motion feature of this surveillance video sample is no body motion.

[0142] In another alternative implementation, in the case where it is determined that the sample body motion features are all body motion based on M consecutive surveillance video frames in the surveillance video sample, it can be determined that the sample body motion feature corresponding to this surveillance video sample is body motion; otherwise, it is determined that the sample body motion feature corresponding to this surveillance video sample is no body motion. Among them, M can be custom-determined according to requirements, such as M being 4, M being 5, etc. In this way, the accuracy of determining the body motion features can be improved.

[0143] Through the above steps S410 to S440, by means of the pixel difference between the current frame and the reference frame, subtle movements of the body can be detected, improving the accuracy of body movement feature detection. For example, when the limbs of the subject being captured are covered by a quilt, the limb key points of the human body cannot be detected by the human key point detection method, and thus it is impossible to determine whether the subject being captured is moving based on the coordinate position changes of the human key points. However, even if covered by a quilt, as long as the limbs move, the images of adjacent frames will change, and then the pixel values at the same position will also change. Therefore, it is still possible to detect whether the limbs of the subject being captured are moving through the target frame difference. That is to say, the method for determining body movement features in the present disclosure can detect subtle body movements to avoid the influence of subtle movements on the abnormal discharge detection structure and assist in improving the accuracy of abnormal discharge detection.

[0144] In an alternative embodiment, the above-mentioned head movement features include head movement and no head movement. Exemplarily, Figure 5 The flowchart showing a method for determining head movement features in an exemplary embodiment of the present disclosure is shown. Refer to Figure 5 This method may include steps S510 to S530.

[0145] In step S510, according to the head position detection model, the head position of the subject being captured in the surveillance video frame of the surveillance video sample is determined.

[0146] Exemplarily, the specific implementation manner of step S510 may refer to the above-mentioned step S410. In the above-mentioned step S410, it has been described in detail how to determine the head position of the subject being captured in the surveillance video frame of the surveillance video sample, and thus will not be elaborated here.

[0147] In step S520, based on the head position, the surveillance video frame of the surveillance video sample is cropped to obtain the head surveillance video of the subject being captured.

[0148] Exemplarily, the specific implementation manner of step S520 may also refer to the relevant content in the above-mentioned step S410, that is, the surveillance video frame in the surveillance video sample is cropped according to the rectangular frame representing the head position to obtain the head surveillance video of the subject being captured.

[0149] In step S530, the head surveillance video is input into a pre-trained head movement detection model. When the output result of the head movement detection model indicates head movement, the sample head movement feature is determined to be head movement; otherwise, the sample head movement feature is determined to be no head movement.

[0150] In an alternative embodiment, head training samples can be collected first, and then the head training samples can be labeled to generate labels for the head training samples. Then, a supervised learning training can be performed on the head motion detection model to be trained according to the head training samples and the labels of the head training samples, so as to obtain a pre-trained head motion detection model.

[0151] For example, the head video frames corresponding to the surveillance video frames in the surveillance video sample can be determined according to the above step S410, and then, according to the head video frames corresponding to the surveillance video frames in the surveillance video sample, the head video sample corresponding to the surveillance video sample can be determined, and this head video sample can be used as the head training sample.

[0152] In an alternative embodiment, the training labels of the head motion detection model are determined according to whether the head of the object in the training video moves and whether there are facial expression changes.

[0153] For example, the labels of the head training samples include two types: one type is that the head of the object being captured does not move and the facial expression does not change during the time period corresponding to the head training sample. For example, in the case where there are no facial micro-movements such as blinking or opening the mouth, the training label of the head training sample is that the head does not move; the other type is that the head of the object being captured moves and / or the face of the object being captured changes during the time period corresponding to the head training sample. For example, in the case of facial expression changes or facial micro-movements such as blinking, the training label of the head training sample is that the head moves. That is to say, the head motion detection model is a binary classification model, which can convert the head motion detection task into a binary classification task of whether the head moves.

[0154] In an alternative embodiment, the head motion detection model can be any machine learning model capable of classification, such as a deep convolutional neural network model, etc. This exemplary embodiment does not make special limitations on this.

[0155] In an alternative embodiment, after the head motion detection model is trained, the head video frame sequence corresponding to the head training sample can be input into the pre-trained head motion detection model, and the model can perform binary classification on the head video frame sequence to determine the classification result of the head video frame sequence. In the case where the classification result is that the head moves, it is determined that the sample head motion feature of this head training sample is that the head moves; in the case where the classification result is that the head does not move, it is determined that the sample head motion feature of this head training sample is that the head does not move.

[0156] In the present disclosure, during the training process, the head motion detection model not only learns the motion features of the head itself, such as large motion features like rotation, but also learns subtle motion features such as facial expression changes. Therefore, the head motion detection model in the present disclosure can automatically recognize both large motions and subtle movements of the head to determine whether there is motion in the head of the object being captured in the training samples during the monitoring video time period, improving the accuracy of head motion detection.

[0157] In an alternative embodiment, the mouth motion features include mouth motion and non-mouth motion. Exemplarily, Figure 6 A schematic flowchart showing a method for determining mouth motion features in an exemplary embodiment of the present disclosure is shown. Refer to Figure 6 , the method may include steps S610 to S640. Among them:

[0158] In step S610, according to the head position detection model, the head position of the object being captured in the monitoring video frame of the monitoring video sample is determined.

[0159] Exemplarily, the specific implementation of step S610 may refer to the relevant content in step S410 above, and will not be elaborated here.

[0160] In step S620, based on the head position, the facial key points of the object under test are detected to obtain the mouth key points of the object being captured.

[0161] In an alternative embodiment, after determining the head position of the object being captured in the monitoring video frame of the monitoring video sample, the monitoring video frame may be cropped according to the head position to obtain a head video sequence. Then, in the head video frame of the head video sequence, according to the face key point detection method, the facial key points of the object under test are detected, and the mouth key points of the object being captured are determined from the facial key points.

[0162] In an alternative embodiment, any open-source face key point detection method may be used to detect the facial key points in the head video frame of the head video sequence. Then, the mouth key points are determined from the detected facial key points.

[0163] In step S630, a first width-to-height ratio of the mouth is determined according to the target mouth key points among the mouth key points.

[0164] In an alternative embodiment, the target mouth key points may include the top key point, the bottom key point, the leftmost key point, and the rightmost key point among the mouth key points.

[0165] For example, the height of the mouth can be determined by the distance between the top key point and the bottom key point among the mouth key points, the width of the mouth can be determined by the distance between the leftmost key point and the rightmost key point among the mouth key points, and then, according to the ratio of the width of the mouth to the length of the mouth, the first width-to-height ratio corresponding to the mouth can be determined. In other words, the first width-to-height ratio of the mouth can be determined by the following formula (2):

[0166]

[0167] In formula (2), AR1 (Aspect Ratio) represents the first width-to-height ratio, width m represents the width of the mouth determined according to the distance between the leftmost key point and the rightmost key point of the mouth, and height m represents the height of the mouth determined according to the vertex key point of the mouth (i.e., the highest key point of the mouth) and the bottom key point of the mouth (i.e., the lowest key point of the mouth).

[0168] In step S640, according to the first change value of the first width-to-height ratio corresponding to adjacent monitoring video frames in the monitoring video sample, the sample mouth movement feature is determined.

[0169] In an alternative embodiment, when the difference between the first width-to-height ratios corresponding to any adjacent monitoring video frames in the monitoring video sample is greater than a second preset value, it can be determined that the sample mouth movement feature is mouth movement. When the difference between the first width-to-height ratios corresponding to any adjacent monitoring video frames in the monitoring video sample is less than or equal to the second preset value, it can be determined that the sample mouth movement feature is no mouth movement.

[0170] In an alternative embodiment, the eye movement feature includes a left eye movement feature and a right eye movement feature. The left eye movement feature includes left eye movement and no left eye movement, and the right eye movement feature includes right eye movement and no right eye movement. Exemplarily, Figure 7 a schematic flowchart showing a method for determining an eye movement feature in an exemplary embodiment of the present disclosure is shown. Refer to Figure 7 and this method may include steps S710 to S740.

[0171] In step S710, according to the head position detection model, the head position of the object to be captured in the monitoring video frame in the monitoring video sample is determined.

[0172] Exemplarily, the specific implementation manner of step S710 may also refer to the relevant content in step S410 above, and details are not described herein again.

[0173] In step S720, facial key points of the object to be captured are detected based on the head position to obtain the left-eye key point and the right-eye key point of the object to be captured.

[0174] For example, similarly, the facial key points of the object to be captured can be detected in the surveillance video frame based on the head position according to the facial key point detection algorithm, so as to determine the left-eye key point and the right-eye key point of the object to be captured from the detected facial key points.

[0175] When performing facial key point detection, each key point has a corresponding identifier. According to the key point identifier, the facial organ represented by the key point can be determined. For example, the 1st to 6th key points represent the left eye, and the 7th to 12th key points represent the right eye, etc. In other words, according to the key point identifier, the left-eye key point, the right-eye key point of the object to be captured, and the above-mentioned mouth key point can be determined.

[0176] In step S730, the second aspect ratio of the left eye in the surveillance video frame is determined according to the target left-eye key point in the left-eye key points, and the left-eye motion feature is determined based on the second change value of the second aspect ratio of the left eye in adjacent surveillance video frames.

[0177] In an alternative embodiment, the target left-eye key point may include the top key point of the left eye, the bottom key point of the left eye, the leftmost key point of the left eye, and the rightmost key point of the left eye.

[0178] For example, the width of the left eye can be determined according to the distance between the leftmost key point and the rightmost key point of the left eye, and the height of the left eye can be determined according to the distance between the top key point (i.e., the highest key point) and the bottom key point (i.e., the lowest key point) of the left eye, and then the second aspect ratio is determined according to the ratio between the width and the height of the left eye.

[0179] In an alternative embodiment, when the difference between the second aspect ratios of the left eye in any adjacent surveillance video frames in the surveillance video sample is greater than a third preset value, the left-eye motion feature can be determined as left-eye movement. When the difference between the second aspect ratios of the left eye in any adjacent surveillance video frames in the surveillance video sample is less than or equal to the third preset value, the left-eye motion feature of the sample can be determined as no left-eye movement. That is to say, as long as there is a difference greater than the third preset value between the second aspect ratios of the left eye in two adjacent surveillance video frames in the surveillance video sample, the left-eye motion feature is determined as left-eye movement. When the differences between the second aspect ratios of the left eye in all adjacent surveillance video frames in the surveillance video sample are less than or equal to the third preset value, the left-eye motion feature is determined as no left-eye movement.

[0180] In step S740, the third aspect ratio of the right eye in the monitored video frame is determined according to the target right eye key points among the right eye key points, and the right eye movement feature is determined based on the third change value of the third aspect ratio of the right eye in adjacent monitored video frames in the monitored video sample.

[0181] In an alternative embodiment, the target right eye key points may include the top key point of the right eye, the bottom key point of the right eye, the leftmost key point of the right eye, and the rightmost key point of the right eye.

[0182] For example, the width of the right eye can be determined according to the distance between the leftmost key point and the rightmost key point of the right eye, the height of the right eye can be determined according to the distance between the top key point (i.e., the highest key point) and the bottom key point (i.e., the lowest key point) of the right eye, and then the third aspect ratio can be determined according to the ratio between the width and the height of the right eye.

[0183] In an alternative embodiment, when the difference between the third aspect ratios of the right eye in any adjacent monitored video frames in the monitored video sample is greater than a fourth preset value, the right eye movement feature can be determined as right eye movement. When the difference between the third aspect ratios of the right eye in any adjacent monitored video frames in the monitored video sample is less than or equal to the fourth preset value, the right eye movement feature of the sample can be determined as the right eye not moving. That is to say, as long as the difference between the third aspect ratios of the right eye in two adjacent monitored video frames in the monitored video sample is greater than the fourth preset value, the right eye movement feature is determined as right eye movement. When the differences between the third aspect ratios of the right eye in all adjacent monitored video frames in the monitored video sample are less than or equal to the fourth preset value, the right eye movement feature is determined as the right eye not moving.

[0184] In an alternative embodiment, the nose movement feature includes nose movement and nose not moving. Exemplarily, Figure 8 The flowchart showing a method for determining the nose movement feature in an exemplary embodiment of the present disclosure is referred to Figure 8 and this method may include steps S810 to S830.

[0185] In step S810, according to the head position detection model, the head position of the object being captured in the monitored video frame of the monitored video sample is determined.

[0186] Exemplarily, the specific implementation manner of step S810 may also refer to the relevant content in step S410 above and will not be elaborated here.

[0187] In step S820, based on the head position, the facial key points of the object being captured are detected to obtain the sequence of nose key point positions of the object being captured in the monitored video sample.

[0188] Exemplarily, as described above, according to any face key point detection algorithm, based on the head position of the object to be captured in the surveillance video frame, the face key points of the object to be captured can be detected in the surveillance video frame, and then the nose key points in the surveillance video frame can be determined according to the face key point identification. According to the nose key points in the surveillance video frames in the surveillance video sample, the nose key point position sequence of the object to be captured in the surveillance video sample is obtained.

[0189] In step S830, according to the change situation of the nose key point positions indicated by the nose key point position sequence, the nose movement characteristics are determined.

[0190] In an alternative embodiment, the position difference of the nose in adjacent surveillance video frames can be determined according to the nose key point position sequence, such as whether the position distance is greater than a fifth preset value, to determine the nose movement characteristics. For example, if the distance between the positions of the nose in any adjacent surveillance video frames in the surveillance video sample is greater than the fifth preset value, it is determined that the nose movement characteristic is nose movement; in the case where the distance between the positions of the nose in any adjacent surveillance video frames in the surveillance video sample is less than or equal to the fifth preset value, it is determined that the nose movement characteristic is no nose movement.

[0191] For example, each nose key point detected in each surveillance video frame has nose key point coordinates. The movement distance of the nose key point between adjacent video frames can be determined according to the coordinates of the nose key points with the same identification in adjacent surveillance video frames. For example, the movement distance of the 3rd face key point (belonging to the nose key point) in adjacent surveillance video frames is determined. In the case where the movement distance of any nose key point between adjacent video frames is greater than the fifth preset value, it can be determined that the nose movement characteristic corresponding to the adjacent video frame is nose movement; in the case where the movement distance of each nose key point between adjacent video frames is less than or equal to the fifth preset value, it can be determined that the nose movement characteristic corresponding to the adjacent video frame is no nose movement. In the case where the nose movement characteristic corresponding to any adjacent surveillance video frame in the surveillance video sample is nose movement, it is determined that the nose movement characteristic corresponding to the surveillance video sample is nose movement; in the case where the nose movement characteristics corresponding to all adjacent surveillance video frames in the surveillance video sample are no nose movement, it is determined that the nose movement characteristic of the surveillance video sample is no nose movement.

[0192] Since the nose does not move on its own like the mouth or eyes, the movement of the nose follows that of the head. For example, when the head deflects, the nose follows. Also, the nose is not blocked by other objects such as quilts. That is to say, the nose key point can be detected in each monitored video frame. Therefore, in this disclosure, the position sequence of the nose key point can be used to detect whether the nose moves. At the same time, the nose movement feature can also reflect the head movement situation, improving the accuracy and comprehensiveness of the movement feature detection.

[0193] In an alternative embodiment, when labeling the tags of the training samples, since subtle movements such as blinking and slight lip movements are not easily observed manually, the movement features of the monitored video samples can be automatically determined according to the above related methods first. For example, it can be determined whether the body of the sampled object in the video monitoring sample moves, whether the eyes move, whether the mouth moves, whether the nose moves, and other movement features. Then, based on the movement features of the monitored video sample, a doctor adds a tag indicating whether there is abnormal discharge to the biomedical signal sample that belongs to the same time period as the monitored video sample. In this way, it can be avoided that when a doctor watches the monitored video manually, subtle movements of the sampled object such as blinking are not noticed, resulting in incorrect tag labeling. This improves the accuracy of the training sample tag labeling and further helps to improve the detection accuracy of the abnormal discharge detection model obtained by training.

[0194] In an alternative embodiment, a sample movement feature vector can be determined based on the determined sample movement features. For example, the dimension of the sample movement feature vector can be determined according to the types of movement features included in the sample movement features. The vector value at the corresponding position in the vector can be determined according to the movement feature of the movement feature type corresponding to this position. For example, if the movement feature of the movement feature type corresponding to a certain position is movement, the vector value at this position is 1; if the movement feature of the movement feature type corresponding to a certain position is non - movement, the vector value at this position is 0.

[0195] In an alternative embodiment, the types of movement features include the above - mentioned 6 types: body movement feature, head movement feature, left - eye movement feature, right - eye movement feature, mouth movement feature, and nose movement feature. And the sample movement features include at least one of the above 6 movement types. If the sample movement features include 3 types of movement features, the sample movement feature vector is a 3 - dimensional vector; if the sample movement features include 1 type of movement feature, the sample movement feature is a 1 - dimensional vector, and so on.

[0196] Taking the sample motion features including body motion features, head motion features, left-eye motion features, right-eye motion features, mouth motion features, and nose motion features as an example, the dimension of the sample motion feature vector is 6. Taking the 6D vector as a row vector as an example, the types of motion features represented by different columns in the 6D row vector can be preset. Taking the first column representing body motion features, the second column representing head motion features, the third column representing left-eye motion features, the fourth column representing right-eye motion features, the fifth column representing mouth motion features, and the sixth column representing nose motion features as an example, if the body motion feature is body motion, the head motion feature is no head motion, the left-eye motion feature is no left-eye motion, the right-eye motion feature is right-eye motion, the mouth motion feature is mouth motion, and the nose motion feature is nose motion, then the vector value of the sample motion feature vector is [1, 0, 0, 1, 1, 1]. Of course, the 6D vector can also be a column vector, and this exemplary embodiment does not make special limitations on this.

[0197] Next, in combination with Figure 9 the above-mentioned exemplary specific implementation manner of inputting the biomedical signal features and the sample motion features into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample will be described. Exemplarily, Figure 9 FIG. shows a flowchart of a method for obtaining the abnormal discharge detection result of a training sample in an exemplary embodiment of the present disclosure. Referring to Figure 9 this, the method may include steps S910 to S940. Among them:

[0198] In step S910, feature enhancement is performed on the sample motion features to increase the feature dimension of the sample motion features and obtain the target sample motion features.

[0199] As described above, the dimension of the sample motion feature vector is less than or equal to 6. Compared with the biomedical signal sample feature vector, the vector length of the sample motion feature is too short. If the two are directly concatenated and input into the abnormal discharge detection model to be trained, the sample motion features may be ignored to a certain extent, affecting the training effect of the model.

[0200] In an optional implementation manner, feature enhancement can be performed on the sample motion features to increase the dimension of the sample motion features. For example, taking the vector dimension of the extracted biomedical signal sample features as 1024, the sample motion features can be increased to 128 dimensions, 256 dimensions, 512 dimensions, or 1024 dimensions, etc., to reduce the dimension gap between the sample motion features and the biomedical signal features and avoid the situation where the sample motion features are ignored due to the large gap between the two.

[0201] In an alternative embodiment, the motion feature enhancement network may include a multi-layer perceptron. Of course, the motion feature enhancement network may also include other networks capable of transforming low-dimensional features into high-dimensional features, and this exemplary embodiment does not make special limitations thereon.

[0202] In step S920, the biomedical signal features and the target sample motion features are concatenated to obtain a first concatenated feature.

[0203] In an alternative embodiment, the feature vectors corresponding to the biomedical signal features and the feature vectors corresponding to the target sample motion features can be directly concatenated to obtain a higher-dimensional vector, and this higher-dimensional vector is used as the first concatenated feature.

[0204] In step S930, the first concatenated feature is input into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample.

[0205] In an exemplary embodiment, the first concatenated feature can be used as the input of the abnormal discharge detection model to be trained, and then the abnormal discharge detection model to be trained can perform feature learning on the input first concatenated feature and output the predicted abnormal discharge detection result according to the feature learning result.

[0206] In an alternative embodiment, the abnormal discharge detection model to be trained can be any machine learning model capable of classification. For example, a classifier model based on a multi-layer perceptron, a classifier model based on a convolutional neural network, etc. That is, in the present disclosure, the abnormal discharge detection task can be transformed into a binary classification task, and the classification results include two types: the existence of abnormal discharge and the non-existence of abnormal discharge.

[0207] Next, Figure 1 the specific implementation of step S140 in

[0208] In an alternative embodiment, after obtaining the abnormal discharge detection result of the training sample, the difference between the predicted abnormal discharge detection result of the training sample and the sample label of the training sample can be calculated, and this difference is used as the training loss value of the abnormal discharge detection model to be trained.

[0209] Among them, the above difference can be specifically measured by the mean absolute error, the mean square error, the Euclidean distance, etc., and can be set according to the actual situation, and the present disclosure does not make special limitations thereon.

[0210] Next, Figure 1 the specific implementation of step S150 in

[0211] In an alternative embodiment, the preset conditions may include that the number of iterative training reaches a preset number or the loss value meets a preset convergence condition, such as being less than a preset loss value.

[0212] For example, according to the direction of the decrease in the loss value, the model parameters of the above-mentioned abnormal discharge detection model to be trained can be continuously updated until the loss value is less than the preset loss value threshold or until the number of iterations reaches the preset number, and then the trained abnormal discharge detection model can be obtained.

[0213] In an alternative embodiment, the above-mentioned biomedical signal feature extraction network, motion feature enhancement network, and abnormal discharge detection model to be trained are jointly trained. In other words, after obtaining the training samples and the sample labels of the training samples, biomedical signal image samples can be first generated, and then the biomedical signal image samples are input into the biomedical signal feature extraction network to obtain a first output. Then, at the same time, the monitoring video samples in the training samples are input into the patient detection model and the face detection model to obtain the body position and head position of the object being collected. Furthermore, based on the head position and body position of the object being collected in the monitoring video frame, the sample motion features corresponding to the training samples are determined according to the above-mentioned motion feature determination method. Then, the sample motion features are input into the motion feature enhancement network to obtain a second output. Then, the first output and the second output are concatenated and input into the abnormal discharge detection model to be trained to predict the abnormal discharge detection results of the training samples.

[0214] When adjusting the network parameters according to the loss value, the network parameters of the biomedical signal feature extraction network, the motion feature enhancement network, and the abnormal discharge detection model to be trained can be adjusted so that the loss value of the subsequent training is less than the loss value of the previous training, that is, iterative learning is carried out in the direction of decreasing the loss value.

[0215] Exemplarily, Figure 10 The flowchart showing another method for training an abnormal discharge detection model in an exemplary embodiment of the present disclosure is shown. Refer to Figure 10, the method may include steps S1010 to S1070. Among them: in step S1010, training samples and sample labels are obtained, and according to the signal curves of the biomedical signal samples in the training samples, biomedical signal image samples are obtained; in step S1020, the biomedical signal image samples are input into a biomedical signal feature extraction network to obtain a first output; in step S1020, the surveillance video samples are input into a patient detection model, and patient position video samples are obtained according to the output results; in step S1030, the patient position video samples are input into a face detection model, and head video samples and body video samples are obtained according to the output results; in step S1040, sample motion features are determined according to the head video samples and body video samples; in step S1050, the sample motion features are input into a motion feature enhancement network to obtain a second output; in step S1060, the first output and the second output are concatenated and then input into a to-be-trained abnormal discharge detection model to obtain a predicted abnormal discharge detection result; in step S1070, a model loss value is calculated according to the predicted abnormal discharge detection result and the sample labels, and the model parameters are updated until a trained abnormal discharge detection model is obtained.

[0216] Among them, in step S1070, updating the model parameters can be understood as updating the parameters of at least one of the biomedical signal feature extraction network, the motion feature enhancement network, and the abnormal discharge detection model.

[0217] After training the above abnormal discharge detection model, abnormal discharge detection can be performed according to the trained abnormal discharge detection model. Refer to Figure 11 , Figure 11 FIG. shows a schematic flowchart of an abnormal discharge detection method in an exemplary embodiment of the present disclosure. The method may include steps S1110 to S1130. Among them: step S1110, obtaining the biomedical signal and the surveillance video of the subject to be measured; step S1120, obtaining a biomedical signal image according to the signal curve of the biomedical signal; step S1130, inputting the biomedical signal image and the surveillance video into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be measured.

[0218] Next, the specific implementation manner of step S1110 will be described.

[0219] In an optional implementation manner, the surveillance video is obtained by shooting when the biomedical signal of the subject to be measured is collected.

[0220] In an alternative embodiment, the subject under test can be a patient seeking medical treatment with epilepsy or epilepsy-like symptoms, or other people seeking medical treatment due to discomfort in the brain, such as those with brain pain, etc. This exemplary embodiment does not make special limitations in this regard.

[0221] For example, a biomedical signal acquisition device can be used to acquire biomedical signals. For example, an electroencephalograph can be used to acquire the electroencephalogram signals of the subject under test. Then, while acquiring the biomedical signals, the subject under test is video-recorded to obtain a monitoring video of the subject under test when acquiring the biomedical signals. Among them, the acquisition duration and recording duration can be determined according to requirements. For example, the biomedical signals are acquired for 30 minutes and the subject under test is video-recorded within these 30 minutes to obtain a monitoring video with a duration of 30 minutes.

[0222] Next, the specific implementation of step S1120 will be described.

[0223] In an alternative embodiment, the biomedical signal includes multi-channel biomedical signals. The exemplary implementation of step S1120 may include: obtaining a biomedical signal image according to the signal curve of the multi-channel biomedical signals; or obtaining multiple biomedical signal images according to the signal curve of the multi-channel biomedical signals.

[0224] Among them, other relevant descriptions of step S1120 can refer to the relevant content in step S120 above, and will not be elaborated here.

[0225] Next, in combination with Figure 12 、 Figure 13 and Figure 14 The exemplary implementation of step S1130 will be described.

[0226] Exemplarily, Figure 12 shows a schematic flowchart of a method for obtaining an abnormal discharge detection result in an exemplary embodiment of the present disclosure. Referring to Figure 12 , the method may include steps S1210 to S1230. Among them: in step S1210, the biomedical signal image is sliced according to a preset duration to obtain a biomedical signal sub-image; in step S1220, the monitoring video is segmented according to the preset duration to obtain a monitoring video segment belonging to the same time period as the biomedical signal sub-image; in step S1230, the biomedical signal sub-image and the monitoring video segment are input into an abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test.

[0227] Exemplarily, the biomedical signal image can be sliced and the surveillance video can be segmented according to the same starting time and the same preset duration to obtain a biomedical signal sub-image and a surveillance video segment in the same time period. Then, abnormal discharge detection is performed based on the biomedical signal sub-image and the surveillance video segment in the same time period.

[0228] Exemplarily, Figure 13 A schematic flowchart showing another method for obtaining an abnormal discharge detection result in an exemplary embodiment of the present disclosure is referred to Figure 13 , the method may include steps S1310 to S1330. Among them: in step S1310, the biomedical signal image is sliced according to a preset duration to obtain a biomedical signal sub-image; in step S1320, when the biomedical signal sub-image indicates that the subject under test has an abnormal discharge, a surveillance video segment in the same time period as the biomedical signal sub-image is intercepted from the surveillance video; in step S1330, the biomedical signal sub-image and the surveillance video segment are input into an abnormal discharge detection model to obtain an abnormal discharge detection result of the subject under test.

[0229] In an alternative embodiment, a first abnormal discharge detection model can be pre-trained. The input of the first abnormal discharge detection model is a biomedical signal image, and the output is whether there is an abnormal discharge in the biomedical signal image. For example, the biomedical signal sub-image can be input into the first abnormal discharge detection model to obtain a prediction result of whether there is an abnormal discharge in the biomedical signal sub-image. When the prediction result of the biomedical signal sub-image is that the subject under test has an abnormal discharge, a surveillance video segment in the same time period as the biomedical signal sub-image is intercepted from the surveillance video. Then, the biomedical signal sub-image and the surveillance video segment determined according to the biomedical signal sub-image are input into the abnormal discharge detection model together for abnormal discharge detection.

[0230] In an alternative embodiment, when performing abnormal discharge detection, the collected biomedical signals and monitoring videos can be obtained. First, the biomedical signals can be segmented to obtain sub-images of the biomedical signals corresponding to each segment. Alternatively, a biomedical signal image can be generated based on the collected biomedical signals first, and then the biomedical signal image can be sliced according to a preset duration to obtain sub-images of the biomedical signals. After obtaining the sub-images of the biomedical signals, the sub-images of the biomedical signals can be input into the above-mentioned first abnormal discharge detection model first to obtain the abnormal discharge prediction results of the sub-images of the biomedical signals. When the abnormal discharge prediction results indicate normal discharge of the sub-images of the biomedical signals, it can be directly determined that the subject under test has normal discharge during the time period corresponding to the sub-images of the biomedical signals. When the abnormal discharge prediction results indicate abnormal discharge of the sub-images of the biomedical signals, a monitoring video segment corresponding to the same time period as the sub-images of the biomedical signals with abnormal discharge can be intercepted from the obtained monitoring videos, and then the sub-images of the biomedical signals and the monitoring video segments determined according to the sub-images of the biomedical signals can be combined, and it can be determined whether the subject under test actually has abnormal discharge during this time period according to the abnormal discharge detection model.

[0231] If the abnormal discharge detection model determines that the abnormal discharge detection result is abnormal discharge, it is determined that the subject under test has abnormal discharge during the time period corresponding to the sub-images of the biomedical signals. If the abnormal discharge detection model determines that the abnormal discharge detection result is normal discharge, that is, there is no abnormal discharge, it is determined that the subject under test has no abnormal discharge during the time period corresponding to the sub-images of the biomedical signals. If the subject under test has no abnormal discharge during the time period corresponding to each sub-image of the biomedical signals, it is determined that the abnormal discharge detection result of the subject under test is normal. If the subject under test has abnormal discharge during the time period corresponding to any sub-image of the biomedical signals, it is determined that the abnormal discharge detection result of the subject under test is that there is abnormal discharge, and the biomedical signal segment with abnormal discharge and its corresponding monitoring video segment can be output to facilitate the doctor to reconfirm the detection result.

[0232] Through the above steps S1310 to S1330, it can be determined whether there is abnormal discharge according to the sub-images of the biomedical signals in a single modality first. When there is abnormal discharge in the sub-images of the biomedical signals, the sub-images of the biomedical signals and the corresponding monitoring video segments are combined, and it is determined whether there is actually abnormal discharge based on the multi-modal input. Since the processing efficiency of the single-modal input is higher than that of the multi-modal input, and the prediction accuracy of the multi-modal input is higher than that of the single-modal input, the detection efficiency of the abnormal discharge can be improved and the accuracy of the abnormal discharge detection can be ensured.

[0233] In an alternative embodiment, inputting the biomedical signal image and the monitoring video into an abnormal discharge detection model to obtain the abnormal discharge detection result of the subject includes: inputting the biomedical signal image into a biomedical signal feature extraction network to obtain biomedical signal features; determining the motion features of the subject according to the monitoring video, where the motion features include at least one of body motion features, head motion features, eye motion features, mouth motion features, and nose motion features; inputting the biomedical signal features and the motion features into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject.

[0234] In an alternative embodiment, the motion features of the subject may include at least one of body motion features, head motion features, eye motion features, mouth motion features, and nose motion features.

[0235] In an alternative embodiment, the body motion features include body motion and no body motion, and the determination method of the body motion features includes: for any video frame of the monitoring video to be detected, determining the head position of the subject in the video frame according to a head position detection model, and cropping the video frame according to the head position to obtain a body video frame corresponding to the video frame; for any one of the body video frames, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining the sum of the pixel value differences at the same position between the body video frame and the reference video frame, and determining the frame difference of the body video frame according to the maximum value among the sums of the pixel value differences; determining the body motion features of the subject according to the frame difference.

[0236] For example, the determination method of the body motion features of the subject may refer to the determination method of the sample body motion features shown above. Figure 4 and will not be elaborated here.

[0237] In an alternative embodiment, the head movement feature includes head movement and no head movement. The determination method of the head movement feature includes: determining the head position of the subject in the video frame of the to-be-detected surveillance video according to a head position detection model; cropping the video frame of the to-be-detected surveillance video based on the head position to obtain a head surveillance video of the subject; inputting the head surveillance video into a pre-trained head movement detection model, and determining that the head movement feature is head movement when the output result of the head movement detection model indicates head movement, otherwise, determining that the head movement feature is no head movement; wherein, the training label of the head movement detection model is determined according to whether the head of the object in the training video moves and whether there are facial expression changes.

[0238] For example, the determination method of the head movement feature of the subject can refer to the above Figure 5 shown determination method of the head movement feature, which will not be elaborated here.

[0239] In an alternative embodiment, the mouth movement feature includes mouth movement and no mouth movement, the eye movement feature includes a left eye movement feature and a right eye movement feature, the left eye movement feature includes left eye movement and no left eye movement, the right eye movement feature includes right eye movement and no right eye movement. The determination methods of the mouth movement feature and the eye movement feature include: determining the head position of the subject in the video frame of the to-be-detected surveillance video according to a head position detection model; detecting the facial key points of the subject based on the head position to obtain the mouth key points, left eye key points and right eye key points of the subject; determining the first aspect ratio of the mouth according to the target mouth key points in the mouth key points; determining the second aspect ratio of the left eye according to the target left eye key points in the left eye key points, and determining the third aspect ratio of the right eye according to the target right eye key points in the right eye key points; determining the mouth movement feature according to the first change value of the first aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; determining the left eye movement feature according to the second change value of the second aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; determining the right eye movement feature according to the third change value of the third aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video.

[0240] For example, the determination method of the mouth movement feature and the determination method of the eye movement feature of the subject can correspondingly refer to the above Figure 6 shown determination method of the sample mouth movement feature and Figure 7 shown determination method of the sample mouth movement feature, which will not be elaborated here.

[0241] In an alternative embodiment, the nose movement feature includes nose movement and non - nose movement. The determination method of the nose movement feature includes: determining the head position of the subject in the video frame of the to - be - detected monitoring video according to the head position detection model; detecting the facial key points of the subject based on the head position to obtain the nose key point position sequence of the subject in the to - be - detected monitoring video; and determining the nose movement feature according to the change of the nose key point positions indicated by the nose key point position sequence.

[0242] For example, the determination method of the nose movement feature of the subject can refer to the determination method of the sample nose movement feature shown above. Figure 8 Details are not described here again.

[0243] Next, Figure 14 A detailed description of the specific implementation manner of inputting the biomedical signal feature and the movement feature into the pre - trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject is given.

[0244] Exemplarily, Figure 14 FIG. shows a flowchart of yet another method for obtaining an abnormal discharge detection result in an exemplary embodiment of the present disclosure. Referring to Figure 14 , the method may include steps S1410 to S1440. Among them: in step S1410, feature enhancement is performed on the movement feature to increase the feature dimension of the movement feature, obtaining a target movement feature; in step S1420, the biomedical signal feature and the target movement feature are spliced to obtain a spliced feature; in step S1430, the spliced feature is input into the pre - trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject.

[0245] For example, the specific implementation manners of steps S1410 to S1430 can correspondingly refer to the specific implementation manners of steps S910 to S930 above. Details are not described here again.

[0246] In an alternative embodiment, when the abnormal discharge detection result predicted by the abnormal discharge detection model is that there is an abnormality, it is determined that the subject has abnormal discharge during the time period corresponding to the biomedical signal sub - image, which indicates that the subject has abnormal discharge. When the abnormal discharge detection result predicted by the abnormal discharge detection model is that there is no abnormality, it is determined that the subject has no abnormal discharge during the time period corresponding to the biomedical signal sub - image.

[0247] Exemplarily, Figure 15The structural diagram of an abnormal discharge detection system in an exemplary embodiment of the present disclosure is shown. It may include a data acquisition module 1510, a biomedical signal feature extraction module 1520, a motion feature determination module 1530, a feature fusion module 1540, and an abnormal detection classification result determination module 1550.

[0248] In an alternative embodiment, the data in the present disclosure is from multimodal electrical data (such as electroencephalogram, electrocardiogram (ECG), and electromyogram (EMG)) and video data. That is, the data acquisition module 1510 acquires two types of data, namely biomedical signal electrical data and monitoring video data. Then, the data acquisition module 1510 sends the acquired biomedical signal electrical data to the biomedical signal feature extraction module 1520 and sends the monitoring video data to the motion feature determination module 1530. Then, the biomedical signal feature extraction module 1510 transforms the electrical data through short-time Fourier transform (STFT). Subsequently, EfficientNetV2-S is used to extract meaningful electrical signal features. The motion feature determination module 1540 decomposes the video into image frames, uses a detection model based on YOLOv5 to detect the patient and the face to obtain the facial video sequence and torso video sequence of the patient, and determines the motion features of the patient based on the facial video sequence and torso video sequence of the patient according to the above motion feature method. Finally, after enhancing the motion features, the feature fusion module 1540 splices and fuses the biomedical features and the enhanced motion features, and inputs them into the classifier in the abnormal detection classification result determination module 1550. The classifier can output the abnormal discharge detection result according to the fused features.

[0249] Taking the detection of abnormal brain discharges as an example, the abnormal discharge detection method in the present disclosure will be described in combination with a specific application scenario.

[0250] Reference Figure 16 , Figure 16 shows a schematic diagram of an exemplary application scenario of the abnormal discharge detection method according to an embodiment of the present disclosure, as Figure 16 shown: 1601 is the subject to be measured, 1603a, 1603b, and 1603c are detection electrodes, 1607 is an electroencephalogram signal acquisition device, 1605 is the interface of the electroencephalogram signal acquisition device 1607, 1609 is an abnormal discharge detection device developed based on the abnormal brain discharge detection method in the present disclosure, 1611 is a display (which may include the mobile phone, computer, etc. of the subject to be measured, and the present disclosure makes no special limitation on this), and 1616 is a camera.

[0251] For example, the detection electrodes 1603a, 1603b, and 1603c connected to the electroencephalogram (EEG) signal acquisition device 1607 can be fixed at various positions on the head of the subject 1601, so as to collect multi-channel EEG signals of the subject 1601 through the EEG signal acquisition device 1607. It should be noted that the above-mentioned EEG signal acquisition device 1607 can also include devices in various forms such as helmets, hats, bedding, and packaged portable detection electrodes. The present disclosure does not make special limitations on this.

[0252] While collecting the above multi-channel EEG signals, the subject 1601 can be synchronously monitored through the camera 1616 to capture video monitoring data including the face and torso of the subject 1601, so as to record the motion state of the subject.

[0253] After obtaining the multi-channel EEG signals and video monitoring data of the subject, the above multi-channel EEG signals and video monitoring data can be transmitted to the abnormal discharge detection device 1609. Furthermore, the abnormal discharge detection device 1609 can output an abnormal discharge detection result for the subject based on the brain abnormal discharge detection method in the present disclosure. After obtaining the abnormal discharge detection result, the abnormal discharge detection result can be displayed on the display 1611, so as to facilitate the doctor or the subject to understand the specific detection result.

[0254] It should be noted that the EEG signal acquisition device 1607 in the present disclosure can also be coupled with the abnormal discharge detection device 1609, that is, integrated into the same device. The present disclosure does not make special limitations on this.

[0255] Reference Figure 17 , Figure 17 shows a schematic diagram of another exemplary application scenario of the abnormal discharge detection method in the embodiments of the present disclosure, as Figure 17 shown: The EEG signal acquisition device 1607 in this scenario can be a portable device, and the camera 1616 can be a camera carried by a portable terminal such as a mobile phone or a PAD. Thus, it is convenient for the subject to collect EEG signals and video monitoring data at any time and place, such as at home or in the office. 1610 is shown as a data transceiver device, and 1615 is a server. The server 1615 can include any form of data processing server such as a cloud server or a distributed server. The present disclosure does not make special limitations on this.

[0256] Thus, after the multi-channel EEG signals of the subject are collected by the EEG signal acquisition device 1607 and the video monitoring data of the subject are collected by the camera 1616, the above multi-channel EEG signals and video monitoring data can be transmitted to the data transceiver device 1610. Furthermore, the data transceiver device 1610 can upload the above multi-channel EEG signals and video monitoring data to the server 1615.

[0257] Furthermore, the server 1615 can output an abnormal discharge detection result for the subject based on the abnormal discharge detection method in the present disclosure. After outputting the abnormal discharge detection result, the abnormal discharge detection result can be displayed on the display 1611 so that doctors or the subject can understand the specific detection result.

[0258] Through the above scenario, the subject can understand the abnormal discharge detection result without leaving home. When encountering problems such as suspected epilepsy, the abnormal discharge detection result can be sent to the doctor to facilitate the doctor to determine the diagnosis result.

[0259] Based on the above technical solutions, the present disclosure can at least achieve the following technical effects:

[0260] On the one hand, the abnormal discharge detection result can be obtained without manual reading of the graph, which improves the efficiency of abnormal discharge recognition. On the other hand, the training labels of the abnormal discharge detection model of the present disclosure are annotated by manually consulting biomedical signal images and corresponding monitoring videos. Therefore, using the biomedical signal images as the input of the abnormal discharge detection model can make the input data format of the model consistent with the data format during the training label annotation, so that the model can better simulate the scenario of doctors reading the graph to judge abnormal discharge and improve the accuracy of abnormal discharge detection. On the other hand, since the abnormal discharge detection result determined by the method of the present disclosure is highly accurate, it can provide high reference value for doctors, assist doctors in making rapid and accurate clinical diagnoses, save the workload of doctors, and is convenient for wide application in clinical practice.

[0261] Exemplary Device

[0262] After introducing the abnormal discharge detection method of the exemplary embodiment of the present disclosure, next, reference is made to Figure 18 the abnormal discharge detection device in the exemplary of the present disclosure is described.

[0263] Figure 18An abnormal discharge detection device 1800 in an exemplary embodiment of the present disclosure is shown, including: a data acquisition module 1810 configured to acquire a biomedical signal and a monitoring video of a subject to be measured, where the monitoring video is obtained by photographing the subject to be measured when acquiring the biomedical signal of the subject to be measured; an image generation module 1820 configured to obtain a biomedical signal image according to a signal curve of the biomedical signal; and a first detection module 1830 configured to input the biomedical signal image and the monitoring video into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject to be measured.

[0264] In an alternative embodiment, the first detection module 1830 is specifically configured to: slice the biomedical signal image according to a preset duration to obtain a sub-image of the biomedical signal; segment the monitoring video according to the preset duration to obtain a monitoring video segment that belongs to the same time period as the sub-image of the biomedical signal; and input the sub-image of the biomedical signal and the monitoring video segment into the abnormal discharge detection model to obtain an abnormal discharge detection result of the subject to be measured.

[0265] In an alternative embodiment, the first detection module 1830 is specifically configured to: slice the biomedical signal image according to a preset duration to obtain a sub-image of the biomedical signal; when the sub-image of the biomedical signal indicates that the subject to be measured has abnormal discharge, extract a monitoring video segment that belongs to the same time period as the sub-image of the biomedical signal from the monitoring video; and input the sub-image of the biomedical signal and the monitoring video segment into the abnormal discharge detection model to obtain an abnormal discharge detection result of the subject to be measured.

[0266] In an alternative embodiment, the biomedical signal includes a multi-channel biomedical signal, and the image generation module 1820 is configured to: obtain one biomedical signal image according to the signal curves of the multi-channel biomedical signal; or obtain multiple biomedical signal images according to the signal curves of the multi-channel biomedical signal.

[0267] In an alternative embodiment, the first detection module 1830 is configured to: input the biomedical signal image into a biomedical signal feature extraction network to obtain biomedical signal features; determine motion features of the subject to be measured according to the monitoring video, where the motion features include at least one of body motion features, head motion features, eye motion features, mouth motion features, and nose motion features; and input the biomedical signal features and the motion features into a pre-trained abnormal discharge detection model to obtain an abnormal discharge detection result of the subject to be measured.

[0268] In an alternative embodiment, the step of inputting the biomedical signal feature and the motion feature into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested includes: enhancing the feature of the motion feature to increase the feature dimension of the motion feature, thereby obtaining a target motion feature; concatenating the biomedical signal feature and the target motion feature to obtain a concatenated feature; and inputting the concatenated feature into the pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject to be tested.

[0269] In an alternative embodiment, when the motion feature includes a body motion feature, the body motion feature includes body motion and body non-motion, and the determination method of the body motion feature includes: for any monitoring video frame of the monitoring video to be detected, determining the head position of the subject to be tested in the monitoring video frame according to a head position detection model, and cropping the monitoring video frame according to the head position to obtain a body video frame corresponding to the monitoring video frame; for any one of the body video frames, determining a reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining a frame difference corresponding to the reference video frame according to the sum of pixel value differences at the same position between the body video frame and the reference video frame; determining a target frame difference of the body video frame according to the maximum value among the frame differences corresponding to the reference video frames, and determining the body motion feature of the subject to be tested based on the target frame difference.

[0270] In an alternative embodiment, when the motion feature includes a head motion feature, the head motion feature includes head motion and head non-motion, and the determination method of the head motion feature includes: determining the head position of the subject to be tested in the video frame of the monitoring video to be detected according to a head position detection model; cropping the video frame of the monitoring video to be detected based on the head position to obtain a head monitoring video of the subject to be tested; inputting the head monitoring video into a pre-trained head motion detection model, and when the output result of the head motion detection model indicates head motion, determining that the head motion feature is head motion, otherwise, determining that the head motion feature is head non-motion; wherein, the training label of the head motion detection model is determined according to whether the head of the object in the training video moves and whether there are facial expression changes.

[0271] In an alternative embodiment, when the motion features include mouth motion features and / or eye motion features, the mouth motion features include mouth movement and non-movement of the mouth, the eye motion features include left-eye motion features and right-eye motion features, the left-eye motion features include left-eye movement and non-movement of the left eye, the right-eye motion features include right-eye movement and non-movement of the right eye, and the determination methods for the mouth motion features and the eye motion features are as follows: according to the head position detection model, determine the head position of the subject in the video frame of the to-be-detected surveillance video; based on the head position, detect the facial key points of the subject to obtain the mouth key points, left-eye key points, and right-eye key points of the subject; determine the first aspect ratio of the mouth according to the target mouth key points among the mouth key points; determine the second aspect ratio of the left eye according to the target left-eye key points among the left-eye key points, and determine the third aspect ratio of the right eye according to the target right-eye key points among the right-eye key points; determine the mouth motion features according to the first change value of the first aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; determine the left-eye motion features according to the second change value of the second aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video; and determine the right-eye motion features according to the third change value of the third aspect ratio corresponding to adjacent frames in the to-be-detected surveillance video.

[0272] In an alternative embodiment, when the motion features include nose motion features, the nose motion features include nose movement and non-movement of the nose, and the determination method for the nose motion features is as follows: according to the head position detection model, determine the head position of the subject in the video frame of the to-be-detected surveillance video; based on the head position, detect the facial key points of the subject to obtain the nose key point position of the subject; and determine the nose motion features according to the change situation of the nose key point positions indicated by the nose key point position sequence in the video frame of the to-be-detected surveillance video.

[0273] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the embodiments of the invention of the above abnormal discharge detection method, and will not be elaborated herein.

[0274] Next, refer to Figure 19 to describe the training device for the abnormal discharge detection model in the exemplary embodiment of the present disclosure. Figure 19The training device 1900 for the abnormal discharge detection model in an exemplary embodiment of the present disclosure is shown, including: a training sample acquisition module 1910 configured to acquire training samples and sample labels corresponding to the training samples, where the training samples include biomedical signal samples and monitoring video samples of a subject to be tested, and the monitoring video samples are obtained by photographing the subject to be tested when collecting the biomedical signal samples of the subject to be tested; an image sample generation module 1920 configured to obtain biomedical signal image samples according to the signal curves of the biomedical signal samples; a second detection module 1930 configured to input the biomedical signal image samples and the monitoring video samples into an abnormal discharge detection model to be trained to obtain the abnormal discharge detection results of the training samples; a loss determination module 1940 configured to determine the loss value of the abnormal discharge detection model to be trained according to the difference between the abnormal discharge detection results of the training samples and the sample labels; and an iterative training module 1950 configured to perform iterative training on the abnormal discharge detection model to be trained according to the loss value until a preset condition is met to obtain a pre-trained abnormal discharge detection model.

[0275] In an alternative embodiment, the second detection module 1930 may be specifically configured to: determine the sample motion features corresponding to the monitoring video samples, where the sample motion features include at least one of sample body motion features, sample head motion features, sample eye motion features, sample mouth motion features, and sample nose motion features; input the biomedical signal image samples into a biomedical signal feature extraction network to obtain a first output; input the sample motion features into a motion feature enhancement network to obtain a second output; splice the first output and the second output to obtain a target splicing result; input the target splicing result into the abnormal discharge detection model to be trained, and obtain the abnormal discharge detection results of the training samples according to the output of the abnormal discharge detection model to be trained.

[0276] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the embodiments of the invention of the above method, and will not be repeated here.

[0277] Exemplary Storage Medium

[0278] The storage medium of the exemplary embodiment of the present disclosure will be described below.

[0279] In this exemplary embodiment, the above method can be implemented by a program product. For example, a portable compact disc read-only memory (CD-ROM) can be used, which includes program code and can run on a device, such as a personal computer. However, the program product of the present disclosure is not limited to this. In this document, a readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0280] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0281] The computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, and this readable medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0282] The program code contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0283] The program code for performing the operations of the present disclosure can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages - such as Java, C++, etc., and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0284] Exemplary Electronic Device

[0285] Reference Figure 20 An electronic device according to an exemplary embodiment of the present disclosure will be described.

[0286] Figure 20 The electronic device 2000 shown is merely an example and should not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.

[0287] As Figure 20 shown, the electronic device 2000 is presented in the form of a general-purpose computing device. The components of the electronic device 2000 may include, but are not limited to: at least one processing unit 2010, at least one storage unit 2020, a bus 2030 connecting different system components (including the storage unit 2020 and the processing unit 2010), and a display unit 2040.

[0288] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 2010, so that the processing unit 2010 executes the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the processing unit 2010 can execute the method steps as Figure 1 shown and / or the steps of the method as Figure 11 shown, etc.

[0289] The storage unit 2020 may include a volatile storage unit, such as a random access storage unit (RAM) 2021 and / or a cache storage unit 2022, and may further include a read-only storage unit (ROM) 2023.

[0290] The storage unit 2020 may further include a program / utility 2024 having a set (at least one) of program modules 2025. Such program modules 2025 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0291] The bus 2030 may include a data bus, an address bus, and a control bus.

[0292] The electronic device 2000 can also communicate with one or more external devices 2100 (such as a keyboard, a pointing device, a Bluetooth device, etc.), and this communication can be carried out through the input / output (I / O) interface 2050. The electronic device 2000 further includes a display unit 2040, which is connected to the input / output (I / O) interface 2050 for display. Also, the electronic device 2000 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 2060. As shown in the figure, the network adapter 2060 communicates with other modules of the electronic device 2000 through the bus 2030. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 2000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0293] It should be noted that although several modules or sub-modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more of the above-described units / modules can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0294] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0295] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefit. This division is only for the convenience of description. The present disclosure aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An abnormal discharge detection method, characterized in that, Including: Obtaining the biomedical signal and the monitoring video of the subject under test, where the monitoring video is obtained by photographing the subject under test when the biomedical signal of the subject under test is collected; Taking the sampling time of the biomedical signal as the abscissa and the sampling value of the biomedical signal as the ordinate to plot the signal curve of the biomedical signal, and determining the signal curve of the biomedical signal as the biomedical signal image; Inputting the biomedical signal image and the monitoring video into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test; The step of inputting the biomedical signal image and the monitoring video into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test includes: inputting the biomedical signal image into a biomedical signal feature extraction network to obtain biomedical signal features; determining the motion features of the subject under test according to the monitoring video; inputting the biomedical signal features and the motion features into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test; Wherein, the motion features include body motion features, the body motion features include body motion and no body motion, and the determination method of the body motion features includes: for any monitoring video frame of the monitoring video, determining the head position of the subject under test in the monitoring video frame according to the head position detection model, and cropping the monitoring video frame according to the head position to obtain the body video frame corresponding to the monitoring video frame; for any body video frame, determining the reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining the frame difference corresponding to the reference video frame according to the sum of the pixel value differences at the same position of the body video frame and the reference video frame; determining the target frame difference of the body video frame according to the maximum value in the frame differences corresponding to the reference video frames, and determining the body motion features of the subject under test based on the target frame difference.

2. The abnormal discharge detection method according to claim 1, wherein The step of inputting the biomedical signal image and the monitoring video into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test includes: Performing image slicing on the biomedical signal image according to a preset duration to obtain biomedical signal sub-images; Segmenting the monitoring video according to the preset duration to obtain a monitoring video segment belonging to the same time period as the biomedical signal sub-images; Inputting the biomedical signal sub-images and the monitoring video segments into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test.

3. The abnormal discharge detection method according to claim 1, wherein The step of inputting the biomedical signal image and the monitoring video into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test includes: Performing image slicing on the biomedical signal image according to a preset duration to obtain biomedical signal sub-images; When the biomedical signal sub-image indicates that there is abnormal discharge in the subject under test, extract a monitoring video segment from the monitoring video that belongs to the same time period as the biomedical signal sub-image; Input the biomedical signal sub-image and the monitoring video segment into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test.

4. The abnormal discharge detection method according to claim 1, wherein The biomedical signal includes multi-channel biomedical signals, and the obtaining of the biomedical signal image according to the signal curve of the biomedical signal includes: Obtain one biomedical signal image according to the signal curves of the multi-channel biomedical signals; or obtain multiple biomedical signal images according to the signal curves of the multi-channel biomedical signals.

5. The abnormal discharge detection method according to any one of claims 1 to 4, characterized in that, The motion features further include at least one of head motion features, eye motion features, mouth motion features, and nose motion features.

6. The abnormal discharge detection method according to claim 5, characterized in that, The inputting the biomedical signal features and the motion features into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test includes: Perform feature enhancement on the motion features to increase the feature dimension of the motion features and obtain target motion features; Concatenate the biomedical signal features and the target motion features to obtain concatenated features; Input the concatenated features into a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the subject under test.

7. The abnormal discharge detection method according to claim 5, wherein When the motion features include head motion features, the head motion features include head motion and no head motion, and the determination method of the head motion features includes: Determine the head position of the subject under test in the video frame of the monitoring video according to the head position detection model; Crop the video frame of the monitoring video based on the head position to obtain the head monitoring video of the subject under test; Input the head monitoring video into a pre-trained head motion detection model. When the output result of the head motion detection model indicates head motion, determine the head motion feature as head motion; otherwise, determine the head motion feature as no head motion; Wherein, the training label of the head motion detection model is determined according to whether the head of the object in the training video moves and whether there are facial expression changes.

8. The abnormal discharge detection method according to claim 5, wherein, When the motion features include mouth motion features and / or eye motion features, the mouth motion features include mouth motion and no mouth motion, the eye motion features include left eye motion features and right eye motion features, the left eye motion features include left eye motion and no left eye motion, the right eye motion features include right eye motion and no right eye motion, and the determination methods of the mouth motion features and the eye motion features include: Determine the head position of the subject under test in the video frame of the monitoring video according to the head position detection model; Detect the facial key points of the subject under test based on the head position to obtain the mouth key points, left eye key points, and right eye key points of the subject under test; Determine the first width-to-height ratio of the mouth according to the target mouth key points among the mouth key points; Determine the second aspect ratio of the left eye based on the target left-eye key points in the left-eye key points, and determine the third aspect ratio of the right eye based on the target right-eye key points in the right-eye key points; Determine the mouth movement feature according to the first change value of the first aspect ratio corresponding to adjacent frames in the surveillance video; Determine the left-eye movement feature according to the second change value of the second aspect ratio corresponding to adjacent frames in the surveillance video; Determine the right-eye movement feature according to the third change value of the third aspect ratio corresponding to adjacent frames in the surveillance video.

9. The abnormal discharge detection method according to claim 5, wherein When the movement feature includes a nose movement feature, the nose movement feature includes nose movement and no nose movement, and the determination method of the nose movement feature includes: Determine the head position of the subject in the video frame of the surveillance video according to the head position detection model; Detect the facial key points of the subject based on the head position to obtain the position of the nose key points of the subject; Determine the nose movement feature according to the change of the nose key point positions indicated by the nose key point position sequence in the video frame of the surveillance video.

10. A training method for an abnormal discharge detection model, characterized in that, Include: Obtain training samples and the sample labels corresponding to the training samples. The training samples include biomedical signal samples of the subject and surveillance video samples, and the surveillance video samples are obtained by photographing the subject when collecting the biomedical signal samples of the subject; Draw a signal curve of the biomedical signal sample with the sampling time of the biomedical signal sample as the abscissa and the sampling value of the biomedical signal sample as the ordinate, and determine the signal curve of the biomedical signal sample as the biomedical signal image sample; Input the biomedical signal image sample into the biomedical signal feature extraction network to obtain a first output; determine the sample movement feature corresponding to the surveillance video sample; Input the first output and the sample movement feature into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample; determine the loss value of the abnormal discharge detection model to be trained according to the difference between the abnormal discharge detection result of the training sample and the sample label; Iteratively train the abnormal discharge detection model to be trained according to the loss value until a preset condition is met to obtain a pre-trained abnormal discharge detection model; Among them, the sample motion feature includes the sample body motion feature, and the determination method of the sample body motion feature includes: for any monitoring video frame of the monitoring video sample, determining the head position of the object being captured in the monitoring video frame according to the head position detection model, and cropping the monitoring video frame according to the head position to obtain the body video frame corresponding to the monitoring video frame; for any body video frame, determining the reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining the frame difference corresponding to the reference video frame according to the sum of the pixel value differences at the same position between the body video frame and the reference video frame; determining the target frame difference of the body video frame according to the maximum value in the frame differences corresponding to the reference video frames, and determining the sample body motion feature according to the target frame difference.

11. The training method of the abnormal discharge detection model according to claim 10, characterized in that The sample motion feature further includes at least one of the sample head motion feature, the sample eye motion feature, the sample mouth motion feature, and the sample nose motion feature; the step of inputting the first output and the sample motion feature into the anomaly discharge detection model to be trained to obtain the anomaly discharge detection result of the training sample includes: inputting the sample motion feature into the motion feature enhancement network to obtain a second output; concatenating the first output and the second output to obtain a target concatenation result; inputting the target concatenation result into the anomaly discharge detection model to be trained, and obtaining the anomaly discharge detection result of the training sample according to the output of the anomaly discharge detection model to be trained.

12. An abnormal discharge detection device, characterized in that, including: a data acquisition module configured to acquire the biomedical signal and the monitoring video of the subject being tested, where the monitoring video is obtained by photographing the subject being tested when acquiring the biomedical signal of the subject being tested; an image generation module configured to plot the signal curve of the biomedical signal with the sampling time of the biomedical signal as the abscissa and the sampling value of the biomedical signal as the ordinate, and determine the signal curve of the biomedical signal as the biomedical signal image; a first detection module configured to input the biomedical signal image into the biomedical signal feature extraction network to obtain biomedical signal features; determine the motion features of the subject being tested according to the monitoring video; input the biomedical signal features and the motion features into the pre-trained anomaly discharge detection model to obtain the anomaly discharge detection result of the subject being tested; Among them, the motion feature includes a body motion feature, the body motion feature includes body motion and non - body motion, and the determination method of the body motion feature includes: for any monitoring video frame of the monitoring video, determining the head position of the subject in the monitoring video frame according to the head position detection model, and cropping the monitoring video frame according to the head position to obtain the body video frame corresponding to the monitoring video frame; for any one of the body video frames, determining the reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each of the reference video frames, determining the frame difference corresponding to the reference video frame according to the sum of the pixel value differences at the same position between the body video frame and the reference video frame; determining the target frame difference of the body video frame according to the maximum value in the frame differences corresponding to the reference video frames, and based on the target frame difference, determining the body motion feature of the subject.

13. A training device for an abnormal discharge detection model, characterized in that, Including: A training sample acquisition module, configured to acquire training samples and the sample labels corresponding to the training samples. The training samples include biomedical signal samples of the subject and monitoring video samples, and the monitoring video samples are obtained by photographing the subject when collecting the biomedical signal samples of the subject. An image sample generation module, configured to draw a signal curve of the biomedical signal sample with the sampling time of the biomedical signal sample as the abscissa and the sampling value of the biomedical signal sample as the ordinate, and determine the signal curve of the biomedical signal sample as the biomedical signal image sample. A second detection module, configured to input the biomedical signal image sample into the biomedical signal feature extraction network to obtain a first output; determining the sample motion feature corresponding to the monitoring video sample. Inputting the first output and the sample motion feature into the abnormal discharge detection model to be trained to obtain the abnormal discharge detection result of the training sample. A loss determination module, configured to determine the loss value of the abnormal discharge detection model to be trained according to the difference between the abnormal discharge detection result of the training sample and the sample label. An iterative training module, configured to perform iterative training on the abnormal discharge detection model to be trained according to the loss value until a preset condition is met, and obtain a pre - trained abnormal discharge detection model. Among them, the sample motion feature includes the sample body motion feature, and the determination method of the sample body motion feature includes: for any monitoring video frame of the monitoring video sample, determining the head position of the object being captured in the monitoring video frame according to the head position detection model, and cropping the monitoring video frame according to the head position to obtain the body video frame corresponding to the monitoring video frame; for any body video frame, determining the reference video frame of the body video frame according to N other body video frames adjacent to the body video frame; for each reference video frame, determining the frame difference corresponding to the reference video frame according to the sum of the pixel value differences at the same position between the body video frame and the reference video frame; determining the target frame difference of the body video frame according to the maximum value in the frame differences corresponding to the reference video frames, and determining the sample body motion feature according to the target frame difference.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 11.

15. An electronic device, characterized in that, Comprising: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method according to any one of claims 1 to 11.

Images