Method and device for detecting patient's body movement, computer-readable storage medium and device

Abstract

The embodiments of the present disclosure relate to a method and apparatus for detecting the body movement of a patient, a computer-readable storage medium and equipment, and relate to the field of artificial intelligence technology. The method for detecting the body movement of a patient comprises: collecting a monitoring video of the patient; performing frame-decoding processing on the monitoring video to obtain a set of image frames corresponding to the monitoring video; the set of image frames comprises multiple frames of images; detecting the body area and face area of ​​the patient from each frame of the image; determining the body movement detection result for the patient based on the pixel difference for the body area between each M frame of images and the position difference for the face area between each M frame of images; wherein M is an integer greater than 1. The present disclosure can output accurate body movement detection results based on the patient's body changes combined with the changes in the face position.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, the embodiments of the present disclosure relate to a method for detecting body movements of a patient, a device for detecting body movements of a patient, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.

[0003] Epilepsy is a chronic brain disease caused by abnormal discharges of brain neurons. The patient's electroencephalogram (EEG) can record waveforms related to brain activity to identify epileptic symptoms. However, during normal activities, the EEG will produce waveforms similar to epileptic seizures. Therefore, the EEG needs to be combined with the patient's monitoring video for synchronous judgment.

[0004] In the related art, human target tracking technology is generally used to extract motion information from patient monitoring videos. However, the motion information extracted by the above scheme is accompanied by a large amount of noise, and there is a problem of misconnection of key points between different human bodies. Summary of the invention

[0005] The above scheme has a low detection accuracy and cannot provide accurate diagnostic reference for doctors. In addition, it has caused a waste of medical resources and a poor medical experience for patients to a certain extent.

[0006] To this end, there is a great need for a patient's body motion detection method that can extract only the patient's motion detection information from complex video scenes with multiple people such as family members, doctors, and nurses, thereby improving the accuracy of epilepsy doctors in identifying motion artifacts and improving the doctors' diagnostic efficiency.

[0007] In this context, embodiments of the present disclosure are intended to provide a method for detecting body movements of a patient, a device for detecting body movements of a patient, a computer-readable storage medium, and an electronic device.

[0008] According to a first aspect of the embodiments of the present disclosure, a method for detecting body movement of a patient is provided, comprising:

[0009] Collect monitoring videos of patients;

[0010] Performing frame de-framing processing on the monitoring video to obtain an image frame set corresponding to the monitoring video; the image frame set includes multiple frames of images;

[0011] Detecting the body region and face region of the patient from each frame of the image;

[0012] The body movement detection result for the patient is determined based on the pixel difference for the body area between each M frames of images and the position difference for the face area between each M frames of images; wherein M is an integer greater than 1.

[0013] In an optional implementation, performing frame-decoding on the monitoring video to obtain a set of image frames corresponding to the monitoring video includes:

[0014] Divide the monitoring video into N video segments, where N is an integer greater than 1;

[0015] De-frame processing is performed on each of the video clips to obtain N image frame subsets corresponding to the N video clips.

[0016] In an optional implementation, detecting the patient's body region from each frame of the image includes:

[0017] Call the pre-trained patient target detection model;

[0018] Marking the coordinates of the patient's body annotation box from each frame of the image in each of the image frame subsets using the pre-trained patient target detection model;

[0019] The coordinates of the marked body annotation frames are merged to obtain the body region of the patient in each frame of the image.

[0020] In an optional implementation, the pre-trained patient target detection model is trained in the following manner:

[0021] Acquire training samples; the training samples include monitoring image samples of each patient sample and annotation labels corresponding to the monitoring image samples, and the annotation labels are used to characterize the actual body annotation frame of each patient sample;

[0022] Inputting the training sample into the patient target detection model to be trained, and obtaining a predicted body annotation box for the patient sample according to the output of the patient target detection model to be trained;

[0023] Determining a loss value of the patient target detection model to be trained according to a position difference between the predicted body annotation box and the actual body annotation box;

[0024] The patient target detection model to be trained is iteratively trained according to the loss value to obtain the trained patient target detection model.

[0025] In an optional implementation, the iterative training of the patient target detection model to be trained according to the loss value to obtain the trained patient target detection model includes:

[0026] According to the loss value, updating the model parameters of the patient target detection model to be trained based on a gradient update algorithm;

[0027] When the loss value satisfies a preset convergence condition, the updating of the model parameters is stopped to obtain the trained patient target detection model.

[0028] In an optional implementation, detecting the patient's face region from each frame of the image includes:

[0029] Calling a pre-trained patient face detection model, and marking a face annotation frame from each frame image in each of the image frame subsets by using the pre-trained patient face detection model;

[0030] Calling a pre-trained facial key point detection model, and annotating the facial key point coordinates in the face annotation frame by using the trained facial key point detection model;

[0031] The coordinates of the facial key points in each frame of the image are merged to determine the facial region of the patient in each frame of the image.

[0032] In an optional implementation, determining the body movement detection result for the patient according to the pixel difference for the body region between each M image frames and the position difference for the face region between each M image frames includes:

[0033] Determine a body motion detection result segment for each image frame subset of the patient according to a pixel difference for the body region between each M image frame subset and a position difference for the face region between each M image frame subset;

[0034] The N body motion detection result segments corresponding to the N image frame subsets are spliced ​​together to obtain a body motion detection result for the patient.

[0035] In an optional implementation, each of the image frame subsets includes K frames of images, each frame of image corresponds to a frame sequence number, and K is greater than or equal to M;

[0036] The pixel difference for the body region between each of the M frames in each of the image frame subsets is determined in the following manner:

[0037] Arrange the K frames of images contained in each of the image frame subsets according to a specified frame sequence number arrangement method to form an image queue;

[0038] Taking each M frames of images in the image queue as an image group, calculating the pixel color difference value for the body area between two adjacent frames of images in each of the image groups, and obtaining M-1 pixel color difference values;

[0039] The M-1 pixel color difference values ​​are determined as pixel differences for the body region between each of the M image frames in each of the image frame subsets.

[0040] In an optional implementation, each of the image frame subsets includes K frames of images, each frame of image corresponds to a frame sequence number, and K is greater than or equal to M;

[0041] The position difference of the face region between each of the M frames of image in each of the image frame subsets is determined by:

[0042] Determine P core face key points from the plurality of face key points, and determine the core face key point coordinates corresponding to each core face key point in each image frame subset; P is an integer greater than or equal to 1;

[0043] Arrange the K frames of images contained in each of the image frame subsets according to a specified frame sequence number arrangement method to form an image queue;

[0044] Taking each M frames of images in the image queue as an image group, calculating the coordinate difference value for each core face key point between two adjacent frames of images in each image group, and obtaining M-1 coordinate difference value sequences; each coordinate difference value sequence is composed of P coordinate difference values ​​arranged in sequence;

[0045] The M-1 coordinate difference value sequences are determined as position differences for the face region between each of the M frames of image in each of the image frame subsets.

[0046] In an optional embodiment, each of the image frame subsets is associated with pixel color difference values, each of the image frame subsets is associated with coordinate difference value sequences, each of which is composed of P coordinate difference values ​​corresponding to P core facial key points;

[0047] The method of determining a body motion detection result segment for each image frame subset of the patient according to a pixel difference for the body region between each M image frame subset and a position difference for the face region between each M image frame subset includes:

[0048] Get the a first accumulated value of the color difference values ​​of the pixels, and performing standardization processing on the first accumulated value to obtain a first probability value; the first probability value is used to characterize the body movement probability of the patient;

[0049] Get the key points corresponding to each of the core faces second accumulated values ​​of the coordinate difference values, and performing standardization processing on each of the second accumulated values ​​to obtain each second probability value; the P second probability values ​​are used to characterize the face motion probability of the patient;

[0050] The first probability value and P second probability values ​​are spliced ​​together to obtain a body motion detection result segment for each image frame subset of the patient.

[0051] In an optional embodiment, each of the image frame subsets is associated with pixel color difference values, each of the image frame subsets is associated with coordinate difference value sequences, each of which is composed of P coordinate difference values ​​corresponding to P core facial key points;

[0052] The method of determining a body motion detection result segment for each image frame subset of the patient according to a pixel difference for the body region between each M image frame subset and a position difference for the face region between each M image frame subset includes:

[0053] From the image frame subset associated with each Filter out the key pixel color difference values ​​from the pixel color difference values, and select the key pixel color difference values ​​associated with each of the core face key points from each of the image frame subsets. Filter out the key coordinate difference values ​​from the coordinate difference values;

[0054] Obtaining a first accumulated value of the color difference values ​​of the filtered key pixels, and performing standardization processing on the first accumulated value to obtain a first probability value; the first probability value is used to characterize the body movement probability of the patient;

[0055] Obtaining the second accumulated value of the key coordinate difference value for each core facial key point that has been screened, and performing standardization processing on each of the second accumulated values ​​to obtain each second probability value; the P second probability values ​​are used to characterize the facial motion probability of the patient;

[0056] The first probability value and P second probability values ​​are spliced ​​together to obtain a body motion detection result segment for each image frame subset of the patient.

[0057] In an optional embodiment, the The key pixel color difference values ​​are screened out from the pixel color difference values, including:

[0058] Obtain a preset pixel difference threshold, and The pixel color difference values ​​are subjected to data filtering processing to obtain the key pixel color difference value.

[0059] In an optional embodiment, the method of obtaining the key points of each core face from each subset of the image frames The key coordinate difference values ​​are selected from the coordinate difference values, including:

[0060] Obtain a preset coordinate difference threshold, and calculate the coordinate difference threshold value associated with each core face key point. The coordinate difference values ​​are subjected to data filtering processing to obtain the key coordinate difference values.

[0061] In an optional embodiment, after determining the body movement detection result for the patient, the method further includes:

[0062] Acquiring biomedical signals of the patient;

[0063] The biomedical signal and the body movement detection result are input into a pre-trained abnormal brain discharge detection model, and the abnormal brain discharge detection result for the patient is obtained according to the output of the abnormal brain discharge detection model.

[0064] According to a second aspect of the present disclosure, a patient's body movement detection device is provided, comprising:

[0065] Video acquisition module, used to collect monitoring videos of patients;

[0066] A video processing module, used for performing frame de-framing processing on the monitoring video to obtain an image frame set corresponding to the monitoring video; the image frame set includes multiple frames of images;

[0067] A detection module, used for detecting the body area and face area of ​​the patient from each frame of image;

[0068] A detection result determination module is used to determine the body movement detection result for the patient based on the pixel difference for the body area between each M frame image and the position difference for the face area between each M frame image; wherein M is an integer greater than 1.

[0069] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting body movements of a patient as described in the first aspect above is implemented.

[0070] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the patient's body movement detection method described in the first aspect above by executing the executable instructions.

[0071] According to the patient's body movement detection method, patient's body movement detection device, computer-readable storage medium and electronic device of the embodiment of the present disclosure, on the one hand, it is possible to combine the pixel difference of the patient's body area and the position difference of the patient's face area to effectively detect the patient's body movement changes and capture the patient's subtle movement changes, so as to facilitate the subsequent output of body movement detection results with higher accuracy. Further, it solves the problem that the patient's body movement detection results cannot be accurately output in the related art, and it is convenient for doctors to determine whether the abnormality of brain waves is caused by the patient's movement based on the body movement detection results, so as to accurately identify the motion artifacts in the electroencephalogram waveform, improve the accuracy of epilepsy diagnosis, and alleviate the waste of medical resources to a certain extent, and ensure the patient's medical experience; on the other hand, by outputting the patient's body movement detection results with higher accuracy, the present disclosure can also provide auxiliary support for the preoperative positioning of neurological diseases such as epilepsy, improve the accuracy of preoperative positioning, and thus help to improve the subsequent success rate of surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0073] Figure 1 A schematic diagram showing a flow chart of a method for detecting a patient's body movement in an embodiment of the present disclosure is shown;

[0074] Figure 2A schematic diagram showing a flow chart of how to detect a patient's body region from each frame of an image in an embodiment of the present disclosure;

[0075] Figure 3 A schematic diagram showing a process of how to train the above patient target detection model in an embodiment of the present disclosure;

[0076] Figure 4 A schematic diagram showing the overall process of how to train a patient body detection model in an embodiment of the present disclosure;

[0077] Figure 5 A schematic diagram showing a flow chart of how to detect a patient's face region from each frame of an image in an embodiment of the present disclosure;

[0078] Figure 6 A flow chart showing how to determine the body motion detection result for a patient according to the pixel difference for the body area between each M frames of images and the position difference for the face area between each M frames of images in an embodiment of the present disclosure;

[0079] Figure 7 A flow chart showing how to determine the pixel difference for a body region between each M image frame subset in the present disclosure;

[0080] Figure 8 A schematic diagram showing a flow chart of how to determine the position difference of a face region between each M frame image in each image frame subset in an embodiment of the present disclosure;

[0081] Fig. 9 A schematic diagram of a process of determining a body motion detection result segment for each image frame subset of a patient according to a first method of determining a body motion detection result segment for each image frame subset of a patient according to a pixel difference for a body region between each M image frame subset and a position difference for a face region between each M image frame subset in an embodiment of the present disclosure;

[0082] Fig.10 Another flow chart of how to determine a body motion detection result segment for each image frame subset of a patient according to a pixel difference for a body region between each M image frame subset in each image frame subset and a position difference for a face region between each M image frame subset in an embodiment of the present disclosure is shown;

[0083] Fig.11 The overall process diagram of the method for detecting the patient's body movement in the embodiment of the present disclosure is shown;

[0084] Fig.12 A schematic diagram of a patient's body movement detection device according to an embodiment of the present disclosure is shown;

[0085] Fig.13A structural diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0086] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0087] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0088] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0089] According to an embodiment of the present disclosure, a method for detecting body movement of a patient, a device for detecting body movement of a patient, a computer-readable storage medium, and an electronic device are provided.

[0090] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0091] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0092] The inventors have found that the existing methods for manually identifying abnormal brain discharge phenomena require a large amount of manpower and have a low level of intelligence.

[0093] In view of the above content, the basic idea of ​​the present disclosure is to provide a patient's body movement detection method, a patient's body movement detection device, a computer-readable storage medium and an electronic device. On the one hand, it can combine the pixel difference of the patient's body area and the position difference of the patient's face area to effectively detect the patient's body movement changes and capture the patient's subtle movement changes, so as to facilitate the subsequent output of body movement detection results with higher accuracy. Further, it solves the problem that the patient's body movement detection results cannot be accurately output in the related art, and it is convenient for doctors to determine whether the abnormality of brain waves is caused by the patient's movement based on the body movement detection results, so as to accurately identify the motion artifacts in the electroencephalogram waveform, improve the accuracy of epilepsy diagnosis, and alleviate the waste of medical resources to a certain extent, and ensure the patient's medical experience; on the other hand, by outputting the patient's body movement detection results with higher accuracy, the present disclosure can also provide auxiliary support for the preoperative positioning of neurological diseases such as epilepsy, improve the accuracy of preoperative positioning, and thus help to improve the subsequent success rate of surgery.

[0094] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0095] Application Scenario Overview

[0096] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0097] The embodiments of the present disclosure support outputting accurate patient body movement monitoring results based on the patient's monitoring video. For example, when a suspected epileptic waveform appears in the patient's electroencephalogram (EEG), the patient's monitoring video can be collected and parsed into image frames. Thereafter, the patient's body area and face area are detected from each frame of the image. Then, the body movement detection result for the patient can be determined based on the pixel difference for the body area between each M frames of the image and the position difference for the face area between each M frames of the image. The body movement detection result can provide a high reference value for doctors, making it easier for doctors to determine whether the abnormality of brain waves is caused by the patient's movement based on the body movement detection result, thereby accurately identifying motion artifacts in the EEG waveform and improving the accuracy of epilepsy diagnosis.

[0098] Exemplary Methods

[0099] The exemplary embodiment of the present disclosure first provides a method for detecting body movement of a patient. Figure 1 FIG. 1 is a flow chart showing a method for detecting a patient's body movement in an embodiment of the present disclosure, referring to FIG. Figure 1According to an embodiment of the present disclosure, a method for detecting a patient's body movement comprises the following steps:

[0100] Step S110, collecting monitoring video of the patient.

[0101] In this step, a video acquisition device can be used to collect monitoring video S of the patient. Exemplarily, the monitoring video S can be one hour, two hours or three hours. The specific duration can be set according to actual conditions, and the present disclosure does not make any special limitation on this.

[0102] Step S120, performing frame de-framing processing on the monitoring video to obtain an image frame set corresponding to the monitoring video.

[0103] In this step, in an optional implementation, the monitoring video S can be deframed using the audio and video codec technology FFMPEG to obtain a set of image frames corresponding to the monitoring video, which can include multiple frames. Among them, deframing refers to the process of decomposing a continuous video stream into a series of separate static image frames (frames), and this process usually involves extracting each frame in the video file to facilitate subsequent image processing and analysis. Exemplarily, the frame rate during deframing can be set to 5, 10 or 15, which can be set according to actual conditions, and the present disclosure does not make any special restrictions on this.

[0104] Among them, FFMPEG is an open source audio and video framework that provides a complete set of tools and libraries for recording, converting and streaming digital audio and video data. The core functions of FFmpeg include but are not limited to:

[0105] Encoding and decoding: Support encoding and decoding of many popular audio and video formats, including but not limited to MP3, AAC, WAV, FLAC, MP4, AVI, MPEG, H.264, H.265, etc.;

[0106] Filters: Provide powerful audio and video filters, which can perform format conversion, add special effects, adjust audio and video synchronization, etc.

[0107] Streaming media: Supports real-time streaming protocols such as RTSP, RTMP, HTTP Live Streaming (HLS), etc.

[0108] Recording and playback: Ability to record audio and video streams from different sources, and also supports playback function;

[0109] Multi-platform compatibility: FFmpeg can run on multiple operating systems, including Linux, Windows, macOS, iOS, and Android.

[0110] In another optional implementation, the monitoring video S can be segmented by time t (for example, t is 10 seconds, and every 10 seconds is a segment) using the audio and video codec technology FFMPEG to obtain N (N is an integer greater than 1) video segments, which can be represented as follows: Afterwards, the audio and video codec technology FFMPEG can be used to defram each video clip to obtain the image frame subset corresponding to each video clip, and finally obtain N image frame subsets corresponding to the above N video clips, and the N image frame subsets are included in the above image frame set.

[0111] Step S130, detecting the patient's body area and face area from each frame of image.

[0112] In this step, after obtaining multiple frames of images corresponding to the monitoring video, the body area and face area of ​​the patient can be detected from each frame of the image.

[0113] The following first describes the process of how to detect the patient's body area from each frame of the image in the present disclosure, referring to Figure 2 , Figure 2 The flowchart of how to detect the patient's body area from each frame of image in the embodiment of the present disclosure is shown, including steps S201 to S203:

[0114] In step S201, a pre-trained patient target detection model is called.

[0115] In this step, you can call the pre-trained patient target detection model. Figure 3 , Figure 3 The flowchart of how to train the patient target detection model in the embodiment of the present disclosure is shown, including steps S301 to S303:

[0116] In step S301, a training sample is obtained.

[0117] In this step, training samples can be obtained. The training samples can include monitoring image samples of each patient sample and annotation labels corresponding to the monitoring image samples. Exemplarily, the training samples can include at least 20,000 monitoring image samples of patient samples and annotation labels corresponding to each monitoring image sample to ensure that the samples are rich enough, thereby improving the model fitting effect.

[0118] The monitoring image samples include monitoring images of patients in a specific time period in the ward. The annotation labels can be used to characterize the actual body annotation frame of each patient sample. For example, a rectangular frame (i.e., the annotation frame) can be used to mark the patient's body position and record the information of the annotation frame, such as: the position coordinates of the annotation frame (such as the coordinates of the upper left corner and the lower right corner), width, height, etc., and can also include the category label in the annotation frame (for example, here it is "patient's body").

[0119] In step S302, the training sample is input into the patient target detection model to be trained, and the predicted body annotation box for the patient sample is obtained according to the output of the patient target detection model to be trained.

[0120] In this step, the above training samples can be input into the patient target detection model to be trained, so that the patient target detection model to be trained can perform target detection on the above monitoring image samples and output a predicted body annotation box for the patient samples.

[0121] In step S303, a loss value of the patient object detection model to be trained is determined according to the position difference between the predicted body annotation box and the actual body annotation box.

[0122] In this step, the loss value of the patient target detection model to be trained can be determined based on the position difference between the predicted body annotation frame and the actual body annotation frame. Exemplarily, the position difference between each coordinate of the two annotation frames can be compared, and then the position difference between multiple coordinates can be averaged to determine the loss value of the patient target detection model to be trained.

[0123] In step S304, the patient target detection model to be trained is iteratively trained according to the loss value to obtain a trained patient target detection model.

[0124] In this step, after determining the loss value of the patient target detection model to be trained, the model parameters of the patient target detection model to be trained can be updated based on the gradient update algorithm according to the loss value. Then, when the loss value meets the preset convergence condition (for example, the loss value is less than the first preset threshold), the model parameters can be stopped from being updated to obtain a trained patient target detection model (which can be referred to as Model).

[0125] In this disclosure The model can filter out regional noise information from irrelevant people such as family members and nurses in the monitoring video and obtain body area information only for the patient himself.

[0126] refer to Figure 4 , Figure 4The overall process diagram of how to train a patient body detection model in the embodiment of the present disclosure is shown, including steps S401 to S408:

[0127] In step S401, start;

[0128] In step S402, a training sample is obtained, where the training sample includes a monitoring image sample of each patient sample and an annotation label corresponding to the monitoring image sample, where the annotation label is used to represent an actual body annotation frame of each patient sample;

[0129] In step S403, the image size and the position of the annotation box in the training sample are optimized;

[0130] In step S404, the model parameters of the patient body detection model to be trained are initialized;

[0131] In step S405, the model is trained using the training samples, and the model parameters are updated using a gradient update algorithm according to the loss value;

[0132] In step S406, the loss value is minimized;

[0133] In step S407, it is determined whether the loss value satisfies the convergence condition;

[0134] If yes, proceed to step S408 and end;

[0135] If not, jump to step S402 to add more training samples and train again.

[0136] After obtaining the trained patient target detection model, you can then refer to Figure 2 In step S202, the coordinates of the patient's body annotation box are marked from each frame of each image frame subset using a pre-trained patient target detection model.

[0137] In this step, the pre-trained patient target detection model can mark the coordinates of the patient's body annotation box from each frame in each image frame subset. The body annotation box can be a rectangle, and the coordinates of the body annotation box can be the coordinates of each vertex of the rectangle.

[0138] In step S203, the coordinates of the marked body annotation boxes are merged to obtain the body region of the patient in each frame of the image.

[0139] In this step, after marking the coordinates of the above body annotation box, the merging function The coordinates of the marked body annotation boxes are merged to obtain the patient's body area in each frame of the image.

[0140] The following is an explanation of how to detect the face area of ​​a patient from each frame of an image in the embodiment of the present disclosure. Figure 5 , Figure 5 The flowchart of how to detect the face area of ​​a patient from each frame of an image in the embodiment of the present disclosure is shown, including steps S501 to S503:

[0141] In step S501, a pre-trained patient face detection model is called, and a face annotation box is marked from each frame image in each image frame subset by using the pre-trained patient face detection model.

[0142] In this step, for example, similar examples can be referred to above. Figure 3 The training process of the patient body detection model shown in the figure first obtains training samples, the training samples include monitoring image samples of each patient sample and annotation labels corresponding to the monitoring image samples (the annotation labels are used to characterize the actual face annotation frames of each patient sample), then the training samples are input into the patient face detection model to be trained, and the predicted face annotation frames for the patient samples are obtained according to the output of the patient face detection model to be trained. Next, the loss value of the patient face detection model to be trained can be determined according to the position difference between the predicted face annotation frame and the actual face annotation frame. Finally, the model parameters of the patient face detection model to be trained can be updated based on the gradient update algorithm. When the loss value meets the training termination condition (for example, the loss value is less than the second preset threshold), the above model parameters can be stopped from being updated to obtain a trained patient face detection model ( Model).

[0143] Afterwards, the trained patient face detection model can be called to mark the face annotation frame from each frame in each image frame subset using the trained face detection model. The model can accurately locate the patient's facial annotation frame, thereby reducing the adverse impact of noise data in the video on subsequent detection results.

[0144] In step S502, a pre-trained facial key point detection model is called, and the facial key point coordinates are annotated from the face annotation frame by the trained facial key point detection model.

[0145] In this step, a trained facial key point detection model can be called. Exemplarily, the facial key point detection model can be a Skps model. Then, the facial key point coordinates are marked from each frame image in each image frame subset through the above pre-trained facial key point detection model.

[0146] Exemplarily, the Skps model in the present disclosure can mark 98 facial key points from each frame of image. Exemplarily, the facial key points can be eyebrows (may include eyebrow beginnings, eyebrow peaks, eyebrow tails, etc.), eyes (may include inner eye corners, outer eye corners, upper eyelid edges, lower eyelid edges, etc.), noses (may include nasal root, nose bridge, nose tip, nose wings, etc.), mouths (may include upper edge of mouth, lower edge of mouth, corners of mouth, etc.), facial contours (may include chin vertex, end points on both sides of jawline, cheeks, etc.).

[0147] In step S503, the coordinates of the facial key points in each frame of image are merged to determine the facial region of the patient in each frame of image.

[0148] In this step, you can use the merge function The coordinates of the facial key points in each of the above-mentioned frames are merged to determine the facial area of ​​the patient in each frame.

[0149] After determining the patient's body area and the patient's face area in each frame image, you can then refer to Figure 1 In step S140, the body movement detection result for the patient is determined based on the pixel difference for the body area between each M frame image and the position difference for the face area between each M frame image.

[0150] In this step, refer to Figure 6 , Figure 6 A flow chart showing how to determine the body motion detection result for a patient according to the pixel difference for the body area between each M frames of images and the position difference for the face area between each M frames of images in an embodiment of the present disclosure includes steps S601 and S602:

[0151] In step S601, a body motion detection result segment for each image frame subset of the patient is determined based on the pixel difference for the body area between each M frame image in each image frame subset, and the position difference for the face area between each M frame image in each image frame subset.

[0152] In this step, M is an integer greater than 1, each image frame subset contains K frames of images, K is greater than or equal to M, and in the following embodiments, K is 12, that is, each image frame subset contains 12 frames of images (each frame of image corresponds to a frame number, and the frame numbers are arranged in ascending order of values, for example: the initial state can be arranged from the 1st frame to the 12th frame), and M is taken as 2 as an example for illustration.

[0153] The following first describes a specific implementation of how to determine the pixel difference for the body area between each M frame image in each image frame subset in the present disclosure:

[0154] refer to Figure 7 , Figure 7 A flow chart showing how to determine the pixel difference for the body area between each M frame image in each image frame subset in the present disclosure includes steps S701 to S703:

[0155] In step S701, K frames of images contained in each image frame subset are arranged according to a specified frame sequence number arrangement to form an image queue.

[0156] In this step, preferably, when M is 2, the above-specified frame number arrangement may be: 1st frame - 1st last frame - 2nd frame - 2nd last frame - ... K / 2th frame - (K / 2)+1st frame. Still taking the above-specified image frame subset containing 12 frames as an example, the image queue formed according to the above-specified frame number arrangement may be: 1st frame - 12th frame - 2nd frame - 11th frame - 3rd frame - 10th frame - 4th frame - 9th frame - 5th frame - 8th frame - 6th frame - 7th frame.

[0157] By arranging the images according to the specified frame number, a larger image frame interval can be ensured, thereby avoiding the problem of being unable to identify subtle movements when performing subsequent body movement detection directly based on two consecutive frames of images, thereby improving the accuracy of subsequent movement detection results.

[0158] Optionally, when M takes a value greater than 2, for example 3, the above-specified frame number arrangement may also be: 1st frame - 2nd frame - 3rd frame - ... Kth frame, for example: 1st frame - 2nd frame - 3rd frame - ... 12th frame, which can be set according to actual conditions, and the present disclosure does not make any special limitation on this.

[0159] In step S702, every M frames of images in the image queue are taken as an image group, and the pixel color difference value for the body area between two adjacent frames of images in each image group is calculated to obtain M-1 pixel color difference values.

[0160] In this step, when M is 2, the above-mentioned image queue can be divided into 6 image groups. The first image group is: the 1st frame to the 12th frame; the second image group is: the 2nd frame to the 11th frame; the third image group is: the 3rd frame to the 10th frame; the fourth image group is: the 4th frame to the 9th frame; the fifth image group is: the 5th frame to the 8th frame; the sixth image group is: the 6th frame to the 7th frame.

[0161] Therefore, the image background difference method can be used to calculate the pixel color difference value for the body area between two adjacent frames in each image group, that is, the pixel RGB difference value between the two body areas. Then, the first frame and the twelfth frame can calculate a pixel color difference value, the second frame and the eleventh frame can calculate a pixel color difference value, the third frame and the tenth frame can calculate a pixel color difference value, the fourth frame and the ninth frame can calculate a pixel color difference value, the fifth frame and the eighth frame can calculate a pixel color difference value, and the sixth frame and the seventh frame can calculate a pixel color difference value. That is, for each image group, a pixel color difference value can be calculated.

[0162] When M takes a value greater than 2, for example, when M takes a value of 3, still taking K taking a value of 12 as an example, the image queue formed according to the specified frame number arrangement method can be: 1st frame - 2nd frame - 3rd frame - ... 12th frame, thus, the above image queue can be divided into 4 image groups, the 1st image group is: 1st frame - 2nd frame - 3rd frame; the second image group is: 4th frame - 5th frame - 6th frame; the third image group is: 7th frame - 8th frame - 9th frame; the fourth image group is: 10th frame - 11th frame - 12th frame. Thus, for each image group, the pixel color difference values ​​for the body region between two adjacent frames of images can be calculated respectively. For example, for the first image group, the pixel color difference values ​​for the body region between the first frame and the second frame can be calculated, and the color difference values ​​for the body region between the second frame and the third frame can be calculated. For the second image group, the pixel color difference values ​​for the body region between the fourth frame and the fifth frame can be calculated, and the color difference values ​​for the body region between the fifth frame and the sixth frame can be calculated. Thus, for each image group, 2 pixel color difference values ​​can be obtained. In step S703, M-1 pixel color difference values ​​are determined as pixel differences for the body region between each M frame of images in each image frame subset.

[0163] In this step, referring to the explanation of the above steps, it can be known that when M is 2, one pixel color difference value can be calculated for each image group, that is, one pixel difference can be calculated for each 2 image frames for the body area. Therefore, for each image frame subset (including 6 image groups), 6 pixel color difference values, that is, 6 pixel differences, can be calculated.

[0164] When M is 3, two pixel color difference values ​​can be calculated for each image group, that is, two pixel differences can be calculated for each image group. Thus, for each image frame subset (including 4 image groups), pixel color difference values, that is, 8 pixel differences.

[0165] By calculating the pixel differences for the body region between every M frames of images, the present disclosure is able to capture large body movements of the patient.

[0166] The following is a description of a specific implementation of how to determine the position difference of the face area between each M frame image in each image frame subset in the present disclosure:

[0167] refer to Figure 8 , Figure 8 A flow chart showing how to determine the position difference of the face area between each M frame image in each image frame subset in the embodiment of the present disclosure includes steps S801 to S804:

[0168] In step S801, P core face key points are determined from a plurality of face key points, and the core face key point coordinates corresponding to each core face key point in each image frame subset are determined.

[0169] In this step, P (P is an integer greater than or equal to 1) core face key points can be determined from multiple face key points. For example, taking P as 4 as an example, the core face key points can be the left eye, the right eye, the nose and the mouth, which can be set according to the actual situation. The present disclosure does not make any special restrictions on this. Afterwards, the core face key point coordinates corresponding to each core face key point in each frame image in each image frame subset can be determined.

[0170] By determining the core facial key points of the left eye, right eye, nose and mouth, on the one hand, the present disclosure can appropriately reduce the amount of data to be processed and improve the data processing speed without affecting the accuracy of subsequent body movement detection results. On the other hand, the present disclosure can judge whether the patient is blinking, closing eyes and other movements based on the coordinate difference of the left eye and the right eye; judge whether the patient is making slight head movements based on the coordinate difference of the nose; judge whether the patient is talking, chewing and other movements based on the coordinate difference of the mouth, thereby accurately analyzing whether the patient has body movements from multiple dimensions.

[0171] In step S802, the K image frames contained in each image frame subset are arranged according to a specified frame sequence number arrangement to form an image queue.

[0172] In this step, referring to the relevant explanation of the above step S701, when M is 2, the image queue formed according to the above-specified frame number arrangement method can be: 1st frame - 12th frame - 2nd frame - 11th frame - 3rd frame - 10th frame - 4th frame - 9th frame - 5th frame - 8th frame - 6th frame - 7th frame.

[0173] When M takes a value greater than 2, for example, 3, the arrangement of the frame numbers specified above may also be: the 1st frame - the 2nd frame - the 3rd frame - ... the 12th frame.

[0174] In step S803, every M frames of images in the image queue are taken as an image group, and the coordinate difference value for each core facial key point between two adjacent frames of images in each image group is calculated to obtain M-1 coordinate difference value sequences.

[0175] In this step, referring to the relevant explanation of the above step S702, when M is 2, the above image queue can be divided into 6 image groups, the first image group is: the 1st frame to the 12th frame; the second image group is: the 2nd frame to the 11th frame; the third image group is: the 3rd frame to the 10th frame; the fourth image group is: the 4th frame to the 9th frame; the fifth image group is: the 5th frame to the 8th frame; the sixth image group is: the 6th frame to the 7th frame.

[0176] Therefore, the key point difference algorithm can be used to calculate the coordinate difference value of each core facial key point between two adjacent frames in each image group to obtain M-1 coordinate difference value sequences. Given that the number of core facial key points is P, each coordinate difference value sequence is composed of P coordinate difference values. Referring to the relevant explanation of the above step S801, it can be seen that each coordinate difference value sequence contains 4 coordinate difference values ​​corresponding to the left eye, right eye, nose and mouth, respectively.

[0177] Exemplarily, the core facial key point is taken as the "nose" as an example for explanation, and the "nose" may include components such as the nose tip, nose wings, and nose bridge. Thus, the coordinate distance between the coordinates of each component in the previous frame image and the coordinates of each component in the next frame image can be calculated (exemplarily, the coordinate distance can be Euclidean distance, cosine distance, etc., which can be set according to actual conditions, and the present disclosure does not make special restrictions on this). After that, the coordinate distances of the above-mentioned multiple components are weighted (the specific weights can be set according to actual conditions, and the present disclosure does not make special restrictions on this) to obtain the coordinate difference value corresponding to the core facial key point "nose".

[0178] Then the first frame and the twelfth frame can calculate a coordinate difference value sequence, the second frame and the eleventh frame can calculate a coordinate difference value sequence, the third frame and the tenth frame can calculate a coordinate difference value sequence, the fourth frame and the ninth frame can calculate a coordinate difference value sequence, the fifth frame and the eighth frame can calculate a coordinate difference value sequence, and the sixth frame and the seventh frame can calculate a coordinate difference value sequence. That is, for each image group, a coordinate difference value sequence can be calculated.

[0179] When M takes a value greater than 2, for example, when M takes a value of 3, still taking K taking a value of 12 as an example, the image queue formed according to the specified frame number arrangement method can be: 1st frame - 2nd frame - 3rd frame - ... 12th frame, thus, the above image queue can be divided into 4 image groups, the 1st image group is: 1st frame - 2nd frame - 3rd frame; the second image group is: 4th frame - 5th frame - 6th frame; the third image group is: 7th frame - 8th frame - 9th frame; the fourth image group is: 10th frame - 11th frame - 12th frame. Thus, for each image group, the coordinate difference value for each core face key point between two adjacent frames can be calculated respectively. For example, for the first image group, the coordinate difference value for each core face key point between the first frame and the second frame can be calculated to obtain a coordinate difference value sequence for P core face key points between the first frame and the second frame, and the coordinate difference value for each core face key point between the second frame and the third frame can be calculated to obtain a coordinate difference value sequence for P core face key points between the second frame and the third frame; for the second image group, the coordinate difference value for each core face key point between the fourth frame and the fifth frame can be calculated to obtain a coordinate difference value sequence for P core face key points between the fourth frame and the fifth frame, and the coordinate difference value for each core face key point between the fifth frame and the sixth frame can be calculated to obtain a coordinate difference value sequence for P core face key points between the fifth frame and the sixth frame. Thus, for each image group, two coordinate difference value sequences can be obtained.

[0180] In step S804, M-1 coordinate difference value sequences are determined as position differences for the face region between each M frame image in each image frame subset.

[0181] In this step, referring to the explanation of the above steps, it can be known that when M is 2, a coordinate difference value sequence can be calculated for each image group, that is, a position difference can be calculated for the face area for every 2 frames of images. Therefore, for each subset of image frames, 6 coordinate difference value sequences can be calculated (each coordinate difference value sequence contains P coordinate difference values ​​corresponding to P dimensions), that is, 6 position differences.

[0182] When M is 3, two coordinate difference value sequences can be calculated for each image group, that is, two position differences can be calculated for each image group. Therefore, for each image frame subset (including 4 image groups), it can be calculated to obtain A sequence of coordinate difference values, i.e. 8 position differences.

[0183] By calculating the position difference of the face area between each M frames of images, the present invention can capture the patient's eye closure, blinking, subtle head movement, talking, chewing and other mouth movements and subtle facial movements.

[0184] In summary, each image frame subset is associated pixel color difference values, and, associated coordinate difference value sequences, each coordinate difference value sequence is composed of P coordinate difference values ​​corresponding to P core face key points.

[0185] The following is a specific implementation of how to determine the body motion detection result segment for each image frame subset of the patient according to the pixel difference for the body area between each M frame image in each image frame subset, and the position difference for the face area between each M frame image in each image frame subset in step S601:

[0186] In a first optional implementation, reference Fig. 9 , Fig. 9 A flowchart of a first method of determining a body motion detection result segment for each image frame subset of a patient according to a pixel difference for a body region between each M image frame subset in each image frame subset and a position difference for a face region between each M image frame subset in an embodiment of the present disclosure is shown, comprising steps S901 to S903:

[0187] In step S901, obtain The first accumulated value is obtained by normalizing the first accumulated value to obtain a first probability value.

[0188] In this step, you can first obtain The first accumulated value of the pixel color difference value is then normalized using the normalization function The first accumulated value is normalized from 0 to 1 to obtain a first probability value, which can be used to characterize the probability of the patient performing large-scale body movements.

[0189] Among them, 0-1 normalization (also known as minimum-maximum normalization or Min-Max scaling) is a data preprocessing technique used to scale the numerical features in the data set to a fixed interval between 0 and 1. This normalization method ensures that all features are on the same scale by adjusting the range of the original data values.

[0190] In step S902, the corresponding key points of each core face are obtained. The second accumulated values ​​of the coordinate difference values ​​are obtained, and each second accumulated value is standardized to obtain each second probability value.

[0191] In this step, the corresponding key points of each core face can be obtained. The second accumulated value of the coordinate difference value can then be used to normalize the The second accumulated value is normalized from 0 to 1 to obtain each second probability value.

[0192] Thus, corresponding to the above-mentioned P core facial key points, P second probability values ​​can be calculated. Exemplarily, taking P as 4 (corresponding to the left eye, right eye, nose, and mouth) as an example, the above-mentioned P second probability values ​​can be called second probability value A, second probability value B, second probability value C, and second probability value D, wherein the second probability value A and the second probability value B represent the probability of the patient performing movements such as blinking and closing eyes, the second probability value C represents the probability of the patient performing subtle head movements, and the second probability value D represents the probability of the patient performing movements such as speaking and chewing.

[0193] In step S903, the first probability value and the P second probability values ​​are concatenated to obtain a body motion detection result segment for each image frame subset of the patient.

[0194] In this step, the first probability value and the P second probability values ​​may be arranged in order to be spliced ​​into a probability sequence to obtain a body movement detection result segment for each image frame subset of the patient.

[0195] In a second alternative embodiment, reference Fig.10 , Fig.10 Another flow chart of how to determine a body motion detection result segment for each image frame subset of a patient according to the pixel difference for the body area between each M image frames in each image frame subset and the position difference for the face area between each M image frames in each image frame subset in the embodiment of the present disclosure is shown, including steps S1001 to S1004:

[0196] In step S1001, from the image frames associated with each image frame subset Filter out the key pixel color difference values ​​from the pixel color difference values, and, from the associated pixel values ​​for each core face key point in each image frame subset, The key coordinate difference values ​​are selected from the coordinate difference values.

[0197] In this step, the preset pixel difference threshold v1 can be obtained and used The comparison function converts the above Each pixel color difference value is compared with the pixel difference threshold v1, and the pixel color difference value less than the pixel difference threshold v1 can be filtered to obtain the key pixel color difference value.

[0198] And, the preset coordinate difference threshold v2 can be obtained, using The comparison function associates each of the above core face key points with The coordinate difference values ​​are compared with the coordinate difference threshold v2. For the coordinate difference values ​​less than the coordinate difference threshold v2, filtering processing can be performed to obtain a key coordinate difference value sequence.

[0199] In step S1002, a first accumulated value of the screened key pixel color difference values ​​is obtained, and the first accumulated value is standardized to obtain a first probability value.

[0200] In this step, the first accumulated value of the color difference value of the selected key pixels can be obtained, and then the normalization function The first accumulated value is normalized from 0 to 1 to obtain a first probability value, which can be used to characterize the probability of the patient performing large-scale body movements.

[0201] In step S1003, the second accumulated value of the key coordinate difference value for each core facial key point that is screened out is obtained, and each second accumulated value is normalized to obtain each second probability value.

[0202] In this step, the second accumulated value of the key coordinate difference value for each core face key point can be obtained, and then the normalization function can be used to The second accumulated value is normalized from 0 to 1 to obtain each second probability value.

[0203] Thus, corresponding to the above-mentioned P core facial key points, P second probability values ​​can be calculated, and the P second probability values ​​are used to characterize the probability of facial movement of the patient. Exemplarily, still taking P as 4 (corresponding to the left eye, right eye, nose, and mouth) as an example, the above-mentioned P second probability values ​​can be called second probability value A, second probability value B, second probability value C, and second probability value D, wherein the second probability value A and the second probability value B represent the probability of the patient performing movements such as blinking and closing eyes, the second probability value C represents the probability of the patient performing subtle head movements, and the second probability value D represents the probability of the patient performing movements such as speaking and chewing.

[0204] In step S1004, the first probability value and the P second probability values ​​are spliced ​​together to obtain a body motion detection result segment for each image frame subset of the patient.

[0205] In this step, the first probability value and the P second probability values ​​may be arranged in order to be spliced ​​into a probability sequence to obtain a body movement detection result segment for each image frame subset of the patient.

[0206] After obtaining the body motion detection results corresponding to each image frame subset, you can then refer to Figure 6In step S602, N body motion detection result segments corresponding to the N image frame subsets are spliced ​​to obtain a body motion detection result for the patient.

[0207] In this step, after obtaining the body motion detection result segments corresponding to each image frame subset, the N body motion detection result segments corresponding to the above N image frame subsets may be spliced ​​to obtain the body motion detection results for the patient.

[0208] Exemplarily, assuming that the body motion detection result segment corresponding to the first image frame subset can be expressed as , the body motion detection result fragment corresponding to the second image frame subset can be expressed as , ..., the body motion detection result corresponding to the Nth image frame subset can be expressed as , then the body movement detection result for the patient can be expressed as .

[0209] After obtaining the body movement detection results for the patient, the present disclosure can also obtain the patient's biomedical signal (the biomedical signal can be a monitoring video of the patient collected at the same time as the above-mentioned monitoring video), wherein the biomedical signal is an active signal spontaneously generated by the physiological process of the living body, which can more accurately reflect the physiological state or physical state of the person, so that the biomedical signal can be used to determine whether the living body is abnormal. Exemplarily, the above-mentioned biomedical signals may include any one or more of the following: physiological signals such as electrocardiogram signals, electroencephalogram signals, electromyogram signals, electrooculogram signals, and electrogastric signals, or may also include non-electrical signals such as body temperature, blood pressure, pulse, and respiration, which can be set according to actual conditions, and the present disclosure does not make special limitations on this.

[0210] Among them, the ECG signal can be an electrocardiogram collected by a multi-lead electrocardiogram machine. An electrocardiogram (ECG) refers to a graph of various forms of potential changes drawn from the body surface by an electrocardiograph during each cardiac cycle, with the pacemaker, atrium, and ventricle being excited successively, accompanied by changes in the electrocardiogram bioelectricity. The electrocardiogram is an objective indicator of the occurrence, propagation, and recovery process of cardiac excitement.

[0211] EEG signals can be electroencephalograms collected by multi-channel electroencephalogram machines. Electroencephalograms (EEGs) are graphs obtained by amplifying and recording the spontaneous bioelectric potential of the cerebral cortex from the scalp through sophisticated instruments. They are spontaneous and rhythmic electrical activities of brain cell groups recorded by electrodes. This electrical activity is a plane diagram of the relationship between the potential and time, with the potential as the vertical axis and time as the horizontal axis. The frequency (period), amplitude and phase of brain waves constitute the basic characteristics of EEGs.

[0212] Electromyography (EMG) is the temporal and spatial superposition of action potentials of motor units in numerous muscle fibers, which can be obtained by sticking electromyography sensors on the skin.

[0213] Electrooculogram (EOG) is a bioelectric signal caused by the potential difference between the cornea and retina of the eye. It is very easy to collect and can be completed with a small number of electrodes.

[0214] Electrogastrogram (EGG) is an electrical signal generated by the contraction of stomach muscles, which can be collected on the surface of the abdominal skin using electrodes.

[0215] After acquiring the patient's biomedical signals, the biomedical signals and the above-mentioned body movement detection results can be input into a pre-trained abnormal brain discharge detection model, so that the abnormal brain discharge monitoring model can output abnormal brain discharge detection results for the patient based on the above-mentioned biomedical signals and the above-mentioned body movement detection results.

[0216] Based on the body movement detection results disclosed in the present invention and the abnormal brain discharge detection results output by the above-mentioned model, doctors only need to roughly browse the key information to quickly obtain epilepsy diagnosis results for patients, thereby providing doctors with auxiliary references during diagnosis and appropriately reducing the doctors' workload.

[0217] refer to Fig.11 , Fig.11 The overall flow chart of the method for detecting the patient's body movement in the embodiment of the present disclosure is shown, including steps S1101 to S1110:

[0218] In step S1101, start;

[0219] In step S1102, the patient monitoring video is split to obtain N video clips;

[0220] In step S1103, N image frame subsets are constructed based on the N video clips;

[0221] In step S1104, the patient's body area and the patient's face area in each frame of image are detected;

[0222] In step S1105, the pixel difference for the body area between every two image frames is calculated, and the position difference for the face area between every two image frames is calculated to obtain a plurality of pixel differences and a plurality of position differences corresponding to each image frame subset;

[0223] In step S1106, the plurality of pixel differences are compared with a pixel difference threshold to screen out key pixel differences, and the position difference is compared with a position difference threshold to screen out key position differences;

[0224] In step S1107, a body motion detection result segment for each image frame subset is determined according to the key pixel difference and the key position difference;

[0225] In step S1108, the N body motion detection result segments corresponding to the N video segments are spliced ​​to obtain the body motion detection result of the patient;

[0226] In step S1109, epilepsy is determined by combining the above body movement detection results and the patient's electroencephalogram (EEG);

[0227] In step S1110, end.

[0228] Based on the above technical solutions, the present disclosure can at least achieve the following technical effects:

[0229] First, it can combine the pixel differences of the patient's body area and the position differences of the patient's face area to effectively detect the patient's body movement changes and capture the patient's subtle movement changes, so as to facilitate the subsequent output of accurate body movement detection results;

[0230] Second, it solves the problem that the related technology cannot accurately output the patient's body movement detection results, making it easier for doctors to determine whether the abnormal brain wave is caused by the patient's movement based on the body movement detection results, thereby accurately identifying the motion artifacts in the EEG waveform, improving the accuracy and efficiency of epilepsy diagnosis, thereby alleviating the waste of medical resources to a certain extent and ensuring the patient's medical experience;

[0231] Third, by outputting the patient's body movement detection results with high accuracy, the present disclosure can also provide auxiliary support for preoperative localization of neurological diseases such as epilepsy, improve the accuracy of preoperative localization, and thus help improve the success rate of subsequent operations.

[0232] Exemplary Devices

[0233] After introducing the patient's body movement detection method according to the exemplary embodiment of the present disclosure, next, refer to Fig.12 A device for detecting body movements of a patient in an exemplary embodiment of the present disclosure is described.

[0234] Fig.12 The patient's body movement detection device 1200 is shown, comprising:

[0235] The video acquisition module 1210 is used to acquire monitoring videos of patients;

[0236] The video processing module 1220 is used to perform frame de-framing on the monitoring video to obtain an image frame set corresponding to the monitoring video; the image frame set includes multiple frames of images;

[0237] A detection module 1230, configured to detect a body region and a face region of the patient from each frame of the image;

[0238] The detection result determination module 1240 is used to determine the body movement detection result for the patient based on the pixel difference for the body area between each M frame image and the position difference for the face area between each M frame image; wherein M is an integer greater than 1.

[0239] In an optional implementation, the video processing module 1220 performs frame de-framing on the monitoring video to obtain a set of image frames corresponding to the monitoring video, including:

[0240] Divide the monitoring video into N video segments, where N is an integer greater than 1;

[0241] De-frame processing is performed on each of the video clips to obtain N image frame subsets corresponding to the N video clips.

[0242] In an optional implementation, the detection module 1230 detects the patient's body region from each frame of the image, including:

[0243] Call the pre-trained patient target detection model;

[0244] Marking the coordinates of the patient's body annotation box from each frame of the image in each of the image frame subsets using the pre-trained patient target detection model;

[0245] The coordinates of the marked body annotation frames are merged to obtain the body region of the patient in each frame of the image.

[0246] In an optional implementation, the patient's body movement detection device 1200 further includes a model training module 1050, wherein the model training module 1250 is configured to:

[0247] The patient target detection model is trained in the following way:

[0248] Acquire training samples; the training samples include monitoring image samples of each patient sample and annotation labels corresponding to the monitoring image samples, and the annotation labels are used to characterize the actual body annotation frame of each patient sample;

[0249] Inputting the training sample into the patient target detection model to be trained, and obtaining a predicted body annotation box for the patient sample according to the output of the patient target detection model to be trained;

[0250] Determining a loss value of the patient target detection model to be trained according to a position difference between the predicted body annotation box and the actual body annotation box;

[0251] The patient target detection model to be trained is iteratively trained according to the loss value to obtain the trained patient target detection model.

[0252] In an optional implementation, the model training module 1250 iteratively trains the patient target detection model to be trained according to the loss value to obtain the trained patient target detection model, including:

[0253] According to the loss value, updating the model parameters of the patient target detection model to be trained based on a gradient update algorithm;

[0254] When the loss value satisfies a preset convergence condition, the updating of the model parameters is stopped to obtain the trained patient target detection model.

[0255] In an optional implementation, the detection module 1230 detects the patient's face region from each frame of the image, including:

[0256] Calling a pre-trained patient face detection model, and marking a face annotation frame from each frame image in each of the image frame subsets by using the pre-trained patient face detection model;

[0257] Calling a pre-trained facial key point detection model, and annotating the facial key point coordinates in the face annotation frame by using the trained facial key point detection model;

[0258] The coordinates of the facial key points in each frame of the image are merged to determine the facial region of the patient in each frame of the image.

[0259] In an optional implementation, the detection result determination module 1240 determines the body movement detection result for the patient according to the pixel difference for the body area between each M frames of images and the position difference for the face area between each M frames of images, including:

[0260] Determine a body motion detection result segment for each image frame subset of the patient according to a pixel difference for the body region between each M image frame subset and a position difference for the face region between each M image frame subset;

[0261] The N body motion detection result segments corresponding to the N image frame subsets are spliced ​​together to obtain a body motion detection result for the patient.

[0262] In an optional implementation, each of the image frame subsets includes K frames of images, each frame of image corresponds to a frame sequence number, and K is greater than or equal to M;

[0263] The pixel difference for the body region between each of the M frames in each of the image frame subsets is determined in the following manner:

[0264] Arrange the K frames of images contained in each of the image frame subsets according to a specified frame sequence number arrangement method to form an image queue;

[0265] Taking each M frames of images in the image queue as an image group, calculating the pixel color difference value for the body area between two adjacent frames of images in each of the image groups, and obtaining M-1 pixel color difference values;

[0266] The M-1 pixel color difference values ​​are determined as pixel differences for the body region between each of the M image frames in each of the image frame subsets.

[0267] In an optional implementation, each of the image frame subsets includes K frames of images, each frame of image corresponds to a frame sequence number, and K is greater than or equal to M;

[0268] The position difference of the face region between each of the M frames of image in each of the image frame subsets is determined by:

[0269] Determine P core face key points from the plurality of face key points, and determine the core face key point coordinates corresponding to each core face key point in each image frame subset; P is an integer greater than or equal to 1;

[0270] Arrange the K frames of images contained in each of the image frame subsets according to a specified frame sequence number arrangement method to form an image queue;

[0271] Taking each M frames of images in the image queue as an image group, calculating the coordinate difference value for each core face key point between two adjacent frames of images in each image group, and obtaining M-1 coordinate difference value sequences; each coordinate difference value sequence is composed of P coordinate difference values ​​arranged in sequence;

[0272] The M-1 coordinate difference value sequences are determined as position differences for the face region between each of the M frames of image in each of the image frame subsets.

[0273] In an optional embodiment, each of the image frame subsets is associated with pixel color difference values, each of the image frame subsets is associated with coordinate difference value sequences, each of which is composed of P coordinate difference values ​​corresponding to P core facial key points;

[0274] The detection result determination module 1240 determines a body motion detection result segment for each image frame subset of the patient according to a pixel difference for the body region between each M image frame subset and a position difference for the face region between each M image frame subset, including:

[0275] Get the a first accumulated value of the color difference values ​​of the pixels, and performing standardization processing on the first accumulated value to obtain a first probability value; the first probability value is used to characterize the body movement probability of the patient;

[0276] Get the key points corresponding to each of the core faces second accumulated values ​​of the coordinate difference values, and performing standardization processing on each of the second accumulated values ​​to obtain each second probability value; the P second probability values ​​are used to characterize the face motion probability of the patient;

[0277] The first probability value and P second probability values ​​are spliced ​​together to obtain a body motion detection result segment for each image frame subset of the patient.

[0278] In an optional embodiment, each of the image frame subsets is associated with pixel color difference values, each of the image frame subsets is associated with coordinate difference value sequences, each of which is composed of P coordinate difference values ​​corresponding to P core facial key points;

[0279] The detection result determination module 1240 determines a body motion detection result segment for each image frame subset of the patient according to a pixel difference for the body region between each M image frame subset and a position difference for the face region between each M image frame subset, including:

[0280] From the image frame subset associated with each Filter out the key pixel color difference values ​​from the pixel color difference values, and select the key pixel color difference values ​​associated with each of the core face key points from each of the image frame subsets. Filter out the key coordinate difference values ​​from the coordinate difference values;

[0281] Obtaining a first accumulated value of the color difference values ​​of the filtered key pixels, and performing standardization processing on the first accumulated value to obtain a first probability value; the first probability value is used to characterize the body movement probability of the patient;

[0282] Obtaining the second accumulated value of the key coordinate difference value for each core facial key point that has been screened, and performing standardization processing on each of the second accumulated values ​​to obtain each second probability value; the P second probability values ​​are used to characterize the facial motion probability of the patient;

[0283] The first probability value and P second probability values ​​are spliced ​​together to obtain a body motion detection result segment for each image frame subset of the patient.

[0284] In an optional implementation, the detection result determination module 1240 determines the number of pixels associated with each of the image frame subsets. The key pixel color difference values ​​are screened out from the pixel color difference values, including:

[0285] Obtain a preset pixel difference threshold, and The pixel color difference values ​​are subjected to data filtering processing to obtain the key pixel color difference value.

[0286] In an optional implementation, the detection result determination module 1240 selects from each of the image frame subsets for each of the core face key points. The key coordinate difference values ​​are selected from the coordinate difference values, including:

[0287] Obtain a preset coordinate difference threshold, and calculate the coordinate difference threshold value associated with each core face key point. The coordinate difference values ​​are subjected to data filtering processing to obtain the key coordinate difference values.

[0288] In an optional implementation, after determining the body movement detection result for the patient, the detection result determination module 1240 is configured to:

[0289] Acquiring biomedical signals of the patient;

[0290] The biomedical signal and the body movement detection result are input into a pre-trained abnormal brain discharge detection model, and the abnormal brain discharge detection result for the patient is obtained according to the output of the abnormal brain discharge detection model.

[0291] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the inventive implementation of the above method and will not be repeated here.

[0292] Exemplary Storage Media

[0293] The storage medium according to the exemplary embodiment of the present disclosure is described below.

[0294] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, apparatus, or device.

[0295] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0296] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0297] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0298] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (FAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0299] Exemplary Electronic Devices

[0300] refer to Fig.13 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0301] Fig.13 The electronic device 1300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0302] like Fig.13 As shown, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310), and a display unit 1340.

[0303] The storage unit stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1310 can perform the following steps: Figure 1 The method steps shown, etc.

[0304] The storage unit 1320 may include a volatile storage unit, such as a random access storage unit (RAM) 1321 and / or a cache storage unit 1322 , and may further include a read-only storage unit (ROM) 1323 .

[0305] The storage unit 1320 may also include a program / utility 1324 having a set (at least one) of program modules 1325, such program modules 1325 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0306] The bus 1330 may include a data bus, an address bus, and a control bus.

[0307] The electronic device 1300 may also communicate with one or more external devices 1400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed through an input / output (I / O) interface 1350. The electronic device 1300 also includes a display unit 1340, which is connected to the input / output (I / O) interface 1350 for display. In addition, the electronic device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 1360. As shown, the network adapter 1360 communicates with other modules of the electronic device 1300 through a bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0308] It should be noted that, although several modules or submodules of the device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0309] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0310] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for detecting body movement of a patient, characterized in that: include: Collect monitoring videos of patients; De-frame the monitoring video to obtain an image frame set corresponding to the monitoring video; the image frame set includes multiple frames of images; the image frame set includes N image frame subsets; each of the image frame subsets includes K frames of images; and the body area and face area of ​​the patient are detected from each frame of the image; Determine a body movement detection result for the patient according to a pixel difference for the body region between each M frame image and a position difference for the face region between each M frame image; wherein M is an integer greater than 1, and K is greater than or equal to M; The pixel difference for the body region between each M image frames includes M-1 pixel color difference values ​​for the body region between each M image frames in each image frame subset; the position difference for the face region between each M image frames includes M-1 coordinate difference value sequences for the face region between each M image frames in each image frame subset; each coordinate difference value sequence is composed of P coordinate difference values ​​for P core face key points; Each of the image frame subsets is associated pixel color difference values, each of the image frame subsets is associated with A sequence of coordinate difference values; Determining the body movement detection result for the patient according to the pixel difference for the body area between each M frames of images and the position difference for the face area between each M frames of images includes: Get the a first accumulated value of the color difference values ​​of the pixels, and performing standardization processing on the first accumulated value to obtain a first probability value; the first probability value is used to characterize the body movement probability of the patient; Get the key points corresponding to each of the core faces second accumulated values ​​of the coordinate difference values, and performing standardization processing on each of the second accumulated values ​​to obtain each second probability value; the P second probability values ​​are used to characterize the face motion probability of the patient; splicing the first probability value and P second probability values ​​to obtain a body movement detection result segment for each image frame subset of the patient; The N body motion detection result segments corresponding to the N image frame subsets are spliced ​​together to obtain a body motion detection result for the patient.

2. The method according to claim 1, characterized in that The performing frame de-framing on the monitoring video to obtain a set of image frames corresponding to the monitoring video includes: Divide the monitoring video into N video segments, where N is an integer greater than 1; De-frame processing is performed on each of the video clips to obtain N image frame subsets corresponding to the N video clips.

3. The method according to claim 2, characterized in that Detecting the patient's body region from each frame of the image, including: Call the pre-trained patient target detection model; Marking the coordinates of the patient's body annotation box from each frame of the image in each of the image frame subsets using the pre-trained patient target detection model; The coordinates of the marked body annotation frames are merged to obtain the body region of the patient in each frame of the image.

4. The method according to claim 3, characterized in that The pre-trained patient target detection model is trained in the following way: Acquire training samples; the training samples include monitoring image samples of each patient sample and annotation labels corresponding to the monitoring image samples, and the annotation labels are used to characterize the actual body annotation frame of each patient sample; Inputting the training sample into the patient target detection model to be trained, and obtaining a predicted body annotation box for the patient sample according to the output of the patient target detection model to be trained; Determining a loss value of the patient target detection model to be trained according to a position difference between the predicted body annotation box and the actual body annotation box; The patient target detection model to be trained is iteratively trained according to the loss value to obtain the trained patient target detection model.

5. The method according to claim 4, characterized in that The iterative training of the patient target detection model to be trained according to the loss value to obtain the trained patient target detection model includes: According to the loss value, updating the model parameters of the patient target detection model to be trained based on a gradient update algorithm; When the loss value satisfies a preset convergence condition, the updating of the model parameters is stopped to obtain the trained patient target detection model.

6. The method according to claim 2, characterized in that Detecting the patient's face region from each frame of the image, including: Calling a pre-trained patient face detection model, and marking a face annotation frame from each frame image in each of the image frame subsets by using the pre-trained patient face detection model; Calling a pre-trained facial key point detection model, and annotating the facial key point coordinates in the face annotation frame by using the trained facial key point detection model; The coordinates of the facial key points in each frame of the image are merged to determine the facial region of the patient in each frame of the image.

7. The method according to claim 1, characterized in that Each image frame in each of the image frame subsets corresponds to a frame sequence number; The pixel difference for the body region between each of the M frames in each of the image frame subsets is determined in the following manner: Arrange the K frames of images contained in each of the image frame subsets according to a specified frame sequence number arrangement method to form an image queue; Taking each M frames of images in the image queue as an image group, calculating the pixel color difference value for the body area between two adjacent frames of images in each of the image groups, and obtaining M-1 pixel color difference values; The M-1 pixel color difference values ​​are determined as pixel differences for the body region between each of the M image frames in each of the image frame subsets.

8. The method according to claim 1, characterized in that Each image frame in each of the image frame subsets corresponds to a frame sequence number; The position difference of the face region between each of the M frames of image in each of the image frame subsets is determined by: Determine P core face key points from the plurality of face key points, and determine the core face key point coordinates corresponding to each core face key point in each image frame subset; P is an integer greater than or equal to 1; Arrange the K frames of images contained in each of the image frame subsets according to a specified frame sequence number arrangement method to form an image queue; Taking each M frames of images in the image queue as an image group, calculating the coordinate difference value for each core face key point between two adjacent frames of images in each image group, and obtaining M-1 coordinate difference value sequences; each coordinate difference value sequence is composed of P coordinate difference values ​​arranged in sequence; The M-1 coordinate difference value sequences are determined as position differences for the face region between each of the M frames of image in each of the image frame subsets.

9. The method according to claim 1, characterized in that: The step of determining the body movement detection result for the patient according to the pixel difference for the body area between each M frames of images and the position difference for the face area between each M frames of images further includes: From the image frame subset associated with each Filter out the key pixel color difference values ​​from the pixel color difference values, and select the key pixel color difference values ​​associated with each of the core face key points from each of the image frame subsets. Filter out the key coordinate difference values ​​from the coordinate difference values; Obtaining a first accumulated value of the color difference values ​​of the filtered key pixels, and performing standardization processing on the first accumulated value to obtain a first probability value; the first probability value is used to characterize the body movement probability of the patient; Obtaining the second accumulated value of the key coordinate difference value for each core facial key point that has been screened, and performing standardization processing on each of the second accumulated values ​​to obtain each second probability value; the P second probability values ​​are used to characterize the facial motion probability of the patient; splicing the first probability value and P second probability values ​​to obtain a body movement detection result segment for each image frame subset of the patient; The N body motion detection result segments corresponding to the N image frame subsets are spliced ​​together to obtain a body motion detection result for the patient.

10. The method according to claim 9, characterized in that The image frame subset associated with each of the image frame subsets The key pixel color difference values ​​are screened out from the pixel color difference values, including: Obtain a preset pixel difference threshold, and The pixel color difference values ​​are subjected to data filtering processing to obtain the key pixel color difference value.

11. The method according to claim 9, characterized in that The image frame subset associated with each core face key point The key coordinate difference values ​​are selected from the coordinate difference values, including: Obtain a preset coordinate difference threshold, and calculate the coordinate difference threshold value associated with each core face key point. The coordinate difference values ​​are subjected to data filtering processing to obtain the key coordinate difference values.

12. The method according to any one of claims 1 to 11, characterized in that: After determining the body movement detection result for the patient, the method further includes: Acquiring biomedical signals of the patient; The biomedical signal and the body movement detection result are input into a pre-trained abnormal brain discharge detection model, and the abnormal brain discharge detection result for the patient is obtained according to the output of the abnormal brain discharge detection model.

13. A patient's body movement detection device, characterized in that: include: Video acquisition module, used to collect monitoring videos of patients; A video processing module, used for performing frame de-framing processing on the monitoring video to obtain an image frame set corresponding to the monitoring video; the image frame set includes multiple frames of images; the image frame set includes N image frame subsets; each of the image frame subsets includes K frames of images; A detection module, used for detecting the body area and face area of ​​the patient from each frame of image; a detection result determination module, configured to determine a body movement detection result for the patient according to a pixel difference for the body region between each M image frames and a position difference for the face region between each M image frames; wherein M is an integer greater than 1; and K is greater than or equal to M; The pixel difference for the body region between each M image frames includes M-1 pixel color difference values ​​for the body region between each M image frames in each image frame subset; the position difference for the face region between each M image frames includes M-1 coordinate difference value sequences for the face region between each M image frames in each image frame subset; each coordinate difference value sequence is composed of P coordinate difference values ​​for P core face key points; Each of the image frame subsets is associated pixel color difference values, each of the image frame subsets is associated with A sequence of coordinate difference values; the detection result determination module is further used to: Get the a first accumulated value of the color difference values ​​of the pixels, and performing standardization processing on the first accumulated value to obtain a first probability value; the first probability value is used to characterize the body movement probability of the patient; Get the key points corresponding to each of the core faces second accumulated values ​​of the coordinate difference values, and performing standardization processing on each of the second accumulated values ​​to obtain each second probability value; the P second probability values ​​are used to characterize the face motion probability of the patient; splicing the first probability value and P second probability values ​​to obtain a body movement detection result segment for each image frame subset of the patient; The N body motion detection result segments corresponding to the N image frame subsets are spliced ​​together to obtain a body motion detection result for the patient.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 12 by executing the executable instructions.

Images