Early warning method for resuscitation of anesthetized patients based on facial micro-motion change detection

Abstract

The present invention relates to an anesthetized patient resuscitation warning method based on facial micro-motion change detection, comprising: collecting and preprocessing patient facial information video data to obtain a test sample; performing face detection on the patient facial image in the test sample to find the face position; using the face feature point algorithm of the cascade regression tree model to detect 68 feature key points on the face, from which 12 feature points related to the eyes and 20 feature points related to the mouth are obtained, and the eye EAR value and the mouth MAR value are calculated according to the segmented eye and mouth areas; by calculating the awakening state evaluation value F, the awakening state is identified, and an early warning is issued according to the awakening state. The present invention obtains the patient's real-time health data by calculating and analyzing the patient's eye and mouth state data, and the nursing staff relies on the obtained health data to take care of the patient, which greatly improves the work efficiency and alleviates the problem of insufficient nursing staff to a certain extent.

Description

Technical Field

[0001] The invention relates to the technical field of image recognition, and in particular to an anesthetized patient resuscitation early warning method based on facial micro-motion change detection. Background Art

[0002] In recent years, with the continuous improvement of medical level, the continuous development of modern anesthesiology and the continuous updating of detection equipment, the application of general anesthesia has become more and more extensive and safer. Anesthesia and surgery will have an impact on the patient's physiological functions, especially the first hour after anesthesia and surgery is the period that requires the most close care. Almost all anesthesia complications that endanger the patient's life safety will appear during this period.

[0003] Traditional anesthesia patient resuscitation monitoring mainly relies on experienced doctors and nurses to monitor the patient's physical condition. There are mainly the following problems: (1) The hospital's human resources are relatively tight, and the number of medical staff in the anesthesia recovery room is insufficient. When there are many surgical patients, there is a phenomenon of delayed care; (2) Manual monitoring is highly subjective, and negligence and misjudgment are often prone to occur during the monitoring process; (3) Since manual monitoring cannot achieve real-time monitoring of each patient, monitoring and care are usually delayed, making it difficult to achieve timely detection and care of anesthesia recovery.

[0004] Traditional monitoring methods can no longer meet the needs of today's hospitals in the current complex and diverse resuscitation room environment. As a branch of artificial intelligence application, intelligent video analysis and understanding has achieved rapid development. Through video monitoring, anesthetized patients can be monitored and warned in real time. Medical staff can easily obtain the resuscitation information of each patient in the resuscitation room, provide resuscitation care in time, improve the quality of anesthesia resuscitation, reduce the probability of complications in patients, and better care for the life and health of patients.

[0005] Therefore, how to detect facial micro-motion changes in anesthetized patients has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the present invention is to provide an anesthetized patient resuscitation early warning method based on facial micro-motion change detection, which can collect the facial expressions and behavioral movements of the anesthetized patient, comprehensively judge the patient's coma resuscitation status, quickly identify the precursor behaviors of the patient's coma resuscitation and issue an early warning, thereby improving the efficiency and accuracy of anesthesia care and greatly alleviating the shortage of human resources in hospitals.

[0007] To achieve the above object, the present invention adopts the following technical solution: a method for early warning of resuscitation of anesthetized patients based on facial micro-motion change detection, the method comprising the following steps in order:

[0008] (1) Collecting and preprocessing the patient's facial information video data to obtain a test sample;

[0009] (2) Perform face detection on the patient's facial image in the test sample to find the face position;

[0010] (3) Using the facial feature point algorithm of the cascade regression tree model to detect 68 key feature points on the face, 12 feature points related to the eyes and 20 feature points related to the mouth are obtained, and the eye EAR value and mouth MAR value are calculated based on the segmented eye and mouth areas;

[0011] (4) By calculating the awakening status evaluation value F, the awakening status is identified and an early warning is issued based on the awakening status.

[0012] The step (1) specifically refers to: collecting the patient's facial information video data, and performing histogram equalization and normalization processing on the video to be tested to generate a test sample.

[0013] The step (2) specifically refers to: performing face detection on the patient's facial image, identifying the target area using a HOG-based face detection algorithm, determining that the patient's face is included in the collected and processed test samples, and then performing a positioning operation to accurately find the face position.

[0014] The step (3) specifically comprises the following steps:

[0015] (3a) From a video frame Extract the eye and mouth areas of each frame, N0 is the number of video frames; use the face feature point algorithm of the cascade regression tree model to detect 68 feature key points on the face, video frame F i It is expressed as:

[0016] F i = {P1, P2, ..., P N};P i =(p x , p y ), N = 68, 0 < i ≤ N

[0017] Among them, P i is the position of the feature point. Among the 68 feature points detected, 12 feature points are used to mark the eye area and 20 feature points are used to mark the mouth area. Among them, feature points 1 to 36 are key points of the nose and cheek, feature points 37 to 42 are key points on the left eye, feature points 43 to 48 are key points on the right eye, and feature points 49 to 68 are key points on the mouth. The left eye, right eye and mouth areas are recorded as

[0018]

[0019]

[0020]

[0021] (3b) After segmenting the eye area using the feature points at the eye position, calculate the left eye EAR value, right eye EAR value and mouth MAR value respectively:

[0022]

[0023]

[0024] Among them, P'1 to P'6 are the key points on the eyes, that is, the left eye The key point or the right eye Q1 to Q6 are the key points on the mouth. Select 6 key points, the EAR value is the aspect ratio of the eyes, and the MAR value is the aspect ratio of the mouth;

[0025] After calculating the left eye EAR value and the right eye EAR value respectively, take the average of the two to get the EAR average value;

[0026] (3c) Pre-set the EAR closed eye threshold EAR threshold , compare the average EAR obtained in step (3b) with the EAR threshold When the average EAR is less than the EAR threshold When the eyes are in a closed state, the eyes are judged to be in an open state; otherwise, the eyes are judged to be in an open state;

[0027] EAR threshold The calculation formula is as follows:

[0028] EAR threshold =EAR min +p×(EAR max -EAR min )

[0029] Among them, EAR max With EAR min are the maximum and minimum values ​​of the EAR average respectively, the proportion parameter p is 0.2, EAR max EAR min It is calculated within the sampling range from eye opening to eye closing;

[0030] Preset MAR mouth opening threshold MAR threshold , compare the MAR value obtained in step (3b) with MAR threshold Compare, when the MAR value is less than MAR thresholdWhen , the mouth is judged to be in a closed state, otherwise, the mouth is judged to be in an open state;

[0031] MAR mouth opening threshold MAR threshold The calculation formula is as follows:

[0032] MAR threshold =MAR min +q×(MAR max -MAR min )

[0033] Among them, MAR max With MAR min are the maximum and minimum values ​​of MAR respectively, the ratio parameter q is 0.5, MAR max 、MAR min It is calculated within the sampling range from opening the mouth to closing the mouth;

[0034] B represents the number of blinks, M represents the number of mouth openings, and when the average EAR is greater than EAR within a detection cycle threshold , and is greater than EAR for three consecutive frames threshold When the blink count B is increased by one; when the MAR value is greater than MAR within a detection cycle threshold , and is greater than MAR for three consecutive frames threshold When the blink count M is incremented by one.

[0035] The step (4) specifically refers to:

[0036] The awakening state evaluation value F is calculated by the following formula:

[0037] F=αV P +βV B +γV M

[0038] Where P represents the time value of a certain proportion of eyes open per unit time, the unit time is 30 seconds or 1 minute, the certain proportion is 20% or 30%; B represents the number of blinks, M represents the number of mouth openings, V represents the number of times the eyes are open, P 、V B 、V M are the evaluation quantities of the awakening state evaluation indicators P, B, and M, respectively; α, β, and γ are the awakening state evaluation indicators V P 、V B 、V M The weight value of

[0039] Set the wake-up threshold F awake and full awakening threshold F totally , the awakening state evaluation value F and the awakening threshold F awake , Full awakening threshold Ftotally Comparison: If F≤F awake , then the current person being tested is judged to be in anesthesia coma and no warning is issued; if F awake ≤F≤F totally , it is determined that the person being tested is currently in a state where he or she may wake up, and an early warning is issued to remind medical staff to pay attention to the patient's condition; if F ≥ F totally , it is determined that the person being tested has entered a fully awake state, a warning is issued, and the nursing staff is immediately notified to check on the patient's condition.

[0040] It can be seen from the above technical scheme that the beneficial effects of the present invention are as follows: First, the present invention no longer relies solely on a certain data to analyze the patient's physiological condition, but realizes the combination of eye and mouth state recognition information, so as to achieve faster and better analysis of the patient's physiological condition; by calculating and analyzing the patient's eye and mouth state data, the patient's real-time health data is obtained, and the nursing staff relies on these obtained health data to take care of the patient, which greatly improves the work efficiency and alleviates the problem of insufficient nursing staff to a certain extent. Second, the present invention integrates the obtained facial expression micro-movement and other data, comprehensively analyzes these data to obtain the patient's awakening status in the anesthesia recovery room PACU, and integrates the patient's various data together, which can more accurately and comprehensively analyze the patient's health status. Third, the present invention can provide timely warnings, and nursing staff can make comprehensive judgments with special medical equipment, and realize a more comprehensive analysis of the health status of patients in the PACU, providing objective information reference for medical staff, greatly improving work efficiency, and reducing the work pressure of nursing staff. Fourth, the present invention realizes real-time monitoring and analysis of the physiological conditions of patients in the PACU, so that when the patient is in danger, the nursing staff can respond in time, protecting the patient's life and health, avoiding the lag of the patient's awakening and the nursing staff, and reducing the work pressure of the nursing staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 This is a flow chart of the method for waking state identification in the present invention. DETAILED DESCRIPTION

[0043] like Figure 1 As shown, a method for early warning of resuscitation of anesthetized patients based on facial micro-motion change detection includes the following steps in sequence:

[0044] (1) Collecting and preprocessing the patient's facial information video data to obtain a test sample;

[0045] (2) Perform face detection on the patient's facial image in the test sample to find the face position;

[0046] (3) Using the facial feature point algorithm of the cascade regression tree model to detect 68 key feature points on the face, 12 feature points related to the eyes and 20 feature points related to the mouth are obtained, and the eye EAR value and mouth MAR value are calculated based on the segmented eye and mouth areas;

[0047] (4) By calculating the awakening status evaluation value F, the awakening status is identified and an early warning is issued based on the awakening status.

[0048] The step (1) specifically refers to: collecting the patient's facial information video data, and performing histogram equalization and normalization processing on the video to be tested to generate a test sample.

[0049] The step (2) specifically refers to: performing face detection on the patient's facial image, identifying the target area using a HOG-based face detection algorithm, determining that the patient's face is included in the collected and processed test samples, and then performing a positioning operation to accurately find the face position.

[0050] The step (3) specifically comprises the following steps:

[0051] (3a) From a video frame Extract the eye and mouth areas of each frame, N0 is the number of video frames; use the face feature point algorithm of the cascade regression tree model to detect 68 feature key points on the face, video frame F i It is expressed as:

[0052] F i = {P1, P2, ..., P N};P i =(p x , p y ), N = 68, 0 < i ≤ N

[0053] Among them, P i is the position of the feature point. Among the 68 feature points detected, 12 feature points are used to mark the eye area and 20 feature points are used to mark the mouth area. Among them, feature points 1 to 36 are key points of the nose and cheek, feature points 37 to 42 are key points on the left eye, feature points 43 to 48 are key points on the right eye, and feature points 49 to 68 are key points on the mouth. The left eye, right eye and mouth areas are recorded as

[0054]

[0055]

[0056]

[0057] (3b) After segmenting the eye area using the feature points at the eye position, calculate the left eye EAR value, right eye EAR value and mouth MAR value respectively:

[0058]

[0059]

[0060] Among them, P'1 to P'6 are the key points on the eyes, that is, the left eye The key point or the right eye Q1 to Q6 are the key points on the mouth. Select 6 key points, the EAR value is the aspect ratio of the eyes, and the MAR value is the aspect ratio of the mouth;

[0061] After calculating the left eye EAR value and the right eye EAR value respectively, take the average of the two to get the EAR average value;

[0062] The EAR average and MAR values ​​are calculated in real time and are not fixed values;

[0063] (3c) Pre-set the EAR closed eye threshold EAR threshold , compare the average EAR obtained in step (3b) with the EAR threshold When the average EAR is less than the EAR threshold When the eyes are in a closed state, the eyes are judged to be in an open state; otherwise, the eyes are judged to be in an open state;

[0064] EAR threshold The calculation formula is as follows:

[0065] EAR threshold =EAR min +p×(EAR max -EAR min )

[0066] Among them, EAR max With EAR min are the maximum and minimum values ​​of the EAR average respectively, and the proportional parameter p is 0.2. Here, EAR max EAR min It is calculated within the sampling range from eye opening to eye closing;

[0067] Preset MAR mouth opening threshold MAR threshold , compare the MAR value obtained in step (3b) with MAR threshold Compare, when the MAR value is less than MAR threshold When , the mouth is judged to be in a closed state, otherwise, the mouth is judged to be in an open state; MAR thresholdIt is a fixed value, not calculated by real-time sampling.

[0068] MAR mouth opening threshold MAR threshold The calculation formula is as follows:

[0069] MAR threshold =MAR min +q×(MAR max -MAR min )

[0070] Among them, MAR max With MAR min are the maximum and minimum values ​​of MAR respectively, and the ratio parameter q is 0.5. Here, MAR max 、MAR min It is calculated within the sampling range from opening the mouth to closing the mouth; MAR threshold It is a fixed value, not calculated by real-time sampling.

[0071] B represents the number of blinks, M represents the number of mouth openings, and when the average EAR is greater than EAR within a detection cycle threshold , and is greater than EAR for three consecutive frames threshold When the blink count B is increased by one; when the MAR value is greater than MAR within a detection cycle threshold , and is greater than MAR for three consecutive frames threshold When the blink count M is incremented by one.

[0072] like Figure 2 As shown, the step (4) specifically refers to:

[0073] The awakening state evaluation value F is calculated by the following formula:

[0074] F=αV P +βV B +γV M

[0075] Where P represents the time value of a certain proportion of eyes open per unit time, the unit time is 30 seconds or 1 minute, the certain proportion is 20% or 30%; B represents the number of blinks, M represents the number of mouth openings, V represents the number of times the eyes are open, P 、V B 、V M are the evaluation quantities of the awakening state evaluation indicators P, B, and M, respectively; α, β, and γ are the awakening state evaluation indicators V P 、V B 、V M The weight value of

[0076] Set the wake-up threshold F awake and full awakening threshold F totally, the awakening state evaluation value F and the awakening threshold F awake , Full awakening threshold F totally Comparison: If F≤F awake , then the current person being tested is judged to be in anesthesia coma and no warning is issued; if F awake ≤F≤F totally , it is determined that the person being tested is currently in a state where he or she may wake up, and an early warning is issued to remind medical staff to pay attention to the patient's condition; if F ≥ F totally , it is determined that the person being tested has entered a fully awake state, a warning is issued, and the nursing staff is immediately notified to check on the patient's condition.

[0077] In actual clinical practice, it is found that some patients in the video have been in anesthesia coma for a long time, more than half an hour or even more than an hour. Since the patient fails to wake up completely and regain self-awareness within a certain period of time, we should be highly alert to the possibility of delayed awakening and actively look for and eliminate possible causes to avoid passive waiting and delaying the diagnosis and treatment of patients. Therefore, when F≤F awake A coma time determination process will also be carried out. If the patient's coma time during the detection period is greater than or equal to 30 minutes and less than 60 minutes, the patient is determined to be in an alert state, and the system needs to issue an early warning to remind medical staff to pay attention to and observe the patient's condition in a timely manner; if the patient does not wake up for more than 60 minutes, the patient is determined to be in a high alert state, and the system will issue a warning to immediately notify the nursing staff to go and check on the patient's condition.

[0078] In summary, the present invention integrates the detection of facial eye and mouth micro-movement changes, and no longer relies solely on traditional medical devices to analyze the patient's awakening status. It realizes the fusion of facial eye and mouth micro-movement change detection information, thereby achieving faster and better analysis of the patient's awakening status; by analyzing the patient's facial micro-movements in the PACU, the patient's real-time health data is obtained, and the nursing staff relies on the obtained health data to care for the patient, which greatly improves work efficiency and alleviates the problem of insufficient nursing staff to a certain extent.

Claims

1. A method for early warning of resuscitation of anesthetized patients based on facial micro-motion change detection, the method comprising the following steps in order: (1) Collecting and preprocessing the patient's facial information video data to obtain a test sample; (2) Perform face detection on the patient's facial image in the test sample to find the face position; (3) Using the facial feature point algorithm of the cascade regression tree model to detect 68 key feature points on the face, 12 feature points related to the eyes and 20 feature points related to the mouth are obtained, and the eye EAR value and mouth MAR value are calculated based on the segmented eye and mouth areas; (4) By calculating the awakening state evaluation value F, the awakening state is identified and an early warning is issued according to the awakening state; The step (3) specifically comprises the following steps: (3a) From a video frame Extract the eye and mouth areas of each frame, N0 is the number of video frames; use the face feature point algorithm of the cascade regression tree model to detect 68 feature key points on the face, video frame F i It is expressed as: F i ={P1,P2,…,P N };P i =(p x ,p y ),N=68,0<i≤N Among them, P i is the position of the feature point. Among the 68 feature points detected, 12 feature points are used to mark the eye area and 20 feature points are used to mark the mouth area. Among them, feature points 1 to 36 are key points of the nose and cheek, feature points 37 to 42 are key points on the left eye, feature points 43 to 48 are key points on the right eye, and feature points 49 to 68 are key points on the mouth. The left eye, right eye and mouth areas are recorded as (3b) After segmenting the eye area using the feature points at the eye position, calculate the left eye EAR value, right eye EAR value and mouth MAR value respectively: Among them, P1′ to P6′ are the key points on the eyes, namely, the left eye The key point or the right eye Q1 to Q6 are the key points on the mouth. Select 6 key points, the EAR value is the aspect ratio of the eyes, and the MAR value is the aspect ratio of the mouth; After calculating the left eye EAR value and the right eye EAR value respectively, take the average of the two to get the EAR average value; (3c) Pre-set the EAR closed eye threshold EAR threshold , compare the average EAR obtained in step (3b) with the EAR threshold When the average EAR is less than the EAR threshold When the eyes are in a closed state, the eyes are judged to be in an open state; otherwise, the eyes are judged to be in an open state; EAR threshold The calculation formula is as follows: EAR threshold =EAR min +p×(EAR max -EAR min ) Among them, EAR max With EAR min are the maximum and minimum values ​​of the EAR average respectively, the proportion parameter p is 0.2, EAR max EAR min It is calculated within the sampling range from eye opening to eye closing; Preset MAR mouth opening threshold MAR threshold , compare the MAR value obtained in step (3b) with MAR threshold Compare, when the MAR value is less than MAR threshold When , the mouth is judged to be in a closed state, otherwise, the mouth is judged to be in an open state; MAR mouth opening threshold MAR threshold The calculation formula is as follows: SEA threshold =SEA min +q×(MAR max -SEA min ) Among them, MAR max With MAR min are the maximum and minimum values ​​of MAR respectively, the ratio parameter q is 0.5, MAR max 、MAR min It is calculated within the sampling range from opening the mouth to closing the mouth; B represents the number of blinks, M represents the number of mouth openings, and when the average EAR is greater than EAR within a detection cycle threshold , and is greater than EAR for three consecutive frames threshold When the blink count B is increased by one; when the MAR value is greater than MAR within a detection cycle threshold , and is greater than MAR for three consecutive frames threshold When the blink count M is incremented by one.

2. The method for early warning of anesthetized patient resuscitation based on facial micro-motion change detection according to claim 1, characterized in that: The step (1) specifically refers to: collecting the patient's facial information video data, and performing histogram equalization and normalization processing on the video to be tested to generate a test sample.

3. The method for early warning of anesthetized patient resuscitation based on facial micro-motion change detection according to claim 1, characterized in that: The step (2) specifically refers to: performing face detection on the patient's facial image, identifying the target area using a HOG-based face detection algorithm, determining that the patient's face is included in the collected and processed test samples, and then performing a positioning operation to accurately find the face position.

4. The method for early warning of anesthetized patient resuscitation based on facial micro-motion change detection according to claim 1, characterized in that: The step (4) specifically refers to: The awakening state evaluation value F is calculated by the following formula: F=αV P +βV B +γV M Where P represents the time value of a certain proportion of eyes open per unit time, the unit time is 30 seconds or 1 minute, the certain proportion is 20% or 30%; B represents the number of blinks, M represents the number of mouth openings, V represents the number of times the eyes are open, P 、V B 、V M are the evaluation quantities of the awakening state evaluation indicators P, B, and M, respectively; α, β, and γ are the awakening state evaluation indicators V P 、V B 、V M The weight value of Set the wake-up threshold F awake and full awakening threshold F totally , the awakening state evaluation value F and the awakening threshold F awake , Full awakening threshold F totally Comparison: If F≤F awake , then the current person being tested is judged to be in anesthesia coma and no warning is issued; if F awake ≤F≤F totally , it is determined that the person being tested is currently in a state where he or she may wake up, and an early warning is issued to remind medical staff to pay attention to the patient's condition; if F ≥ F totally , it is determined that the person being tested has entered a fully awake state, a warning is issued, and the nursing staff is immediately notified to check on the patient's condition.

Images