An anesthesia recovery prediction system and method based on facial features

Abstract

This invention proposes an anesthesia recovery prediction system and method based on facial features. The anesthesia recovery prediction system includes a processing device, a 3D camera, a visible light camera, and a millimeter-wave radar module. The processing device is communicatively connected to the 3D camera, the visible light camera, and the millimeter-wave radar module. Sliding tracks are set on both sides of the upper half of the hospital bed. Two 3D cameras are mounted on the sliding tracks on the left and right sides of the bed via first movable seats. The millimeter-wave radar module and the visible light camera are mounted on the sliding track on one side of the bed via second movable seats. The first and second movable seats are sequentially set on the sliding tracks. In this invention, the recovery status is analyzed layer by layer based on changes in the eyes, facial expressions, and mouth features, which reduces the workload of medical staff while meeting monitoring requirements.

Description

Technical Field

[0001] This invention relates to the field of anesthesia monitoring technology, and in particular to an anesthesia recovery prediction system and method based on facial features. Background Technology

[0002] During the recovery process after anesthesia, the residual effects of anesthetic drugs, muscle relaxants, and nerve blocking drugs have not disappeared, and the body's protective reflexes have not fully recovered, which can easily lead to complications such as airway obstruction, vomiting, and aspiration. This results in a variety of risk factors during anesthesia recovery. Among these, the first hour after surgery is the period that requires the most close monitoring, as almost all serious complications that endanger the patient's life occur during this period.

[0003] Current technology relies on professional nursing staff for anesthesia resuscitation monitoring. During monitoring, these staff must continuously patrol the ward, checking parameters such as the patient's electrocardiogram, blood pressure, and pulse until the patient's vital signs stabilize. This entire monitoring process demands a tremendous amount of effort from the professional nursing staff, and hospitals often face a shortage of such personnel, making it impossible to meet the stringent requirements for anesthesia resuscitation monitoring. Summary of the Invention

[0004] To address the technical problem of high workload and inability to guarantee monitoring quality in existing artificial anesthesia recovery monitoring technologies, the present invention proposes an anesthesia recovery prediction system based on facial features, which includes a processing device, a 3D camera, a visible light camera, and a millimeter-wave radar module. The processing device is communicatively connected to the 3D camera, the visible light camera, and the millimeter-wave radar module, respectively.

[0005] Sliding tracks are provided on both the left and right sides of the upper part of the hospital bed. Two 3D cameras are mounted on the sliding tracks on the left and right sides of the hospital bed via a first movable seat. The millimeter-wave radar module and the visible light camera are mounted on the sliding track on one side of the hospital bed via a second movable seat. The first movable seat and the second movable seat are sequentially mounted on the sliding tracks.

[0006] Preferably, the first movable seat includes a first sliding block, a first support rod, a first sliding rod, and a first rotating shaft. The first sliding block is connected to a sliding track, the first support rod is disposed above the first sliding block, the first sliding rod is retractably disposed inside the first support rod, and the 3D camera is connected to the first sliding rod through the first rotating shaft.

[0007] Preferably, the second movable seat includes a second sliding block, a second telescopic rod, a second connecting rod, a second sliding rod, a second rotating shaft, a third connecting rod, and a detection end. The second sliding block is connected to a sliding track. The second telescopic rod is disposed between the second sliding block and the second connecting rod. The second sliding rod is telescopically disposed inside the second connecting rod. The third connecting rod is connected to the second sliding rod via the second rotating shaft. The third connecting rod is connected to the detection end. A visible light camera is disposed at the center of the bottom of the detection end. The two ends of the bottom of the detection end are connected to a millimeter-wave radar module via a horizontal rotating disk and a vertical rotating disk.

[0008] Based on the above-mentioned anesthesia recovery prediction system, the anesthesia recovery prediction method based on facial features proposed in this invention specifically includes the following steps:

[0009] S1. Adjust the position of the first movable seat so that the 3D camera is located on both sides of the patient's head, and then adjust the position of the second movable seat so that it is close to the first movable seat;

[0010] S2. Automatically adjusts the positions of the 3D camera, visible light camera, and millimeter-wave radar module;

[0011] S3. Set the reference time to the current time;

[0012] S4. Calculate eye response parameters based on data from the visible light camera and millimeter-wave radar module;

[0013] S5. If the eye reaction parameter is greater than the reaction threshold, proceed to S6. If the eye reaction parameter is less than or equal to the reaction threshold, calculate the time difference between the current time and the reference time. If the time difference is greater than the first time threshold, the processing device needs to issue the first alarm to the medical staff. Otherwise, return to S4.

[0014] S6. Calculate parameters for eye change patterns based on data from the visible light camera and millimeter-wave radar module;

[0015] S7. If the eye change pattern parameter is less than the pattern threshold, proceed to S8. If the eye change pattern parameter is greater than or equal to the pattern threshold, calculate the time difference between the current time and the reference time. If the time difference is greater than the second time threshold, the processing device issues a second alarm to the medical staff. Otherwise, return to S6.

[0016] S8. Calculate the awakening parameters based on the data from the visible light camera. If the awakening parameters are less than the awakening threshold, the processing device sends a reminder message to the medical staff. At the same time, calculate the rotation angle of the patient's head based on the data from the 3D camera. If the rotation angle is greater than the angle threshold, the processing device sends a third alarm to the medical staff.

[0017] Preferably, in step S2, the second telescopic rod, the second sliding rod, and the second rotating shaft are adjusted based on the image captured by the visible light camera so that the detection end faces the patient's face. After the detection end faces the patient's face, the horizontal rotating disk and the vertical rotating disk are adjusted according to the image captured by the visible light camera so that the millimeter-wave radar module is aligned with the left and right eyes respectively. At the same time, the first sliding rod and the first rotating shaft are adjusted according to the image captured by the visible light camera so that the distance and angle between the 3D camera and the patient's head meet the set requirements.

[0018] Preferably, in step S4, the calculation process of the eye reaction parameter is as follows: the ultrasonic image of the eye is obtained based on the data from the millimeter-wave radar module; the average pupil diameter and pupil reaction speed of both eyes are calculated based on the ultrasonic image; and the eye change speed is obtained based on the image captured by the visible light camera. The eye reaction parameter is calculated as: average pupil diameter of both eyes / (pupil reaction speed × eye change speed).

[0019] Preferably, in S4, the pupillary reaction speed is the average of the average time taken for the ultrasound signal amplitude of the left eye to change from its maximum value to its minimum value, and the average time taken for the ultrasound signal amplitude of the right eye to change from its maximum value to its minimum value.

[0020] Preferably, in S4, the calculation process of the eye change speed is to use an end-to-end neural network yolov4-tiny to process the image captured by the visible light camera, obtain the left eye region and the right eye region, calculate the average time required for the left eye to complete one cycle of change and the average time required for the right eye to complete one cycle of change, and take the average of the two as the eye change speed.

[0021] Preferably, in step S6, the calculation process of the eye change pattern parameter is as follows: the ultrasonic image of the eye is acquired based on the data from the millimeter-wave radar module, and the pupil reaction speed is calculated based on the ultrasonic image. The pupil reaction speed is the average of the time taken for the ultrasonic signal amplitude of the left eye to change from the maximum value to the minimum value and the time taken for the ultrasonic signal amplitude of the right eye to change from the maximum value to the minimum value. The image captured by the visible light camera is processed using an end-to-end neural network yolov4-tiny to acquire the left eye region and the right eye region, and the time required for the left eye to complete one cycle of change and the time required for the right eye to complete one cycle of change are calculated. The average of the two is taken as the eye change speed. The eye change pattern parameter = |(pupil reaction speed / eye change speed)-1|, where |……| is the operation of taking the absolute value.

[0022] Preferably, in step S8, the calculation process of the awakening parameter involves using an end-to-end neural network yolov4-tiny to process images captured by a visible light camera, acquiring the left eye region, right eye region, infraorbital region, cheek region, and lip region. For the infraorbital region and cheek region, based on reference points, the time points at which facial changes such as muscle contraction and muscle twitching occur are determined. For the lip region, the time points at which mouth opening occurs are determined. For the left and right eye regions, the time intervals of each change cycle of the left and right eyes are acquired. Combining the time points of facial changes and mouth opening, facial changes and mouth opening are divided into each change cycle of the eyes. The total number of facial changes and mouth openings is taken as the number of expressions in each change cycle of the eyes. The average number of expressions in each change cycle of the eyes is calculated. The average time required for the left eye to complete one cycle of change and the average time required for the right eye to complete one cycle of change are calculated. The average of the two is taken as the speed of eye change. The awakening parameter = speed of eye change / average number of expressions in each change cycle of the eyes.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] By analyzing changes in the eyes, facial expressions, and mouth features, the awakening state is analyzed layer by layer, which reduces the workload of medical staff while meeting monitoring requirements. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the anesthesia recovery prediction system of the present invention;

[0026] Figure 2 This is a schematic diagram of the structure of the first movable base of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of the second movable base of the present invention.

[0028] Figure descriptions: 11. First sliding block; 12. First support rod; 13. First sliding rod; 14. First rotating shaft; 15. 3D camera; 21. Second sliding block; 22. Second telescopic rod; 23. Second connecting rod; 24. Second sliding rod; 25. Second rotating shaft; 26. Third connecting rod; 27. Detection end; 28. Visible light camera; 29. ​​Horizontal rotating disk; 30. Vertical rotating disk; 31. Millimeter-wave radar module. Detailed Implementation

[0029] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] like Figure 1As shown, the anesthesia recovery prediction system based on facial features proposed in this invention includes a processing device, a 3D camera, a visible light camera, and a millimeter-wave radar module. The processing device is communicatively connected to the 3D camera, the visible light camera, and the millimeter-wave radar module. Sliding rails are provided on both the left and right sides of the upper part of the hospital bed. Two 3D cameras are mounted on the sliding rails on the left and right sides of the bed via first movable seats. The millimeter-wave radar module and the visible light camera are mounted on the sliding rail on one side of the bed via second movable seats. The first and second movable seats are sequentially positioned on the sliding rails.

[0031] like Figure 2 As shown, the first movable base includes a first sliding block 11, a first support rod 12, a first sliding rod 13, and a first rotating shaft 14. The first sliding block 11 is connected to a sliding track. The first support rod 12 is positioned above the first sliding block 11. The first sliding rod 13 is telescopically located inside the first support rod 12 and is used to adjust the distance between the 3D camera 15 and the patient's head. The 3D camera 15 is connected to the first sliding rod 13 via the first rotating shaft 14 and can rotate ±90° vertically relative to the first sliding rod 13. The first sliding rod 13 moves under the drive of a micro motor, and the first rotating shaft 14 rotates under the drive of the micro motor. The 3D camera used is Microsoft's Kinect. By integrating the images captured by the 3D cameras on both sides, a stereoscopic image of the patient's head can be obtained, allowing for the determination of the tilt angle of the patient's head.

[0032] like Figure 3As shown, the second movable base includes a second sliding block 21, a second telescopic rod 22, a second connecting rod 23, a second sliding rod 24, a second rotating shaft 25, a third connecting rod 26, and a detection end 27. The second sliding block 21 is connected to a sliding track. The second telescopic rod 22 is disposed between the second sliding block 21 and the second connecting rod 23. The second sliding rod 24 is telescopically disposed inside the second connecting rod 23. The third connecting rod 26 is connected to the second sliding rod 24 via the second rotating shaft 25. The third connecting rod 26 can rotate ±90° in the vertical direction relative to the second sliding rod 24. The third connecting rod 26 is connected to the detection end 27. A visible light camera 28 is disposed at the center of the bottom of the detection end 27. The two ends of the bottom of the detection end 27 are connected to a millimeter-wave radar module 31 via a horizontal rotating disk 29 and a vertical rotating disk 30. The second telescopic rod 22 extends and retracts under the drive of a micro motor, the second sliding rod 24 moves under the drive of a micro motor, and the second rotating shaft 25, the horizontal rotating disk 29, and the vertical rotating disk 30 rotate under the drive of a micro motor. The second telescopic rod 22, the second sliding rod 24, and the second rotating shaft 25 cooperate to adjust the detection position of the detection end 27. The visible light camera 28 is used to capture facial images. The millimeter-wave radar module 31 can be aligned with the left and right eyes under the drive of the horizontal rotating disk 29 and the vertical rotating disk 30 to acquire ultrasound images of the eyes.

[0033] The processing device receives data from a 3D camera, a visible light camera, and a millimeter-wave radar module, and predicts anesthesia recovery based on the received data.

[0034] Based on the above-mentioned anesthesia recovery prediction system, the anesthesia recovery prediction method based on facial features specifically includes the following steps:

[0035] S1. Adjust the position of the first movable seat so that the 3D camera is located on both sides of the patient's head, and then adjust the position of the second movable seat so that it is close to the first movable seat.

[0036] S2. Automatically adjust the positions of the 3D camera, visible light camera, and millimeter-wave radar module. Specifically, the adjustment process involves adjusting the second retractable rod 22, the second sliding rod 24, and the second rotating shaft 25 based on images captured by the visible light camera, ensuring that the detection end 27 faces the patient's face. After the detection end faces the patient's face, the horizontal rotating disk 29 and the vertical rotating disk 30 are adjusted according to images captured by the visible light camera, ensuring that the millimeter-wave radar module is aligned with the left and right eyes respectively. Simultaneously, the first sliding rod 13 and the first rotating shaft 14 are adjusted according to images captured by the visible light camera, ensuring that the distance and angle between the 3D camera and the patient's head meet the set requirements.

[0037] S3. Set the reference time to the current time.

[0038] S4. Calculate eye response parameters based on data from the visible light camera and millimeter-wave radar module. The calculation process for eye response parameters involves acquiring ultrasound images of the eyes based on data from the millimeter-wave radar module, calculating the average pupil diameter and pupillary response speed of both eyes based on the ultrasound images, and the pupillary response speed being the average of the average time taken for the ultrasound signal amplitude of the left eye to change from its maximum value to its minimum value, and the average time taken for the ultrasound signal amplitude of the right eye to change from its maximum value to its minimum value. An end-to-end neural network yolov4-tiny is used to process the images captured by the visible light camera, acquiring the left and right eye regions, and calculating the average time required for the left eye to complete one cycle of change and the average time required for the right eye to complete one cycle of change. The average of these two values ​​is taken as the eye change speed. Eye response parameter = average pupil diameter of both eyes / (pupil response speed × eye change speed). Since eye changes can reflect autonomic nerve activity, eye response parameters can characterize whether the brain begins to gradually return to normal activity during anesthesia recovery.

[0039] S5. If the eye reaction parameter is greater than the reaction threshold, proceed to S6. If the eye reaction parameter is less than or equal to the reaction threshold, calculate the time difference between the current time and the reference time. If the time difference is greater than the first time threshold, it means that the time for the patient to show neural activity is greater than the empirical value, and medical staff need to observe and handle it. The processing device needs to issue the first alarm to the medical staff. The first time threshold is the length of time required for the patient's brain nerves to show activity after anesthesia, obtained from analyzing a large number of cases. Otherwise, return to S4.

[0040] S6. Calculate the eye change pattern parameters based on data from the visible light camera and millimeter-wave radar module. The calculation process for the eye change pattern parameters involves acquiring ultrasound images of the eye based on data from the millimeter-wave radar module, calculating the pupillary reaction speed based on the ultrasound images, and taking the average of the time taken for the ultrasound signal amplitude of the left eye to change from its maximum value to its minimum value and the time taken for the ultrasound signal amplitude of the right eye to change from its maximum value to its minimum value. An end-to-end neural network, yolov4-tiny, is used to process the images captured by the visible light camera, acquiring the left and right eye regions, calculating the time required for the left eye to complete one cycle of change and the time required for the right eye to complete one cycle of change, and taking the average of these two times as the eye change speed. The eye change pattern parameter is calculated as |(pupil reaction speed / eye change speed) - 1|, where |……| represents the absolute value operation.

[0041] S7. If the eye change pattern parameter is less than the pattern threshold, proceed to S8. If the eye change pattern parameter is greater than or equal to the pattern threshold, calculate the time difference between the current time and the reference time. If the time difference is greater than the second time threshold, it means that the time required for the patient's neural activity to stabilize is greater than the empirical value, and medical staff need to observe and handle the situation. The processing device issues a second alarm to the medical staff. The second time threshold is the length of time required for the patient's brain nerves to recover normal activity after anesthesia, obtained from analyzing a large number of cases. Otherwise, return to S6.

[0042] S8. Calculate the awakening parameters based on the data from the visible light camera. If the awakening parameters are less than the awakening threshold, it means that the patient's activity has returned to normal. The processing device sends a reminder message to the medical staff. At the same time, calculate the rotation angle of the patient's head based on the data from the 3D camera. If the rotation angle is greater than the angle threshold, it means that the patient is experiencing significant pain. The processing device sends a third alarm to the medical staff. The calculation process for the awakening parameter involves using an end-to-end neural network, YOLOv4-tiny, to process images captured by a visible light camera. This process acquires data from the left eye region, right eye region, infraorbital region, cheek region, and lip region. For the infraorbital and cheek regions, it identifies time points of facial changes such as muscle contraction and twitching based on reference points. For the lip region, it identifies time points of mouth opening. For the left and right eye regions, it acquires the time intervals of each change cycle for both eyes. Combining the time points of facial changes and mouth opening, the facial changes and mouth openings are categorized into different eye change cycles. The total number of facial changes and mouth openings is used as the number of facial expressions in each eye change cycle. The average number of facial expressions in each eye change cycle is calculated. The average time required for the left eye to complete one cycle and the average time required for the right eye to complete one cycle are also calculated. The average of these two values ​​is taken as the eye change speed. The awakening parameter is calculated as: Eye change speed / Average number of facial expressions in each eye change cycle.

[0043] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. It should be noted that any equivalent variations made to the present invention by those skilled in the art without departing from its design structure and principles are considered within the scope of protection of the present invention.

Claims

1. A face feature-based anesthesia wake-up prediction method based on an anesthesia wake-up prediction system, the anesthesia wake-up prediction system comprising a processing device, a 3D camera, a visible light camera and a millimeter wave radar module, the processing device being communicatively connected with the 3D camera, the visible light camera and the millimeter wave radar module respectively; Sliding rails are arranged on the left and right sides of the upper half of a hospital bed, two 3D cameras are arranged on the sliding rails on the left and right sides of the hospital bed through a first moving seat, the millimeter wave radar module and the visible light camera are arranged on the sliding rail on one side of the hospital bed through a second moving seat, and the first moving seat and the second moving seat are arranged on the sliding rails in sequence; characterized in that The anesthesia wake-up prediction method specifically comprises the following steps: S1. Adjusting the position of the first moving seat so that the 3D camera is located on both sides of the patient's head, and then adjusting the position of the second moving seat so that it is next to the first moving seat; S2. Automatically adjusting the positions of the 3D camera, the visible light camera and the millimeter wave radar module; S3. Setting a reference time as the current time; S4. Calculating an eye response parameter according to the data of the visible light camera and the millimeter wave radar module; S5. If the eye response parameter is greater than a response threshold, proceeding to S6, if the eye response parameter is less than or equal to the response threshold, calculating the time difference between the current time and the reference time, if the time difference is greater than a first time threshold, the processing device needs to issue a first alarm to medical staff, otherwise, returning to S4; S6. Calculating an eye change rule parameter according to the data of the visible light camera and the millimeter wave radar module; S7. If the eye change rule parameter is less than a rule threshold, proceeding to S8, if the eye change rule parameter is greater than or equal to the rule threshold, calculating the time difference between the current time and the reference time, if the time difference is greater than a second time threshold, the processing device issues a second alarm to medical staff, otherwise, returning to S6; S8. Calculating a wake-up parameter according to the data of the visible light camera, if the wake-up parameter is less than a wake-up threshold, the processing device issues a reminder information to medical staff, at the same time, calculating the rotation angle of the patient's head according to the data of the 3D camera, if the rotation angle is greater than an angle threshold, the processing device issues a third alarm to medical staff; In S4, the calculation process of the eye response parameter is to obtain an ultrasonic image of the eye based on the data of the millimeter wave radar module, to calculate the average value of the pupil diameters of both eyes and the pupil response speed according to the ultrasonic image, to obtain the eye change speed according to the image captured by the visible light camera, and the eye response parameter = the average value of the pupil diameters of both eyes / (the pupil response speed x the eye change speed); In S6, the calculation process of the eye change rule parameter is to obtain an ultrasonic image of the eye based on the data of the millimeter wave radar module, to calculate the pupil response speed according to the ultrasonic image, to obtain the eye change speed according to the image captured by the visible light camera, and the eye change rule parameter = |(the pupil response speed / the eye change speed) - 1|, wherein |……| is an absolute value operation. The pupil response speed is the average of the time for the left eye ultrasound signal amplitude to change from maximum to minimum and the time for the right eye ultrasound signal amplitude to change from maximum to minimum; The calculation process of the eye change speed is to process the image captured by the visible light camera using an end-to-end neural network yolov4-tiny, obtain the left eye region and the right eye region, calculate the average time required for the left eye to complete one cycle of change and the average time required for the right eye to complete one cycle of change, and take the average of the two as the eye change speed.

2. The anesthesia emergence prediction method according to claim 1, characterized in that, The first moving seat comprises a first sliding block, a first support rod, a first sliding rod and a first rotating shaft, the first sliding block is connected to the sliding rail, the first support rod is arranged above the first sliding block, the first sliding rod is telescopically arranged in the first support rod, and the 3D camera is connected to the first sliding rod through the first rotating shaft.

3. The anesthesia emergence prediction method according to claim 2, characterized in that, The second moving seat comprises a second sliding block, a second telescopic rod, a second connecting rod, a second sliding rod, a second rotating shaft, a third connecting rod and a detection end, the second sliding block is connected to the sliding rail, the second telescopic rod is arranged between the second sliding block and the second connecting rod, the second sliding rod is telescopically arranged in the second connecting rod, the third connecting rod is connected to the second sliding rod through the second rotating shaft, the third connecting rod is connected to the detection end, a visible light camera is arranged at the bottom central position of the detection end, and millimeter wave radar modules are connected to the detection end through horizontal rotating discs and vertical rotating discs at both ends of the bottom of the detection end.

4. The anesthesia emergence prediction method according to claim 3, characterized in that, In the S2, the second telescopic rod, the second sliding rod and the second rotating shaft are adjusted based on the image captured by the visible light camera, so that the detection end faces the patient's face, after the detection end faces the patient's face, the horizontal rotating discs and the vertical rotating discs are adjusted based on the image captured by the visible light camera, so that the millimeter wave radar modules are respectively aligned with the left and right eyes, and at the same time, the first sliding rod and the first rotating shaft are adjusted based on the image captured by the visible light camera, so that the distance and angle between the 3D camera and the patient's head meet the set requirements.

5. The anesthesia emergence prediction method according to claim 3, characterized in that, In the S8, the calculation process of the wake-up parameter is to process the image captured by the visible light camera by using an end-to-end neural network yolov4-tiny, to obtain the left eye region, the right eye region, the infraorbital region, the cheek region and the lip region, to determine each time point of the facial changes such as muscle contraction and muscle twitch in the infraorbital region and the cheek region based on the reference point, to determine each time point of the mouth opening in the lip region, to obtain the time interval of each change cycle of the left eye and the right eye in the left eye region and the right eye region, to divide the facial changes and the mouth opening into each change cycle of the eyes in combination with each time point of the facial changes and each time point of the mouth opening, to take the total number of the facial changes and the mouth opening as the expression number of each change cycle of the eyes, to calculate the average number of expressions of each change cycle of the eyes, to calculate the average time required for the left eye to complete a cycle of changes and the average time required for the right eye to complete a cycle of changes, to take the average value of the two as the eye change speed, and the wake-up parameter = eye change speed / average number of expressions of each change cycle of the eyes.

Images