Anesthesia awakening prediction system and method based on facial features

Abstract

The invention provides an anesthesia awakening prediction system and method.The anesthesia awakening prediction system comprises a processing device, a 3D camera, a visible light camera and a millimeter wave radar module, and the processing device is in communication connection with the 3D camera, the visible light camera and the millimeter wave radar module; sliding rails are arranged on the left side and the right side of the upper half portion of the sickbed, the two 3D cameras are arranged on the sliding rails on the left side and the right side of the sickbed through first moving seats, the millimeter wave radar module and the visible light camera are arranged on the sliding rails on one side of the sickbed through second moving seats, and the first moving seats and the second moving seats are sequentially arranged on the sliding rails. According to the method, the awakening state is analyzed layer by layer based on the eye change, the facial expression and the mouth feature, and the workload of medical staff is reduced while the monitoring requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of anesthesia monitoring, and in particular to an anesthesia awakening prediction system and method based on facial features. Background Art

[0002] During the recovery process after anesthesia, since the residual effects of anesthetics, muscle relaxants and nerve blocks have not disappeared, the body's protective reflexes have not fully recovered, and complications such as airway obstruction, vomiting, and aspiration are prone to occur, resulting in the existence of multiple risk factors in anesthesia recovery. Among them, the period within one hour after surgery is the period that requires the closest attention, and almost all complications that are serious enough to endanger the patient's life safety occur during this period.

[0003] In the existing technology, it is necessary to rely on professional nurses to carry out anesthesia resuscitation monitoring. During the monitoring process, professional nurses need to constantly patrol the wards to check the patient's electrocardiogram, blood pressure, pulse and other parameters until the patient's vital signs are stable. The entire monitoring process requires professional nurses to pay great efforts, and hospitals often face a shortage of professional nurses and cannot meet the strict requirements of anesthesia resuscitation monitoring. Summary of the invention

[0004] In order to solve the technical problems in the prior art that the workload of artificial anesthesia resuscitation monitoring is large and the monitoring quality cannot be guaranteed, the anesthesia awakening prediction system based on facial features proposed by the present invention includes a processing device, a 3D camera, a visible light camera and a millimeter wave radar module, and the processing device is respectively communicated with the 3D camera, the visible light camera and the millimeter wave radar module;

[0005] Sliding tracks are arranged on both sides of the upper half of the bed, and two 3D cameras are arranged on the sliding tracks on the left and right sides of the bed through a first movable seat. The millimeter wave radar module and the visible light camera are arranged on the sliding track on one side of the bed through a second movable seat, and the first movable seat and the second movable seat are arranged on the sliding track in sequence.

[0006] Preferably, the first movable seat includes a first sliding block, a first support rod, a first sliding rod and a first rotating shaft, the first sliding block is connected to a sliding track, the first support rod is arranged above the first sliding block, the first sliding rod is telescopically arranged inside the first support rod, and the 3D camera is connected to the first sliding rod via the first rotating shaft.

[0007] Preferably, the second movable seat includes a second sliding block, a second retractable rod, a second connecting rod, a second sliding rod, a second rotating shaft, a third connecting rod and a detection end, the second sliding block is connected to the sliding track, the second retractable rod is arranged between the second sliding block and the second connecting rod, the second sliding rod is telescopically arranged inside the second connecting rod, the third connecting rod is connected to the second sliding rod through the second rotating shaft, the third connecting rod is connected to the detection end, a visible light camera is arranged at the central position of the bottom of the detection end, and the two ends of the bottom of the detection end are connected to the millimeter wave radar module through a horizontal rotating disk and a vertical rotating disk.

[0008] Based on the above anesthesia recovery prediction system, the anesthesia recovery prediction method based on facial features proposed in the present invention specifically includes the following steps:

[0009] S1. Adjust the position of the first movable seat so that the 3D cameras are located on both sides of the patient's head, and then adjust the position of the second movable seat so that it is close to the first movable seat;

[0010] S2, automatically adjust the positions of the 3D camera, visible light camera and millimeter wave radar module;

[0011] S3, setting the reference time to the current time;

[0012] S4, calculating eye reaction parameters according to data from the visible light camera and the millimeter wave radar module;

[0013] S5, if the eye reaction parameter is greater than the reaction threshold, enter S6, if the eye reaction parameter is less than or equal to the reaction threshold, calculate the time difference between the current time and the reference time, if the time difference is greater than the first time threshold, the processing device needs to issue a first alarm to the medical staff, otherwise, return to S4;

[0014] S6. Calculate eye change law parameters based on data from the visible light camera and millimeter wave radar module;

[0015] S7, if the eye change regularity parameter is less than the regularity threshold, enter S8, if the eye change regularity parameter is greater than or equal to the regularity threshold, calculate the time difference between the current time and the reference time, if the time difference is greater than the second time threshold, the processing device sends a second alarm to the medical staff, otherwise, return to S6;

[0016] S8. Calculate the awakening parameter based on the data from the visible light camera. If the awakening parameter is less than the awakening threshold, the processing device sends a reminder message to the medical staff. At the same time, calculate the rotation angle of the patient's head based on the data from the 3D camera. If the rotation angle is greater than the angle threshold, the processing device sends a third alarm to the medical staff.

[0017] Preferably, in S2, the second retractable rod, the second sliding rod and the second rotating shaft are adjusted based on the image taken by the visible light camera so that the detection end is facing the patient's face. After the detection end is facing the patient's face, the horizontal rotating disk and the vertical rotating disk are adjusted according to the image taken by the visible light camera so that the millimeter wave radar module is aimed at the eyes on the left and right sides respectively. At the same time, the first sliding rod and the first rotating shaft are adjusted according to the image taken by the visible light camera so that the distance and angle between the 3D camera and the patient's head meet the set requirements.

[0018] Preferably, in S4, the calculation process of the eye reaction parameter is to obtain an ultrasonic image of the eye based on the data of the millimeter wave radar module, calculate the average pupil diameter of both eyes and the pupil reaction speed according to the ultrasonic image, and obtain the eye change speed according to the image taken by the visible light camera, and the eye reaction parameter = average pupil diameter of both eyes / (pupil reaction speed×eye change speed).

[0019] Preferably, in S4, the pupil reaction speed is the average of the average time taken for the ultrasonic signal amplitude of the left eye to change from a maximum value to a minimum value and the average time taken for the ultrasonic signal amplitude of the right eye to change from a maximum value to a minimum value.

[0020] Preferably, in S4, the calculation process of the eye change speed is to use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area and the right eye area, calculate the average time required for the left eye to complete a cycle of change and the average time required for the right eye to complete a cycle of change, and take the average of the two as the eye change speed.

[0021] Preferably, in S6, the calculation process of the eye change law parameter is to obtain an ultrasonic image of the eye based on the data of the millimeter wave radar module, calculate the pupil reaction speed according to the ultrasonic image, the pupil reaction speed is the average of the time it takes for the ultrasonic signal amplitude of the left eye to change from the maximum value to the minimum value and the time it takes for the ultrasonic signal amplitude of the right eye to change from the maximum value to the minimum value, use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area and the right eye area, calculate the time required for the left eye to complete a cycle of change and the time required for the right eye to complete a cycle of change, and take the average of the two as the eye change speed, the eye change law parameter = |(pupil reaction speed / eye change speed)-1|, where |...| is an operation of taking absolute values.

[0022] Preferably, in S8, the calculation process of the awakening parameter is to use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area, the right eye area, the infraorbital area, the cheek area and the lip area, for the infraorbital area and the cheek area, judge the various time points of facial changes such as muscle contraction and muscle twitching based on the reference point, for the lip area, judge the various time points of opening the mouth, for the left eye area and the right eye area, obtain the time interval of each change cycle of the left eye and the right eye, combine the various time points of facial changes and the various time points of opening the mouth, divide the facial changes and opening the mouth into various change cycles of the eyes, take the total number of facial changes and opening the mouth as the number of expressions in each change cycle of the eyes, calculate the average number of expressions in the eye change cycle, calculate the average time required for the left eye to complete a cycle of change and the average time required for the right eye to complete a cycle of change, and take the average of the two as the eye change speed, awakening parameter = eye change speed / average number of expressions in the eye change cycle.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The awakening state is analyzed layer by layer based on eye changes, facial expressions and mouth features, which reduces the workload of medical staff while meeting monitoring requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural schematic diagram of the anesthesia recovery prediction system of the present invention;

[0026] Figure 2 It is a structural schematic diagram of the first mobile seat of the present invention;

[0027] Figure 3 It is a structural schematic diagram of the second movable seat of the present invention.

[0028] Description of the drawings: 11. First sliding block, 12. First supporting rod, 13. First sliding rod, 14. First rotating shaft, 15. 3D camera, 21. Second sliding block, 22. Second telescopic rod, 23. Second connecting rod, 24. Second sliding rod, 25. Second rotating shaft, 26. Third connecting rod, 27. Detection end, 28. Visible light camera, 29. Horizontal rotating disk, 30. Vertical rotating disk, 31. Millimeter wave radar module. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] like Figure 1As shown, the facial feature-based anesthesia awakening prediction system proposed by the present invention includes a processing device, a 3D camera, a visible light camera and a millimeter wave radar module, and the processing device is respectively connected to the 3D camera, the visible light camera and the millimeter wave radar module for communication. Sliding tracks are set on both sides of the upper half of the bed, and two 3D cameras are set on the sliding tracks on the left and right sides of the bed through a first moving seat. The millimeter wave radar module and the visible light camera are set on the sliding track on one side of the bed through a second moving seat, and the first moving seat and the second moving seat are sequentially set on the sliding track.

[0031] like Figure 2 As shown, the first movable seat includes a first sliding block 11, a first support rod 12, a first sliding rod 13 and a first rotating shaft 14. The first sliding block 11 is connected to the sliding track. The first support rod 12 is arranged above the first sliding block 11. The first sliding rod 13 is telescopically arranged inside the first support rod 12 to adjust the distance between the 3D camera 15 and the patient's head. The 3D camera 15 is connected to the first sliding rod 13 through the first rotating shaft 14 and can rotate ±90° in the vertical direction relative to the first sliding rod 13. The first sliding rod 13 moves under the drive of the micro motor, and the first rotating shaft 14 rotates under the drive of the micro motor. The 3D camera uses Kinect launched by Microsoft. By integrating the images taken by the 3D cameras on the left and right sides, a stereoscopic image of the patient's head can be obtained to determine the tilt angle of the patient's head.

[0032] like Figure 3As shown, the second movable seat includes a second sliding block 21, a second telescopic rod 22, a second connecting rod 23, a second sliding rod 24, a second rotating shaft 25, a third connecting rod 26 and a detection end 27. The second sliding block 21 is connected to the sliding track, the second telescopic rod 22 is arranged between the second sliding block 21 and the second connecting rod 23, the second sliding rod 24 is telescopically arranged inside the second connecting rod 23, the third connecting rod 26 is connected to the second sliding rod 24 through the second rotating shaft 25, the third connecting rod 26 can rotate ±90° in the vertical direction relative to the second sliding rod 24, and the third connecting rod 26 is connected to the detection end 27. A visible light camera 28 is arranged at the bottom center of the detection end 27, and the two ends of the bottom of the detection end 27 are connected to the millimeter wave radar module 31 through a horizontal rotating disk 29 and a vertical rotating disk 30. The second telescopic rod 22 is extended and retracted under the drive of the micro motor, the second sliding rod 24 is moved under the drive of the micro motor, and the second rotating shaft 25, the horizontal rotating disk 29 and the vertical rotating disk 30 are rotated under the drive of the micro motor. The second telescopic rod 22, the second sliding rod 24 and the second rotating shaft 25 cooperate with each other to adjust the detection position of the detection end 27. The visible light camera 28 is used to take facial images. The millimeter wave radar module 31 can be aimed at the eyes on both sides under the drive of the horizontal rotating disk 29 and the vertical rotating disk 30 to obtain ultrasonic images of the eyes.

[0033] The processing device receives data from the 3D camera, the visible light camera and the millimeter wave radar module, and makes anesthesia recovery prediction based on the received data.

[0034] Based on the above anesthesia recovery prediction system, the anesthesia recovery prediction method based on facial features specifically includes the following steps:

[0035] S1. Adjust the position of the first movable seat so that the 3D camera is located on both sides of the patient's head, and then adjust the position of the second movable seat so that it is close to the first movable seat.

[0036] S2, automatically adjust the positions of the 3D camera, the visible light camera and the millimeter wave radar module. The specific adjustment process is to adjust the second telescopic rod 22, the second sliding rod 24 and the second rotating shaft 25 based on the image captured by the visible light camera, so that the detection end 27 is facing the patient's face. After the detection end is facing the patient's face, the horizontal rotating disk 29 and the vertical rotating disk 30 are adjusted according to the image captured by the visible light camera, so that the millimeter wave radar module is respectively aimed at the eyes on the left and right sides. At the same time, the first sliding rod 13 and the first rotating shaft 14 are adjusted according to the image captured by the visible light camera, so that the distance and angle between the 3D camera and the patient's head meet the set requirements.

[0037] S3. Set the reference time to the current time.

[0038] S4. Calculate eye reaction parameters based on the data from the visible light camera and the millimeter wave radar module. The eye reaction parameter calculation process is to obtain an ultrasonic image of the eye based on the data from the millimeter wave radar module, calculate the average pupil diameter of both eyes and the pupil reaction speed based on the ultrasonic image, the pupil reaction speed is the average of the average time that the ultrasonic signal amplitude of the left eye changes from the maximum value to the minimum value and the average of the average time that the ultrasonic signal amplitude of the right eye changes from the maximum value to the minimum value, use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area and the right eye area, calculate the average time required for the left eye to complete a cycle of change and the average time required for the right eye to complete a cycle of change, and use the average of the two as the eye change speed, eye reaction parameter = average pupil diameter of both eyes / (pupil reaction speed × eye change speed), because eye changes can reflect autonomic nervous activity, eye reaction parameters can characterize whether the brain begins to gradually resume normal activity during anesthesia recovery.

[0039] S5. If the eye reaction parameter is greater than the reaction threshold, enter S6. If the eye reaction parameter is less than or equal to the reaction threshold, calculate the time difference between the current time and the reference time. If the time difference is greater than the first time threshold, it means that the time when the patient's neural activity occurs is greater than the empirical value, and medical staff is required to observe and process it. The processing device needs to issue a first alarm to the medical staff. The first time threshold is the length of time required for the patient's brain nerves to become active after anesthesia, obtained by analyzing a large number of cases. Otherwise, return to S4.

[0040] S6. Calculate the eye change law parameter based on the data of the visible light camera and the millimeter wave radar module. The eye change law parameter calculation process is to obtain the ultrasonic image of the eye based on the data of the millimeter wave radar module, calculate the pupil reaction speed based on the ultrasonic image, the pupil reaction speed is the average of the time it takes for the ultrasonic signal amplitude of the left eye to change from the maximum value to the minimum value and the time it takes for the ultrasonic signal amplitude of the right eye to change from the maximum value to the minimum value, use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area and the right eye area, calculate the time required for the left eye to complete a cycle of change and the time required for the right eye to complete a cycle of change, and use the average of the two as the eye change speed, eye change law parameter = | (pupil reaction speed / eye change speed) - 1 |, where | ... | is an absolute value operation.

[0041] S7. If the eye change regularity parameter is less than the regularity threshold, enter S8. If the eye change regularity parameter is greater than or equal to the regularity threshold, calculate the time difference between the current time and the reference time. If the time difference is greater than the second time threshold, it means that the time required for the patient's neural activity to stabilize is greater than the empirical value, and medical staff is required to observe and process. The processing device sends a second alarm to the medical staff. The second time threshold is the length of time required for the patient's brain nerves to resume normal activity after anesthesia, obtained by analyzing a large number of cases. Otherwise, return to S6.

[0042] S8. Calculate the awakening parameter based on the data from the visible light camera. If the awakening parameter is less than the awakening threshold, it means that the patient's activities have returned to normal. The processing device sends a reminder message to the medical staff. At the same time, calculate the rotation angle of the patient's head based on the data from the 3D camera. If the rotation angle is greater than the angle threshold, it means that the patient feels greater pain. The processing device sends a third alarm to the medical staff. The calculation process of the awakening parameter is to use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area, right eye area, infraorbital area, cheek area and lip area, and for the infraorbital area and cheek area, judge the various time points of facial changes such as muscle contraction and muscle twitching based on the reference points, and for the lip area, judge the various time points of mouth opening, and for the left eye area and the right eye area, obtain the time interval of each change cycle of the left eye and the right eye, combine the various time points of facial changes and the various time points of mouth opening, divide facial changes and mouth opening into various change cycles of the eyes, and use the total number of facial changes and mouth opening as the number of expressions in each change cycle of the eyes, calculate the average number of expressions in the eye change cycle, calculate the average time required for the left eye to complete a cycle of change and the average time required for the right eye to complete a cycle of change, and use the average of the two as the eye change speed. Awakening parameter = eye change speed / average number of expressions in the eye change cycle.

[0043] The above disclosure is only the preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the present invention. It should be pointed out that for those skilled in the art, any equivalent changes made to the scheme of the present invention without departing from the design structure and principle of the present invention are considered to be within the protection scope of the present invention.

Claims

1. A facial feature-based anesthesia recovery prediction system, characterized in that ,The anesthesia awakening prediction system includes a processing device, a 3D camera, a visible light camera and a millimeter wave radar module, and the processing device is communicatively connected with the 3D camera, the visible light camera and the millimeter wave radar module respectively; Sliding tracks are arranged on both sides of the upper half of the bed, and two 3D cameras are arranged on the sliding tracks on the left and right sides of the bed through a first movable seat. The millimeter wave radar module and the visible light camera are arranged on the sliding track on one side of the bed through a second movable seat, and the first movable seat and the second movable seat are arranged on the sliding track in sequence.

2. The anesthesia recovery prediction system according to claim 1, characterized in that: The first movable seat includes a first sliding block, a first support rod, a first sliding rod and a first rotating shaft, the first sliding block is connected to a sliding track, the first support rod is arranged above the first sliding block, the first sliding rod is telescopically arranged inside the first support rod, and the 3D camera is connected to the first sliding rod via the first rotating shaft.

3. The anesthesia recovery prediction system according to claim 2, characterized in that: The second movable seat includes a second sliding block, a second retractable rod, a second connecting rod, a second sliding rod, a second rotating shaft, a third connecting rod and a detection end, the second sliding block is connected to the sliding track, the second retractable rod is arranged between the second sliding block and the second connecting rod, the second sliding rod is telescopically arranged inside the second connecting rod, the third connecting rod is connected to the second sliding rod through the second rotating shaft, and the third connecting rod is connected to the detection end, a visible light camera is arranged at the central position of the bottom of the detection end, and the two ends of the bottom of the detection end are connected to the millimeter wave radar module through a horizontal rotating disk and a vertical rotating disk.

4. A facial feature-based anesthesia recovery prediction method, based on the anesthesia recovery prediction system of claim 3, characterized in that ,The anesthesia awakening prediction method specifically includes the following steps: S1. Adjust the position of the first movable seat so that the 3D cameras are located on both sides of the patient's head, and then adjust the position of the second movable seat so that it is close to the first movable seat; S2, automatically adjust the positions of the 3D camera, visible light camera and millimeter wave radar module; S3, setting the reference time to the current time; S4, calculating eye reaction parameters according to data from the visible light camera and the millimeter wave radar module; S5, if the eye reaction parameter is greater than the reaction threshold, enter S6, if the eye reaction parameter is less than or equal to the reaction threshold, calculate the time difference between the current time and the reference time, if the time difference is greater than the first time threshold, the processing device needs to issue a first alarm to the medical staff, otherwise, return to S4; S6. Calculate eye change law parameters based on data from the visible light camera and millimeter wave radar module; S7, if the eye change regularity parameter is less than the regularity threshold, enter S8, if the eye change regularity parameter is greater than or equal to the regularity threshold, calculate the time difference between the current time and the reference time, if the time difference is greater than the second time threshold, the processing device sends a second alarm to the medical staff, otherwise, return to S6; S8. Calculate the awakening parameter based on the data from the visible light camera. If the awakening parameter is less than the awakening threshold, the processing device sends a reminder message to the medical staff. At the same time, calculate the rotation angle of the patient's head based on the data from the 3D camera. If the rotation angle is greater than the angle threshold, the processing device sends a third alarm to the medical staff.

5. The anesthesia recovery prediction method according to claim 4, characterized in that: In S2, the second retractable rod, the second sliding rod and the second rotating shaft are adjusted based on the image taken by the visible light camera so that the detection end is facing the patient's face. After the detection end is facing the patient's face, the horizontal rotating disk and the vertical rotating disk are adjusted according to the image taken by the visible light camera so that the millimeter wave radar module is aimed at the eyes on the left and right sides respectively. At the same time, the first sliding rod and the first rotating shaft are adjusted according to the image taken by the visible light camera so that the distance and angle between the 3D camera and the patient's head meet the set requirements.

6. The anesthesia recovery prediction method according to claim 4, characterized in that: In S4, the calculation process of the eye reaction parameter is to obtain an ultrasonic image of the eye based on the data of the millimeter wave radar module, calculate the average pupil diameter of both eyes and the pupil reaction speed according to the ultrasonic image, and obtain the eye change speed according to the image taken by the visible light camera. The eye reaction parameter = average pupil diameter of both eyes / (pupil reaction speed×eye change speed).

7. The anesthesia recovery prediction method according to claim 6, characterized in that: In S4, the pupil reaction speed is the average of the average time it takes for the ultrasonic signal amplitude of the left eye to change from the maximum value to the minimum value and the average time it takes for the ultrasonic signal amplitude of the right eye to change from the maximum value to the minimum value.

8. The anesthesia recovery prediction method according to claim 6, characterized in that: In S4, the calculation process of the eye change speed is to use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area and the right eye area, calculate the average time required for the left eye to complete a cycle of change and the average time required for the right eye to complete a cycle of change, and use the average of the two as the eye change speed.

9. The anesthesia recovery prediction method according to claim 4, characterized in that: In the S6, the calculation process of the eye change law parameter is to obtain an ultrasonic image of the eye based on the data of the millimeter wave radar module, and calculate the pupil reaction speed according to the ultrasonic image, the pupil reaction speed is the average of the time taken for the ultrasonic signal amplitude of the left eye to change from the maximum value to the minimum value and the time taken for the ultrasonic signal amplitude of the right eye to change from the maximum value to the minimum value, and use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area and the right eye area, calculate the time required for the left eye to complete a cycle of change and the time required for the right eye to complete a cycle of change, and take the average of the two as the eye change speed, the eye change law parameter = |(pupil reaction speed / eye change speed)-1|, where |...| is an operation of taking absolute values.

10. The anesthesia recovery prediction method according to claim 4, characterized in that: In the S8, the calculation process of the awakening parameter is to use the end-to-end neural network yolov4-tiny to process the image taken by the visible light camera, obtain the left eye area, the right eye area, the infraorbital area, the cheek area and the lip area, for the infraorbital area and the cheek area, based on the reference point, judge the various time points of facial changes such as muscle contraction and muscle twitching, for the lip area, judge the various time points of opening the mouth, for the left eye area and the right eye area, obtain the time interval of each change cycle of the left eye and the right eye, combine the various time points of facial changes and the various time points of opening the mouth, divide the facial changes and opening the mouth into various change cycles of the eyes, take the total number of facial changes and opening the mouth as the number of expressions in each change cycle of the eyes, calculate the average number of expressions in the change cycle of the eyes, calculate the average time required for the left eye to complete a cycle of change and the average time required for the right eye to complete a cycle of change, and take the average of the two as the eye change speed, awakening parameter = eye change speed / average number of expressions in the change cycle of the eyes.

Images