Painless endoscopic patient departure assessment system based on multi-modal biological feature recognition

Abstract

The invention relates to the technical field of medical evaluation, in particular to a painless endoscopic patient leave-room evaluation system based on multi-modal biological feature recognition, comprising: a multi-source data acquisition module configured to be capable of acquiring multi-source data of a patient entering a target area; the data fusion processing module is configured to be capable of acquiring the multi-source data from the multi-source data acquisition module and extracting a plurality of key features after preprocessing the multi-source data to form a key feature set; the decision module is configured to acquire the key feature set from the data fusion processing module and generate an evaluation report and an off-room decision by using a decision model according to the key feature set; and the traceability module is configured to construct a traceability framework and perform traceability on the evaluation report and the off-room decision based on the traceability framework.

Description

Technical Field

[0001] This invention relates to the field of medical assessment technology, and in particular to a painless endoscopic patient exit assessment system based on multimodal biometric recognition. Background Technology

[0002] Postoperative assessment for patients undergoing painless endoscopy refers to a series of evaluations conducted after the procedure, from the time the patient awakens from anesthesia until they are allowed to leave. The purpose is to ensure that the patient's vital signs have stabilized and that they can leave safely, avoiding any unexpected events during transport. The main assessment items include vital signs, level of consciousness, motor function, and pain level.

[0003] Existing assessments largely rely on the experience of medical staff, which introduces a degree of instability and makes them prone to decision-making errors, such as allowing patients to leave the ward prematurely before they have fully recovered. Furthermore, it is difficult to effectively trace the root cause of decisions after problems arise. Summary of the Invention

[0004] This invention utilizes collected multi-source data, analyzes the multi-source data through a decision-making module, and automatically generates off-site decisions, ensuring the stability of the decisions.

[0005] The technical solution proposed in this invention is: a painless endoscopic patient exit assessment system based on multimodal biometric recognition, comprising: A multi-source data acquisition module, configured to acquire multi-source data of patients entering the target area; The data fusion processing module is configured to acquire multi-source data from the multi-source data acquisition module, and extract multiple key features after preprocessing the multi-source data to form a key feature set. The decision module is configured to obtain a key feature set from the data fusion processing module and generate an evaluation report and an exit decision based on the key feature set using a decision model. The traceability module is configured to build a traceability framework and trace the assessment report and exit decision based on the traceability framework. The traceability module is connected to the multi-source data acquisition module, the data fusion processing module and the decision module.

[0006] 2. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 1, characterized in that the acquisition of multi-source data of patients entering the target area includes: Preoperative facial video data, eye video data, and gait video data of patients entering the target area were collected to form a preoperative baseline dataset; Postoperative facial video data, eye video data, and gait video data of patients entering the target area were collected to form a postoperative comparison dataset.

[0007] 3. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 2, characterized in that, after preprocessing the multi-source data, multiple key features are extracted to form a key feature set, including: The preoperative baseline dataset was preprocessed and several key preoperative features were extracted. The postoperative comparison dataset was preprocessed and several key postoperative features were extracted.

[0008] 4. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 3, characterized in that the preprocessing of the preoperative baseline dataset and extraction of multiple key preoperative features includes: Preprocessing of the preoperative baseline dataset includes: The MTCNN face detection model was used to locate the face region and key facial points in the preoperative facial video; the key points included the corners of the eyes, the tip of the nose, the corners of the mouth, and the outline of the eyebrows. Region of interest of pupil Extraction, specifically, is the process of extracting the region of interest (ROI) of the pupil from the preoperative eye video. ; Extracting the skeletal joint sequence of the human body from gait video data; Several key preoperative features were extracted, including: Based on a facial action coding system, intensity values ​​of preoperative facial action units are extracted from the facial region using a convolutional neural network. ; Region of interest of the pupil Ellipse fitting was performed to calculate the preoperative pupil diameter. ; Preoperative walking speed was calculated based on skeletal joint sequence. stride width and torso sway angle This constitutes the preoperative gait feature set. ; Constructing a preoperative key feature set: , .

[0009] 5. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 4, characterized in that the preprocessing of the postoperative comparison dataset and extraction of multiple key postoperative features includes: Preprocessing of the postoperative comparison dataset includes: The MTCNN face detection model was used to locate the face region and facial key points in the postoperative facial video; Region of interest of pupil Extraction, specifically, is the extraction of the region of interest (ROI) of the pupil from the eye area in postoperative eye videos. ; Extracting the skeletal joint sequence of the human body from gait video data; Several key postoperative features were extracted, including: Based on a facial action coding system, intensity values ​​of postoperative facial action units are extracted from the facial region using a convolutional neural network. ; Region of interest of the pupil Ellipse fitting was performed to calculate the postoperative pupil diameter. ; Postoperative walking speed was calculated based on skeletal joint sequence. stride width and torso sway angle This constitutes a postoperative gait feature set. ; Constructing a set of key postoperative features: .

[0010] 6. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 5, characterized in that the step of generating an assessment report and exit decision based on a decision model using a key feature set includes: Obtain the preoperative key feature set and the postoperative key feature set; Calculate the pain expression index ;in, This represents a set of action units associated with pain; Indicates action unit Weight in pain expression; Calculate pupil recovery ; Calculate the pupil stability index ;in, Indicates the standard deviation of postoperative pupil diameter; Represents a constant term; Calculate the gait deviation index ; ;in, , ; Indicates the number of elements in the postoperative gait feature set; Indicates the weights of gait features; The decision vector is constructed using the pain expression index, pupil recovery rate, pupil stability index, and gait deviation index. ; ; Input the decision vector into the pre-trained decision model to generate decision scores. The decision model is a meta-classifier. Based on decision score The decision to leave the room is generated according to the decision-making rules, namely: if Generate an exit decision: allow exit; if The decision to leave the room is: continue observation; if Generate an exit decision: exiting the room is not recommended or intervention is required; among them, and Indicates the threshold for leaving the room to make decisions. .

[0011] 7. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 6, characterized in that it further comprises: When calculating the pain index, a non-linear fusion of the intensity values ​​of motor units is performed based on a machine learning model, including: An activation feature vector is constructed by using the activation level of the action unit's intensity value as a feature. Among them, activation level ; Indicates the number of action units; Input the activation vector into the pre-trained pain index prediction model; Output pain index prediction value ; The decision vector is constructed using the pain index prediction value, pupil recovery degree, pupil stability index, and gait deviation index. ',Will 'Input a pre-trained decision model and generate a decision score' ; Based on decision score The optimized exit decision is generated according to the decision-making rules.

[0012] 8. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 7, characterized in that it further comprises: Infrared video data of the patient's face entering the target area is acquired and fused with previously obtained facial video data under visible light to improve detection accuracy in low light conditions, including: The infrared video data is preprocessed, specifically for facial region recognition and key point localization. The infrared video data includes preoperative and postoperative infrared video data. Confidence scores are defined for visible light facial video data and infrared video data respectively to reflect the reliability of the corresponding optical modal data under current environmental conditions, including: Calculate the confidence level of visible light ,in, This indicates the sharpness of a facial video image under visible light. This indicates the brightness of a facial video image under visible light. Infrared light reliability ; Multiple key infrared feature values ​​were extracted from infrared video data and normalized to form a preoperative infrared key feature set. Postoperative infrared key feature set ; , ;in Indicates the temperature of the left and right sides of the face before surgery; Calculating the thermal imaging stress index ; in, ; Indicates the temperature of the orbital area after surgery. Temperature of the orbital area before surgery The difference, This indicates the temperature difference between the left and right sides of the face after surgery. This indicates the temperature of the left side of the face after surgery. This indicates the temperature on the right side of the face after surgery. Indicates the temperature of the nasal tip area after surgery. Temperature of the nasal tip area before surgery The difference; , and Indicates the weight of temperature features; Constructing infrared stress feature vectors , , The infrared stress feature vector and the activation feature vector are then concatenated to obtain a fused feature vector. ; Will Input a pre-trained pain index prediction model, output a fused pain index prediction value ; Calculate gait confidence ; in, This indicates the percentage of skeletal joints in the human body whose skeletal joint sequence was successfully tracked throughout the entire walking process. ;in, This indicates the number of skeletal joints that were successfully tracked. This represents the total number of bone joints in the bone joint sequence. This represents the proportion of unoccluded skeletal joints in the skeletal joint sequence. ;in, Indicates the number of skeletal joints that are not obscured; This indicates the number of valid skeletal joint sequence sequences detected during the monitoring period; Then, the gait deviation index ; Constructing a fusion decision vector: ; Will Input the pre-trained decision model to generate the final decision score. Based on decision-making The final decision to leave the room is generated according to the decision-making rules; Set the confidence threshold for visible light. Gait confidence threshold Overall threshold ; if or This triggers a decision warning, prompting medical staff to make a decision about leaving the room.

[0013] 9. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 8, characterized in that the traceability framework for assessment reports and exit decisions includes: The framework for tracing origins includes: Constructing a unified spatiotemporal framework and data lake, namely: The obtained preoperative baseline dataset, postoperative comparison dataset, and infrared video data are synchronized in time using hardware synchronization or software timestamps. A unique identification code is created for each patient entering the target area, and the preoperative baseline dataset, postoperative comparison dataset, and infrared video data are combined with the corresponding identification code. The preoperative baseline dataset, postoperative comparison dataset, infrared video data, preoperative key feature set, postoperative key feature set, decision vector, infrared stress feature vector, visible light confidence, infrared light confidence, gait confidence, preoperative infrared key feature set, postoperative infrared key feature set, fused feature vector and fused decision vector are stored in a structured data lake. The decision model is subjected to game-method SHAP analysis to calculate the contribution of each element in the fused decision vector to the final decision score, and a contribution vector is output. ;in, These represent the contributions of the pain index, pupil recovery rate, pupil stability index, and gait deviation index, respectively. Generate a traceable exit assessment report. The exit assessment includes patient identification coding, exit decision, contribution of pain index, contribution of pupil recovery, contribution of pupil stability, and contribution of gait deviation index. , and .

[0014] The present invention also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the painless endoscopic patient exit assessment system based on multimodal biometric recognition.

[0015] The beneficial effects of this invention are: 1. This invention collects multi-source data from patients before and after surgery, using preoperative data as a baseline to ensure that decisions are based on the specific patient, thus guaranteeing the accuracy of the decisions; by introducing infrared thermal imaging, it ensures facial feature extraction in low-light environments, improving the robustness of the system.

[0016] 2. This invention, through confidence analysis of multi-source data, allows for the final decision to be made manually (by medical staff) when the data quality on which the decision to leave the room is poor. This decision-making method reduces the probability of decision-making errors and allows for the traceability of assessment reports and decisions to leave the room by constructing a traceability framework. Attached Figure Description

[0017] Figure 1 This is a block diagram of the painless endoscopic patient exit assessment system based on multimodal biometric recognition, as described in this invention. Detailed Implementation

[0018] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious modifications will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0019] It is understood that the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0020] Example 1: refer to Figure 1The technical solution provided by this invention is: a painless endoscopic patient exit assessment system based on multimodal biometric recognition, including a multi-source data acquisition module, a data fusion processing module, a decision-making module, and a traceability module.

[0021] The multi-source data acquisition module is configured to acquire multi-source data from patients entering the target area.

[0022] The data fusion processing module is configured to acquire multi-source data from the multi-source data acquisition module, and extract multiple key features after preprocessing the multi-source data to form a key feature set.

[0023] The decision module is configured to obtain a key feature set from the data fusion processing module and generate an assessment report and an exit decision based on the key feature set using a decision model.

[0024] The tracing module is configured to build a tracing framework and trace the assessment report and exit decision based on the tracing framework. The tracing module is connected to the multi-source data acquisition module, the data fusion processing module and the decision module.

[0025] The process of collecting multi-source data from patients entering the target area includes the following steps: Preoperative facial video data, eye video data, and gait video data of patients entering the target area were collected to form a preoperative baseline dataset; Postoperative facial video data, eye video data, and gait video data of patients entering the target area are collected to form a postoperative comparison dataset. In this embodiment, facial and eye videos can be collected using a high-definition camera for expression and pupil analysis; gait videos can be collected using a depth camera for non-contact gait analysis.

[0026] The process of preprocessing multi-source data to extract multiple key features and constructing a key feature set includes the following steps: The preoperative baseline dataset was preprocessed and several key preoperative features were extracted, specifically: Preprocessing of the preoperative baseline dataset includes: The MTCNN face detection model was used to locate the face region and key facial points in the preoperative facial video; the key points included the corners of the eyes, the tip of the nose, the corners of the mouth, and the outline of the eyebrows. Region of interest of pupil Extraction, specifically, is the process of extracting the region of interest (ROI) of the pupil from the preoperative eye video. ; Extracting the skeletal joint sequence of the human body from gait video data; Several key preoperative features were extracted, including: Based on the Facial Action Coding System (FACS), intensity values ​​of preoperative facial action units are extracted from the facial region using a convolutional neural network. In this embodiment The selection should focus on the intensity values ​​of the action units associated with pain and discomfort in the facial motion coding system, such as AU4 (brow drooping), AU6 or AU7 (orbicularis oculi muscle contraction), AU9 or AU10 (levator labii superioris), AU12 (corner of mouth pulling) and AU25 / 26 / 27 (mouth opening).

[0027] Region of interest of the pupil Ellipse fitting was performed to calculate the preoperative pupil diameter. ; Preoperative walking speed was calculated based on skeletal joint sequence. stride width and torso sway angle This constitutes the preoperative gait feature set. ; Constructing a preoperative key feature set: , .

[0028] The postoperative comparison dataset was preprocessed and several key postoperative features were extracted. Specifically, the postoperative comparison dataset was preprocessed, including: The MTCNN face detection model was used to locate the face region and facial key points in the postoperative facial video; Region of interest of pupil Extraction, specifically, is the extraction of the region of interest (ROI) of the pupil from the eye area in postoperative eye videos. ; Extracting the skeletal joint sequence of the human body from gait video data; Several key postoperative features were extracted, including: Based on a facial action coding system, intensity values ​​of postoperative facial action units are extracted from the facial region using a convolutional neural network. ; Region of interest of the pupil Ellipse fitting was performed to calculate the postoperative pupil diameter. ; Postoperative walking speed was calculated based on skeletal joint sequence. stride width and torso sway angle This constitutes a postoperative gait feature set. ; Constructing a set of key postoperative features: .

[0029] The process of generating an assessment report and an exit decision based on a key feature set using a decision model includes the following steps: Obtain the preoperative key feature set and the postoperative key feature set; Calculate the pain expression index ;in, This represents a set of action units associated with pain, such as (AU4, AU6, AU7, AU9). Indicates action unit Weight in pain expression; Calculate pupil recovery ; Calculate the pupil stability index ;in, Indicates the standard deviation of postoperative pupil diameter; Represents a constant term; Calculate the gait deviation index ; ;in, , ; Indicates the number of elements in the postoperative gait feature set; Indicates the weights of gait features; The decision vector is constructed using the pain expression index, pupil recovery rate, pupil stability index, and gait deviation index. ; ; Input the decision vector into the pre-trained decision model to generate decision scores. The decision model is a meta-classifier, such as a random forest classifier. Based on decision score The decision to leave the room is generated according to the decision-making rules, namely: if Generate an exit decision: allow exit; if The decision to leave the room is: continue observation; if Generate an exit decision: exiting the room is not recommended or intervention is required; among them, and Indicates the threshold for leaving the room to make decisions. .

[0030] The process of tracing the source of assessment reports and exit decisions based on the tracing framework includes the following steps: The framework for tracing origins includes: Constructing a unified spatiotemporal framework and data lake, namely: The obtained preoperative baseline dataset, postoperative comparison dataset, and infrared video data are synchronized in time using hardware synchronization or software timestamps. A unique identification code is created for each patient entering the target area, and the preoperative baseline dataset, postoperative comparison dataset, and infrared video data are combined with the corresponding identification code.

[0031] The preoperative baseline dataset, postoperative comparison dataset, and infrared video data, combined with the corresponding identification codes, are stored in a structured data lake to facilitate traceability and querying.

[0032] Example 2: In Example 1, a linear weighted summation method was used to calculate the pain index. However, synergistic and inhibitory relationships exist between different action units. For example, AU6 cheek lifting and AU7 eyelid tightening often occur simultaneously, and their combined effect is not linearly additive. Therefore, based on Example 1, we propose the following technical solution: When calculating the pain index, a non-linear fusion of the intensity values ​​of motor units is performed based on a machine learning model, including: An activation feature vector is constructed by using the activation level of the action unit's intensity value as a feature. Among them, activation level , Indicates the number of action units.

[0033] The activation vector is input into a pre-trained pain index prediction model, which outputs a predicted pain index value. In this embodiment, the prediction model is the gradient boosting machine XGBoost.

[0034] The decision vector is constructed using the pain index prediction value, pupil recovery degree, pupil stability index, and gait deviation index. ',Will 'Input a pre-trained decision model and generate a decision score' Based on decision-making The optimized exit decision is generated according to the decision-making rules.

[0035] Example 3: In low light, the quality of facial video data degrades, leading to inaccurate or even malfunctioning AU (Action Unit) detection. Therefore, we are considering introducing infrared thermal imaging technology to fuse visible light facial video data with infrared video data (thermal imaging). This allows us to still obtain effective facial features even when facial video data is unavailable in low light or visible light. The specific solution is as follows: Infrared video data of the patient's face entering the target area is acquired and fused with previously obtained facial video data under visible light to improve detection accuracy in low light conditions, including: Preprocessing of infrared video data is performed, namely facial region recognition and key point localization in the infrared video data; the infrared video data includes preoperative infrared video data and postoperative infrared video data; in this embodiment, a hardware-synchronized RGB-IR camera can be used to obtain one frame of visible light image and one frame of infrared thermal image at the same time, so that the video data of the two modes are aligned on the time axis.

[0036] Confidence scores are defined for visible light facial video data and infrared video data respectively to reflect the reliability of the corresponding optical modal data under current environmental conditions, including: Calculate the confidence level of visible light ,in, This indicates the sharpness of a facial video image under visible light. The brightness of a facial video image under visible light can be calculated using OpenCV.

[0037] Infrared light reliability Because infrared imaging is minimally affected by light in low-light conditions, its confidence level can be set to a constant, high value, or the signal-to-noise ratio of thermal images can be utilized.

[0038] Multiple key infrared feature values ​​were extracted from infrared video data and normalized to form a preoperative infrared key feature set. Postoperative infrared key feature set ; in, ; ; Indicates the temperature of the left and right sides of the face before surgery; Calculating the thermal imaging stress index ; in, ; Indicates the temperature of the orbital area after surgery. Temperature of the orbital area before surgery The difference, This indicates the temperature difference between the left and right sides of the face after surgery. This indicates the temperature of the left side of the face after surgery. This indicates the temperature on the right side of the face after surgery. Indicates the temperature of the nasal tip area after surgery. Temperature of the nasal tip area before surgery The difference; , and Indicates the weight of temperature features; Constructing infrared stress feature vectors , , The infrared stress feature vector and the activation feature vector are then concatenated to obtain a fused feature vector. ; Will Input a pre-trained pain index prediction model, output a fused pain index prediction value ; Calculate gait confidence ; in, This indicates the percentage of skeletal joints in the human body whose skeletal joint sequence was successfully tracked throughout the entire walking process. ;in, This indicates the number of skeletal joints that were successfully tracked. This represents the total number of bone joints in the bone joint sequence. This represents the proportion of unoccluded skeletal joints in the skeletal joint sequence. ;in, Indicates the number of skeletal joints that are not obscured; This indicates the number of valid skeletal joint sequence sequences detected during the monitoring period; Then, the gait deviation index ; Constructing a fusion decision vector: ; Will Input the pre-trained decision model to generate the final decision score. Based on decision-making The final decision to leave the room is generated according to the decision-making rules; Set the confidence threshold for visible light. Gait confidence threshold Overall threshold ; if or If the system automatically makes a decision to leave the room, but the data used for that decision is of poor quality, the final decision-making power is returned to the medical staff, thus reducing the probability of decision-making errors.

[0039] Based on the fusion of the above multiple modal data, the decision-making process for leaving the room is optimized, namely: Building a source tracing framework, namely, building a unified spatiotemporal framework and data lake, specifically includes the following steps: The obtained preoperative baseline dataset, postoperative comparison dataset, and infrared video data are synchronized in time using hardware synchronization or software timestamps. A unique identification code is created for each patient entering the target area, and the preoperative baseline dataset, postoperative comparison dataset, and infrared video data are combined with the corresponding identification code. The preoperative baseline dataset, postoperative comparison dataset, infrared video data, preoperative key feature set, postoperative key feature set, decision vector, infrared stress feature vector, visible light confidence, infrared light confidence, gait confidence, preoperative infrared key feature set, postoperative infrared key feature set, fused feature vector and fused decision vector are stored in a structured data lake. The decision model is subjected to game-method SHAP analysis to calculate the contribution of each element in the fused decision vector to the final decision score, and a contribution vector is output. ;in, These represent the contributions of the pain index, pupil recovery rate, pupil stability index, and gait deviation index, respectively. Generate a traceable exit assessment report. The exit assessment includes patient identification coding, exit decision, contribution of pain index, contribution of pupil recovery, contribution of pupil stability, and contribution of gait deviation index. , and .

[0040] Through the confidence level of multimodal data ( , and This allows healthcare professionals to understand the reliability of conclusions while generating decisions, and to know where to start verifying and revising decisions when uncertain.

[0041] The present invention also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the painless endoscopic patient exit assessment system based on multimodal biometric recognition.

[0042] The processes described above with reference to the flowcharts in the embodiments disclosed in this invention can be implemented as computer software programs. The embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this invention. It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical cables, RF, etc., or any suitable combination thereof.

[0043] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0044] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the principles described, the implementation of the present invention may have any changes or modifications.

Claims

1. A painless endoscopic patient exit assessment system based on multimodal biometric recognition, characterized in that, include: A multi-source data acquisition module, configured to acquire multi-source data of patients entering the target area; The data fusion processing module is configured to acquire multi-source data from the multi-source data acquisition module, and extract multiple key features after preprocessing the multi-source data to form a key feature set. The decision module is configured to obtain a key feature set from the data fusion processing module and generate an evaluation report and an exit decision based on the key feature set using a decision model. The traceability module is configured to build a traceability framework and trace the assessment report and exit decision based on the traceability framework. The traceability module is connected to the multi-source data acquisition module, the data fusion processing module and the decision module.

2. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 1, characterized in that, The collection of multi-source data from patients entering the target area includes: Preoperative facial video data, eye video data, and gait video data of patients entering the target area were collected to form a preoperative baseline dataset; Postoperative facial video data, eye video data, and gait video data of patients entering the target area were collected to form a postoperative comparison dataset.

3. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 2, characterized in that, The process of extracting multiple key features after preprocessing multi-source data to form a key feature set includes: The preoperative baseline dataset was preprocessed and several key preoperative features were extracted. The postoperative comparison dataset was preprocessed and several key postoperative features were extracted.

4. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 3, characterized in that, The process of preprocessing the preoperative baseline dataset and extracting multiple key preoperative features includes: Preprocessing of the preoperative baseline dataset includes: The MTCNN face detection model was used to locate the face region and key facial points in the preoperative facial video; the key points included the corners of the eyes, the tip of the nose, the corners of the mouth, and the outline of the eyebrows. Region of interest of pupil Extraction, specifically, is the process of extracting the region of interest (ROI) of the pupil from the preoperative eye video. ; Extracting the skeletal joint sequence of the human body from gait video data; Several key preoperative features were extracted, including: Based on a facial action coding system, intensity values ​​of preoperative facial action units are extracted from the facial region using a convolutional neural network. ; Region of interest of the pupil Ellipse fitting was performed to calculate the preoperative pupil diameter. ; Preoperative walking speed was calculated based on skeletal joint sequence. stride width and torso sway angle This constitutes the preoperative gait feature set. ; Constructing a preoperative key feature set: , 。 5. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 4, characterized in that, The preprocessing of the postoperative comparison dataset and extraction of multiple key postoperative features include: Preprocessing of the postoperative comparison dataset includes: The MTCNN face detection model was used to locate the face region and facial key points in the postoperative facial video; Region of interest of pupil Extraction, specifically, is the extraction of the region of interest (ROI) of the pupil from the eye area in postoperative eye videos. ; Extracting the skeletal joint sequence of the human body from gait video data; Several key postoperative features were extracted, including: Based on a facial action coding system, intensity values ​​of postoperative facial action units are extracted from the facial region using a convolutional neural network. ; Region of interest of the pupil Ellipse fitting was performed to calculate the postoperative pupil diameter. ; Postoperative walking speed was calculated based on skeletal joint sequence. stride width and torso sway angle This constitutes a postoperative gait feature set. ; Constructing a postoperative key feature set: 。 6. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 5, characterized in that, The process of generating assessment reports and exit decisions based on key feature sets using decision models includes: Obtain the preoperative key feature set and the postoperative key feature set; Calculate the pain expression index ;in, This represents a set of action units associated with pain; Indicates action unit Weight in pain expression; Calculate pupil recovery ; Calculate the pupil stability index ;in, Indicates the standard deviation of postoperative pupil diameter; Represents a constant term; Calculate the gait deviation index ; ;in, , ; Indicates the number of elements in the postoperative gait feature set; Indicates the weights of gait features; The decision vector is constructed using the pain expression index, pupil recovery rate, pupil stability index, and gait deviation index. ; ； Input the decision vector into the pre-trained decision model to generate decision scores. The decision model is a meta-classifier. Based on decision score The decision to leave the room is generated according to the decision-making rules, namely: if Generate an exit decision: allow exit; if The decision to leave the room is: continue observation; if Generate an exit decision: exiting the room is not recommended or intervention is required; among them, and Indicates the threshold for leaving the room to make decisions. .

7. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 6, characterized in that, Also includes: When calculating the pain index, a non-linear fusion of the intensity values ​​of motor units is performed based on a machine learning model, including: An activation feature vector is constructed by using the activation level of the action unit's intensity value as a feature. Among them, activation level ; Indicates the number of action units; Input the activation vector into the pre-trained pain index prediction model; Output pain index prediction value ; The decision vector is constructed using the pain index prediction value, pupil recovery degree, pupil stability index, and gait deviation index. ',Will 'Input a pre-trained decision model and generate a decision score' ; Based on decision score The optimized exit decision is generated according to the decision-making rules.

8. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 7, characterized in that, Also includes: Infrared video data of the patient's face entering the target area is acquired and fused with previously obtained facial video data under visible light to improve detection accuracy in low light conditions, including: The infrared video data is preprocessed, specifically for facial region recognition and key point localization. The infrared video data includes preoperative and postoperative infrared video data. Confidence scores are defined for visible light facial video data and infrared video data respectively to reflect the reliability of the corresponding optical modal data under current environmental conditions, including: Calculate the confidence level of visible light ,in, This indicates the sharpness of a facial video image under visible light. This indicates the brightness of a facial video image under visible light. Infrared light reliability ; Multiple key infrared feature values ​​were extracted from infrared video data and normalized to form a preoperative infrared key feature set. Postoperative infrared key feature set ; , ;in Indicates the temperature of the left and right sides of the face before surgery; Calculating the thermal imaging stress index ; in, ; Indicates the temperature of the orbital area after surgery. Temperature of the orbital area before surgery The difference, This indicates the temperature difference between the left and right sides of the face after surgery. This indicates the temperature of the left side of the face after surgery. This indicates the temperature on the right side of the face after surgery. Indicates the temperature of the nasal tip area after surgery. Temperature of the nasal tip area before surgery The difference; , and Indicates the weight of temperature features; Constructing infrared stress feature vectors , , The infrared stress feature vector and the activation feature vector are then concatenated to obtain a fused feature vector. ; Will Input a pre-trained pain index prediction model, output a fused pain index prediction value ; Calculate gait confidence ; in, This indicates the percentage of skeletal joints in the human body whose skeletal joint sequence was successfully tracked throughout the entire walking process. ;in, This indicates the number of skeletal joints that were successfully tracked. This represents the total number of bone joints in the bone joint sequence. This represents the proportion of unoccluded skeletal joints in the skeletal joint sequence. ;in, Indicates the number of skeletal joints that are not obscured; This indicates the number of valid skeletal joint sequence sequences detected during the monitoring period; Then, the gait deviation index ; Constructing a fusion decision vector: ； Will Input the pre-trained decision model to generate the final decision score. Based on decision-making The final decision to leave the room is generated according to the decision-making rules; Set the confidence threshold for visible light. Gait confidence threshold Overall threshold ; if or This triggers a decision warning, prompting medical staff to make a decision about leaving the room.

9. The painless endoscopic patient exit assessment system based on multimodal biometric recognition according to claim 8, characterized in that, The source tracing framework for assessment reports and exit decisions includes: The framework for tracing origins includes: Constructing a unified spatiotemporal framework and data lake, namely: The obtained preoperative baseline dataset, postoperative comparison dataset, and infrared video data are synchronized in time using hardware synchronization or software timestamps. A unique identification code is created for each patient entering the target area, and the preoperative baseline dataset, postoperative comparison dataset, and infrared video data are combined with the corresponding identification code. The preoperative baseline dataset, postoperative comparison dataset, infrared video data, preoperative key feature set, postoperative key feature set, decision vector, infrared stress feature vector, visible light confidence, infrared light confidence, gait confidence, preoperative infrared key feature set, postoperative infrared key feature set, fusion feature vector and fusion decision vector are stored in a structured data lake. The decision model is subjected to game-method SHAP analysis to calculate the contribution of each element in the fused decision vector to the final decision score, and a contribution vector is output. ;in, These represent the contributions of the pain index, pupil recovery rate, pupil stability index, and gait deviation index, respectively. Generate a traceable exit assessment report. The exit assessment includes patient identification coding, exit decision, contribution of pain index, contribution of pupil recovery, contribution of pupil stability, and contribution of gait deviation index. , and .

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the painless endoscopic patient exit assessment system based on multimodal biometric recognition as described in any one of claims 1-9.

Images