Evaluation method and device for anesthesia abnormal event processing, and electronic device

Abstract

The application discloses anesthetic abnormal event processing evaluation method and device and electronic equipment, relates to the technical field of data processing, and through inputting real-time intervention data of an operator into a large language model containing structured prompt words, reasoning, by the large language model, a pathophysiological evolution trend corresponding to the intervention data based on medical logic constraint rules, generating a nonlinear feedback conforming to pathophysiological laws in real time and dynamically, realizing coupling between physiological responses of a simulation dummy and intervention behaviors of the operator, and improving dynamic response capability in a simulation teaching process. Meanwhile, by performing time sequence behavior coding on audio and video stream data of the operator, intervention reaction time of the operator at each observation node in the anesthetic abnormal event processing process is accurately extracted, the intervention reaction time is compared with a preset time threshold to generate a structured review report, so that evaluation of the operator's reaction can be accurately quantitatively evaluated based on objective time sequence data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an assessment method and device for handling anesthetic adverse events, and electronic equipment. Background Technology

[0002] In simulated teaching of anesthesia abnormalities, existing simulation training systems typically use predefined scripts to drive changes in the physiological parameters of the simulated mannequin. The parameter response depends on the operator manually triggering or calling a preset linear change curve according to the predefined script. Meanwhile, the evaluation of the trainee's performance mainly relies on the instructor's on-site observation and post-training subjective scoring.

[0003] However, existing technologies lack dynamic coupling between changes in the physiological parameters of simulated dummies and the actual intervention operations performed by trainees, making it impossible to generate real-time feedback that conforms to pathophysiological principles based on differences in the trainees' actions. Furthermore, current assessment methods cannot accurately evaluate the trainees' actions, making it difficult to objectively assess their responses during the progression of adverse anesthetic events.

[0004] Therefore, how to improve the dynamic response capability in the simulated teaching process and accurately evaluate the trainees' operations are technical problems that urgently need to be solved. Summary of the Invention

[0005] This application provides an assessment method, device, and electronic equipment for handling anesthetic adverse events. Its main purpose is to address how to improve dynamic response capabilities during simulated teaching processes and to accurately assess trainees' actions.

[0006] According to a first aspect of this application, an assessment method for managing anesthetic adverse events is provided, comprising: Acquire multimodal operational data of the operator during the handling of adverse events in anesthesia. The multimodal operational data includes at least audio and video stream data and real-time intervention data. Real-time intervention data is input into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the human simulator terminal. The parameter adjustment instructions are obtained by the large language model based on pre-configured medical logic constraint rules and reasoning about the evolution trend of physiological parameters corresponding to the real-time intervention data. Perform temporal behavior encoding on the audio and video stream data to obtain the intervention reaction time of at least one observation node in the process of handling anesthesia abnormal events. The observation node is an event node used to characterize the operator's operation behavior. The intervention response time is compared with a preset time threshold, and a structured debriefing report containing visual evaluation indicators is generated based on the comparison results.

[0007] According to a second aspect of this application, an assessment device for handling anesthetic adverse events is provided, comprising: The acquisition unit is used to acquire multimodal operational data of the operator during the handling of anesthesia adverse events. The multimodal operational data includes at least audio and video stream data and real-time intervention data. The input unit is used to input real-time intervention data into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the human terminal. The parameter adjustment instructions are obtained by the large language model based on pre-configured medical logic constraint rules and reasoning about the evolution trend of physiological parameters corresponding to the real-time intervention data. The encoding unit is used to perform temporal behavior encoding on audio and video stream data to obtain the intervention reaction time of at least one observation node in the process of handling anesthesia abnormal events. The observation node is an event node used to characterize the operator's operation behavior. The generation unit is used to compare the intervention response time with a preset time threshold and generate a structured debriefing report containing visual evaluation indicators based on the comparison results.

[0008] According to a third aspect of this application, an electronic device is provided, comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the method described in the first aspect above.

[0009] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method of the first aspect described above.

[0010] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method as described in the first aspect above.

[0011] The assessment method, device, and electronic equipment for handling anesthesia adverse events provided in this application input the operator's real-time intervention data into a large language model containing structured prompts. The large language model uses medical logic constraints to infer the pathophysiological evolution trend corresponding to the intervention data, enabling the physiological parameter changes of the simulated terminal to dynamically generate nonlinear feedback conforming to pathophysiological laws in real time, based on the operator's actual actions. This achieves coupling between the simulated mannequin's physiological response and the operator's intervention behavior, improving the dynamic response capability during simulated teaching. Simultaneously, by performing temporal behavioral encoding on the operator's audio and video stream data, the intervention reaction time of the operator at each observation node during the handling of anesthesia adverse events is accurately extracted. This intervention reaction time is compared with a preset time threshold to generate a structured debriefing report, allowing for precise quantitative assessment of the operator's response based on objective temporal data.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein: Figure 1 A flowchart illustrating an assessment method for handling anesthetic adverse events provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an assessment device for handling anesthetic adverse events provided in an embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] The following description, with reference to the accompanying drawings, describes an evaluation method, apparatus, and electronic device for handling anesthetic adverse events according to embodiments of this application.

[0016] Figure 1 This is a flowchart illustrating an assessment method for handling anesthetic adverse events provided in an embodiment of this application.

[0017] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain multimodal operation data of the operator during the handling of anesthesia adverse events. The multimodal operation data includes at least audio and video stream data and real-time intervention data.

[0018] In the embodiments of this application, multimodal operational data refers to a set of raw interactive data containing multiple data modalities, used to characterize the operator's behavior and operational records throughout the simulation exercise. Audio and video stream data refers to a multimedia data stream formed by encoding and encapsulating video image signals and audio sound signals synchronously collected at the simulation exercise site. This audio and video stream data records the visual behavior (including operational actions, spatial movement, posture, equipment operation, etc.) and voice interaction content (including command issuance, information notification, repetition confirmation, and calls for help, etc.) of the operator and their team members throughout the simulation exercise.

[0019] Real-time intervention data refers to the recorded data of specific intervention actions performed by operators in response to the current abnormal event state during simulation exercises. This real-time intervention data is organized using an event-based data structure, and each piece of real-time intervention data includes at least an intervention action type identifier, a timestamp of occurrence, and optional parameter information (such as drug dosage, operation amplitude, or operation frequency). In one possible implementation, the real-time intervention data is entered and uploaded to the system in real-time by instructors or observers using terminal devices at the simulation exercise site.

[0020] Step 102: Input the real-time intervention data into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the simulated human terminal. The parameter adjustment instructions are obtained by the large language model based on pre-configured medical logic constraint rules and reasoning about the evolution trend of physiological parameters corresponding to the real-time intervention data.

[0021] In the embodiments of this application, structured prompts refer to input text sequences organized according to a preset template format and logical hierarchy, which are used to clearly define the reasoning direction and output format of the large language model. Structured prompts contain clearly defined field labels, value range constraints, and inference rule descriptions, making the output of the large language model predictable, structured, and controllable.

[0022] In one possible implementation, the structured prompt words are constructed by converting medical logic constraint rules into a sequence of instructions that can be recognized and followed by a large language model according to a preset parsing syntax. This sequence of instructions includes a role setting field, a task description field, an input data field, a constraint condition field, and an output format field.

[0023] Large language models refer to deep neural network language models that are pre-trained on large-scale corpora. They have the ability to understand semantics from natural language input, perform logical reasoning, and generate outputs that conform to human language habits.

[0024] This application constructs specialized structured cue words to constrain and guide the reasoning logic of a general large language model, enabling it to produce reasoning results that conform to medical logic within the specific task domain of handling anesthesia-related adverse events.

[0025] Medical logic constraint rules refer to a set of medical knowledge rules pre-extracted and formally expressed based on authoritative medical guidelines. These rules can be organized and stored in the form of condition-consequence logic pairs to define the behavioral boundaries of large language models when reasoning about changes in physiological parameters. In one implementation, medical logic constraint rules include at least: constraints on the direction of physiological parameter changes that should occur after a specific intervention (e.g., heart rate should increase and blood pressure should rise after adrenaline injection), constraints on the magnitude of change (e.g., a reasonable range of heart rate changes), and constraints on the rate of change (e.g., blood pressure recovery should not exceed the physiological limit rate).

[0026] The evolution trend of physiological parameters refers to the trajectory of changes in the vital signs parameters of a manipulator terminal over a future time window. This trend is characterized in the form of a time-series sequence, describing the direction, magnitude, and rate of change of various vital signs parameters (including but not limited to heart rate, blood pressure, blood oxygen saturation, and end-tidal carbon dioxide partial pressure) of the manipulator terminal from the current moment. Specifically, the evolution trend of physiological parameters includes at least the direction (increase or decrease), magnitude (range of change), and rate (change per unit time) of the corresponding vital signs parameter of the manipulator terminal.

[0027] The parameter adjustment command is a low-level control command that the simulator terminal can directly parse and execute. The parameter adjustment command carries multiple parameter adjustment values ​​arranged in a time sequence. After receiving the parameter adjustment command, the simulator terminal updates its internal physiological parameter model according to the time point and adjustment value specified in the parameter adjustment command.

[0028] Step 103: Perform temporal behavior encoding on the audio and video stream data to obtain the intervention reaction time of at least one observation node in the process of handling anesthesia abnormal events. The observation node is an event node used to characterize the operator's operation behavior.

[0029] In the embodiments of this application, temporal behavior coding refers to the process of segmenting, labeling, and structurally encoding continuous audio and video stream data according to the time dimension. Specifically, temporal behavior coding identifies behavioral events with specific semantic meanings by jointly analyzing the visual content (operator's body movements, hand operations, gaze direction, facial expressions, etc.) and audio content (voice commands, announcements, dialogues, etc.) in the audio and video stream, and records and stores the occurrence time, duration, and behavior category of these behavioral events on the timeline in the form of structured data.

[0030] In one implementation, the preset behavior coding analysis algorithm is implemented by combining rule-based behavior recognition logic with machine learning-based pattern recognition logic.

[0031] An observation node refers to an event node used to characterize the operator's operational behavior, corresponding to a key behavioral step with clinical decision-making significance in the anesthesia adverse event handling process. In one implementation, the observation node includes at least four key steps with a progressive time sequence: the reporting event node (the moment when the operator reports the abnormal signs to team members), the judgment event node (the moment when the operator completes information integration and analysis and forms a preliminary judgment), the decision event node (the moment when the operator makes a decision among multiple options and issues instructions), and the execution event node (the moment when the operator or team members actually perform the intervention operation).

[0032] Intervention response time refers to the length of time from the occurrence of a trigger signal in an anesthesia-related abnormal event to the operator's corresponding response. The intervention response time is calculated separately for different observation points. For example, for a notification event point, the intervention response time can be the time difference between the moment the abnormal vital signs appear and the moment the operator first issues a notification; for a decision event point, the intervention response time can be the time difference between obtaining key diagnostic information and the operator issuing a clear decision instruction.

[0033] Step 104: Compare the intervention response time with the preset time threshold, and generate a structured debriefing report containing visual evaluation indicators based on the comparison results.

[0034] In the embodiments of this application, the preset time threshold refers to a time benchmark value pre-set for each observation node to evaluate whether the operator's response speed is qualified. The preset time threshold is pre-determined based on clinical guidelines, expert consensus, or statistical data of historically excellent operators for handling anesthesia adverse events. Different observation nodes can be configured with different preset time thresholds. For example, for a reporting event node, the preset time threshold can be set to complete the reporting within 10 seconds after the occurrence of abnormal vital signs; for a decision-making event node, the preset time threshold can be set to make a decision within 15 seconds after obtaining key information.

[0035] Visual evaluation indicators refer to operator evaluation results data presented in a graphical or chart-like visual form. Structured debriefing reports refer to standardized evaluation reports generated according to preset data structures and organizational templates. Their content includes at least the operator's identity information, the abnormal event type identifier of this exercise, the intervention response time data of each observation node, the comparison results with preset time thresholds, and visual evaluation indicators.

[0036] The assessment method for handling anesthetic adverse events provided in this application achieves automation and precision by combining dynamic physiological extrapolation based on a large language model with temporal behavioral coding quantitative analysis. It can automatically generate nonlinear physiological parameter feedback that conforms to pathophysiological laws based on the operator's actual intervention. Simultaneously, it can quantify the operator's intervention reaction time at each observation node with millisecond-level precision, and achieve data-driven objective assessment through automated comparison and visual feedback. This effectively solves the technical problems of rigid scenario evolution, subjective evaluation systems, and lagging feedback mechanisms in existing technologies.

[0037] In one possible implementation of this application embodiment, when acquiring multimodal operation data of the operator during the handling of anesthesia abnormal events, it can be achieved in the following ways, but not limited to: acquiring video stream data of the operator during the handling of anesthesia abnormal events through a video acquisition device, and synchronously acquiring environmental audio data through an audio device, wherein the video acquisition device includes at least a panoramic acquisition device and a close-up acquisition device; merging the video stream data and the environmental audio data to obtain audio-visual stream data; acquiring the intervention information input by the operator in real time through a terminal device, and using the intervention information as real-time intervention data.

[0038] In the embodiments of this application, the video acquisition device refers to a device deployed at the simulated exercise site to acquire continuous video image signals of the operator and their team members throughout the exercise. Video stream data refers to a continuous sequence of image frames output by the video acquisition device, containing multiple temporally consecutive frames. Each frame represents the visual scene of the simulated exercise site at a certain moment in the form of a two-dimensional pixel matrix. The video stream data has a fixed frame rate and image resolution to ensure clear recording of details such as the operator's fine hand movements, drug extraction operations, injection actions, and changes in monitor screen parameters.

[0039] A panoramic acquisition device refers to a camera positioned high to the side or top corner of the simulated training site, with its lens covering the entire simulated operating area. A close-up acquisition device refers to a camera positioned to the side or directly above the simulated operating table, with its lens focusing on the operator's hand area, the ampoule area, the syringe area, and the parameter display panel area of ​​the mannequin.

[0040] Audio equipment refers to hardware devices deployed at the simulated exercise site for synchronously acquiring sound signals from the environment. Environmental audio data refers to the digital sound signal data stream acquired by the audio equipment, which records all sound information generated at the simulated exercise site throughout the entire exercise, including but not limited to verbal instructions, information reports, decision-making discussions, medical order repetitions, and requests for assistance between operators and their team members, as well as device sound signals such as alarm prompts from the simulator terminal and heart rate prompts from the monitor.

[0041] In one implementation, the merging operation refers to the process of encapsulating and integrating time-synchronized video stream data and ambient audio data according to a unified timeline to form a multimedia data file containing visual and audio channels. Specifically, the first video stream captured by the panoramic acquisition device, the second video stream captured by the close-up acquisition device, and the ambient audio data captured by the audio device are time-aligned based on their respective timestamp information, and then encapsulated in a multi-track format according to a preset multimedia encapsulation format (such as MP4, MOV, or MKV).

[0042] Terminal devices refer to portable electronic interactive terminals used by instructors or observers in simulated training scenarios. Terminal devices have data input and communication functions and include, but are not limited to, tablets, smartphones, dedicated handheld data terminals, or laptops. Intervention information refers to descriptive records of the clinical intervention actions actually performed by the operator during the simulated training. Each intervention information entry must contain at least the following data fields: intervention action type field (used to identify the name or code of the specific intervention action performed, such as "intravenous injection of adrenaline," "adjustment of ventilator ventilation," etc.), occurrence timestamp field (used to record the time when the intervention action occurred, based on the terminal device's system clock), and optional parameter information field (used to record quantitative parameters related to the intervention action, such as drug dosage values, ventilation adjustment values, compression depth, or compression frequency, etc.).

[0043] In one possible embodiment, data acquisition can be achieved by deploying a 3D vision and high-fidelity audio network, including panoramic cameras capturing team collaboration, close-up cameras recording precise drug delivery actions, and omnidirectional microphones. Simultaneously, a mobile interactive device interface for the instructor's end-user is included for real-time recording of the operator's intervention actions.

[0044] In one possible implementation of this application embodiment, when real-time intervention data is input into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the simulated human terminal, it can be implemented in the following ways, but not limited to: retrieving target medical knowledge matching the real-time intervention data from a preset medical knowledge base based on retrieval enhancement generation technology, and concatenating the target medical knowledge with the real-time intervention data to form an enhanced prompt context; in the large language model, performing reasoning processing based on the structured prompt words and the enhanced prompt context to obtain the physiological parameter evolution trend, wherein the structured prompt words contain medical logic constraint rules constructed based on medical guidelines to constrain the reasoning direction of the large language model; and converting the physiological parameter evolution trend into parameter adjustment instructions, wherein the physiological parameter evolution trend at least includes the direction, magnitude, and rate of change of the vital sign parameters corresponding to the simulated human terminal.

[0045] In the embodiments of this application, retrieval-enhanced generation technology refers to a technical architecture that combines the information retrieval process with the text generation process of a large language model. This architecture retrieves auxiliary knowledge related to the current input from external knowledge sources before feeding the input data into the large language model for reasoning. The retrieved auxiliary knowledge is then combined with the original input data to form an enhanced model input, thereby improving the reasoning accuracy and knowledge richness of the large language model in specific domain tasks. In this application, retrieval-enhanced generation technology is used to obtain medical knowledge that semantically matches the current real-time intervention data from a pre-set medical knowledge base, providing supplementary domain knowledge support for the reasoning of the large language model.

[0046] A pre-built medical knowledge base refers to a structured or semi-structured collection of medical knowledge stored in advance, containing information related to the management of adverse anesthetic events. The medical knowledge stored in a pre-built medical knowledge base includes at least: descriptions of the pathophysiological mechanisms of various adverse anesthetic events; standard treatment procedures for various adverse events; expected changes in physiological parameters corresponding to each treatment step; pharmacokinetic and pharmacodynamic parameters of commonly used drugs; dose-response relationship data between drug dosage and changes in physiological parameters; and clinical decision-making rules extracted based on authoritative medical guidelines.

[0047] Target medical knowledge refers to one or more medical knowledge entries retrieved from a pre-defined medical knowledge base that have the highest semantic relevance to the real-time intervention data. The nature of the content of the target medical knowledge depends on the specific type of real-time intervention data. For example, when the real-time intervention data is an injection of 1 mg of epinephrine, the target medical knowledge may include the recommended dose range of epinephrine in specific scenarios such as malignant hyperthermia or anaphylactic shock, the expected time curves of heart rate and blood pressure changes after injection, and possible adverse reactions and precautions.

[0048] Enhanced contextual prompts refer to the complete contextual text used to input large language models, which is obtained by combining the original input data (i.e., real-time intervention data) with the retrieved supplementary data (i.e., target medical knowledge) according to a preset splicing template.

[0049] In one possible embodiment, the large language model integrates Retrieval Enhancement Generation (RAG) technology with structured prompts, and incorporates medical logic constraints based on authoritative medical guidelines. It receives real-time actions from the trainee (e.g., "inject 1mg epinephrine") from the acquisition module, performs concurrent calculations in the background at the second level, and outputs multi-branch, non-linear physiological parameter change trajectories based on a pre-defined Observation-Judgment-Decision-Execution (OODA) pathophysiological decision tree, feeding these trajectories back to the simulated human control terminal. For timeouts or erroneous actions, the system can trigger spontaneous penalties for worsening vital signs.

[0050] In one possible implementation of this application embodiment, the structured prompt words contain multiple pathophysiological branch judgment nodes, each branch judgment node having corresponding triggering conditions and physiological parameter change rules; in the large language model, reasoning processing is performed based on the structured prompt words and enhanced prompting context to obtain the physiological parameter evolution trend, including: in response to real-time intervention data satisfying the triggering conditions corresponding to the target branch judgment node, the large language model outputs the physiological parameter evolution trend along the parameter reasoning path corresponding to the target branch judgment node, where the target branch judgment node is any branch judgment node; in response to the occurrence time of real-time intervention data being later than the target time threshold corresponding to the target branch judgment node, the large language model outputs the physiological parameter evolution trend including the vital sign deterioration parameter along the punitive reasoning path corresponding to the target branch judgment node, where the physiological parameter change rules include the parameter reasoning path and the punitive reasoning path.

[0051] In the embodiments of this application, the pathophysiology branch decision node refers to a conditional decision point embedded in the internal logical framework of the structured prompt words, used to dynamically select branches for the reasoning path of the large language model at different stages in the process of handling anesthesia adverse events. The structured prompt words containing multiple pathophysiology branch decision nodes enable the large language model to adaptively select the reasoning branch that matches the current clinical situation when faced with different real-time intervention data, thereby outputting the evolution trend of physiological parameters that conforms to the changes in specific pathophysiological states.

[0052] Triggering conditions refer to the logical criteria used to determine whether the current real-time intervention data should trigger the corresponding branch judgment node. In one implementation, the triggering conditions include at least: intervention action type matching conditions (e.g., intervention action type is intravenous injection of adrenaline or intervention action type is adjustment of ventilator parameters) and optional parameter range conditions (e.g., adrenaline dose is in the range of 0.5 mg to 2.0 mg or ventilation adjustment range is within ±20%).

[0053] Physiological parameter change rules refer to a set of rules defined under branch decision nodes to guide the reasoning of physiological parameter evolution trends in large language models. A physiological parameter change rule contains multiple sub-rules, each specifying how a particular vital sign parameter should change in the clinical context corresponding to the branch decision node. Physiological parameter change rules are organized and expressed in the form of condition-outcome logical pairs, and their content includes at least: specifying the direction, magnitude, and rate of change of the vital sign parameter after triggering the current branch decision node. For example, corresponding to the branch decision node of adrenaline injection, its physiological parameter change rules could include the following sub-rules: the direction of heart rate change is upward, the magnitude of the heart rate increase is from 5 to 30 beats per minute, and the rate of increase is from 1 to 10 beats per minute.

[0054] The parameter inference path refers to the computational logic chain followed by the large language model to infer the output data (the evolution trend of physiological parameters) from the current input data (real-time intervention data) under the constraint of the physiological parameter change rules of the branch judgment node.

[0055] The target time threshold refers to the critical time value used to determine whether the operator's response is timely. Each branch judgment node can be configured with an independent preset time threshold according to the importance and urgency of the corresponding clinical intervention.

[0056] The punitive inference path refers to an alternative computational logic chain followed by large language models when inferring the evolution trend of physiological parameters under conditions of operator response delay. It differs from the normal parameter inference path in the orientation of its output. A key characteristic of the punitive inference path is that its output includes parameters indicating deterioration of physiological characteristics. These parameters are quantitative indicators used to characterize a shift in the physiological state of a simulated person towards a worse trajectory compared to the normal evolutionary path.

[0057] In one possible implementation of this application embodiment, when obtaining the intervention reaction time of at least one observation node in the process of handling anesthesia abnormal events, it can be achieved in the following ways, but not limited to: based on a preset behavior coding analysis algorithm, the operator's operation behavior in the audio and video stream data is time-axis labeled according to a predefined behavior dictionary to obtain the labeling results; the behavior timestamps corresponding to multiple observation nodes are extracted from the labeling results, and the intervention reaction time of each observation node is calculated according to the behavior timestamps, wherein the multiple observation nodes include at least a notification event node, a judgment event node, a decision event node, and an execution event node.

[0058] In the embodiments of this application, the preset behavior coding analysis algorithm refers to a pre-configured computational algorithm for automatically identifying, classifying, and locating operator behaviors in audio and video stream data. The behavior dictionary refers to a knowledge base containing definitions of various standard operational behavior categories and their corresponding visual-auditory discrimination features during the handling of anesthesia abnormalities. Time axis annotation refers to the process of recording and marking various operator behaviors identified in the audio and video stream data according to their occurrence time position on the time axis of the audio and video stream data. The annotation result of the time axis annotation is an annotation data structure, and each annotation record contains at least the following fields: behavior category identifier (pointing to a behavior entry in the behavior dictionary), start timestamp (the start time of the behavior on the audio and video stream time axis), and end timestamp (the end time of the behavior on the audio and video stream time axis). In one embodiment, the accuracy of the time axis annotation reaches the millisecond level, that is, both the start timestamp and the end timestamp are recorded in milliseconds, and the reference origin of the timestamp is the start acquisition time of the audio and video stream data.

[0059] A behavior timestamp refers to the time position marker of a behavior event corresponding to a specific observation node on the timeline of the audio / video stream in the annotation results. The intervention response time is calculated as the difference between the behavior timestamp of the observation node and the baseline time. The baseline time can be set according to the type of observation node and the characteristics of the abnormal event.

[0060] In one possible implementation of this application embodiment, after calculating the intervention response time of each observation node based on the behavior timestamp, it can also be implemented in the following ways, but not limited to: performing voice activity detection on the audio waveform in the audio and video stream data to obtain the time point of the operator's operation behavior and the time point of the operation behavior recitation; using the difference between the time point of the operation behavior and the time point of the recitation as the team collaboration index corresponding to the operator; and storing the intervention response time in association with the team collaboration index.

[0061] In the embodiments of this application, performing speech activity detection on the audio waveform in the audio / video stream data refers to processing the audio channel data in the audio / video stream data using signal processing algorithms to identify which time periods in the audio waveform contain valid speech activity (i.e., speaking sounds made by the operator or team members) and which time periods do not contain valid speech activity (i.e., silence or ambient noise). In one embodiment, speech activity detection is implemented using a traditional signal processing method based on a combination of short-time energy analysis and zero-crossing rate analysis.

[0062] The time point at which an operation is issued refers to the moment when an operator issues a specific operational instruction. The time point at which an operation is relayed refers to the moment when a team member, after receiving the instruction from the operator, relays and confirms the instruction. In one implementation, after obtaining the time point of the operation's issuance and the time point of its relay, the difference between the issuance and relay times is used as the team collaboration indicator for the operator. Team collaboration indicators are numerical parameters used to quantitatively evaluate the team's communication efficiency and the completeness of the information loop during the handling of abnormal events.

[0063] In one possible implementation of this application embodiment, when comparing the intervention response time with a preset time threshold and generating a structured debriefing report containing visual evaluation indicators based on the comparison results, it can be implemented in the following ways, but not limited to: determining at least one time window threshold matching the anesthesia abnormality based on a preset medical knowledge base, wherein the preset time threshold includes at least one time window threshold; comparing the intervention response time of each observation node with the corresponding time window threshold to obtain the comparison results, and marking the observation nodes corresponding to intervention response times greater than the time window threshold as break nodes; generating an evaluation map in the form of a radar chart based on the comparison results, and extracting video segments corresponding to the break nodes from the audio and video stream data, and embedding the evaluation map and video segments into the structured debriefing report.

[0064] In the embodiments of this application, the time window threshold refers to a pre-set critical time value for each observation node under a specific abnormal event type, used to determine whether the operator's response is within a reasonable time range. The numerical sources of the time window threshold include, but are not limited to: time requirements for each treatment step explicitly recommended in authoritative medical guidelines; reference time ranges determined through statistical analysis based on large-scale clinical statistical data or high-level simulated teaching historical data; and experience thresholds jointly agreed upon by senior anesthesiology experts based on clinical and teaching experience. Different types of time window thresholds may carry different confidence level labels or source labels in a preset medical knowledge base for selection and use in different assessment scenarios.

[0065] For each observation node, the difference between its corresponding intervention response time and the time window threshold is calculated. When the difference is negative or zero, it means that the intervention response time is less than or equal to the time window threshold, and the operator's response speed at that observation node is qualified. When the difference is positive, it means that the intervention response time is greater than the time window threshold, and the operator's response speed at that observation node is unqualified.

[0066] A break point indicates that in the operator's clinical decision-making chain, the corresponding link has experienced a response delay, causing a break in the continuity of the clinical decision-making process.

[0067] In one implementation, the determination of the breakpoint can also consider the grading of the excess level. Specifically, based on the determination that the intervention response time is greater than the corresponding time window threshold, the excess ratio can be further calculated, i.e., excess ratio = (intervention response time) / (intervention response time) * ... The time window threshold is calculated as (time window threshold / time window threshold) × 100%. When the excess rate falls within different ranges, the breakpoints can be classified into different severity levels, such as: slight delay (excess rate less than 20%), moderate delay (excess rate 20% to 50%), and severe delay (excess rate greater than 50%). Breakpoints of different severity levels can be distinguished by different colors or marking methods in subsequent feedback reports and visualizations. It should be noted that the specific range values ​​for severity level division are only illustrative, and those skilled in the art can adaptively adjust the level division criteria according to different teaching scenarios and assessment requirements.

[0068] A radar chart is a multidimensional data visualization chart based on radial coordinate axes on a two-dimensional plane. It has multiple coordinate axes radiating outward from the same center point. Each coordinate axis represents an evaluation dimension, and the position of the data point on each coordinate axis indicates the evaluation value in that dimension.

[0069] In one implementation, the evaluation map is a multi-dimensional visualization evaluation chart in the form of a radar chart generated based on the comparison results. It is characterized by using each observation node as an evaluation dimension and using the difference between the intervention response time of each observation node and the corresponding time window threshold as the evaluation value of the data points in each dimension.

[0070] A video clip is a continuous multimedia data segment extracted from a complete audio and video stream within a preset time range. Its start and end times are determined based on the behavior timestamp corresponding to the break point and preset buffer durations before and after. The video clip includes both visual images before and after the operation at the break point, as well as audio content at the corresponding moment. This allows evaluators and operators to review the operator's specific behavior and the surrounding context when a response delay occurs by replaying the video clip.

[0071] In one possible implementation of this application embodiment, the evaluation map uses multiple observation nodes as evaluation dimensions, and the difference between the intervention response time of each observation node and the corresponding time window threshold is used as the evaluation score of each evaluation dimension. Video clips are extracted from audio and video stream data based on the type of breakpoint and the corresponding behavior timestamp.

[0072] Among them, the type of fracture node refers to the specific category identifier of the observation node marked as a fracture node, including the reporting event node type, the judgment event node type, the decision event node type, and the execution event node type. Different types correspond to different video capture strategies and buffer duration configurations.

[0073] Corresponding to the aforementioned assessment method for handling anesthetic adverse events, this application also proposes an assessment device for handling anesthetic adverse events. Since the device embodiment of this application corresponds to the aforementioned method embodiment, details not disclosed in the device embodiment can be referred to the aforementioned method embodiment, and will not be repeated here.

[0074] Figure 2 This is a schematic diagram of the structure of an assessment device for handling anesthetic adverse events provided in an embodiment of this application, as shown below. Figure 2 As shown, it includes: The acquisition unit 21 is used to acquire multimodal operation data of the operator during the handling of anesthesia abnormal events. The multimodal operation data includes at least audio and video stream data and real-time intervention data. Input unit 22 is used to input real-time intervention data into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the human terminal. The parameter adjustment instructions are obtained by the large language model based on pre-configured medical logic constraint rules and reasoning the evolution trend of physiological parameters corresponding to the real-time intervention data. Encoding unit 23 is used to perform temporal behavior encoding on audio and video stream data to obtain the intervention reaction time of at least one observation node in the process of handling anesthesia abnormal events. The observation node is an event node used to characterize the operator's operation behavior. The generation unit 24 is used to compare the intervention response time with a preset time threshold and generate a structured review report containing visual evaluation indicators based on the comparison results.

[0075] Furthermore, in one possible implementation of this application embodiment, the acquisition unit 21 is specifically used for: The video capture device collects video stream data of the operator during the handling of anesthesia adverse events, and the audio capture device simultaneously collects environmental audio data. The video capture device includes at least a panoramic capture device and a close-up capture device. The video stream data and the ambient audio data are merged to obtain the audio and video stream data; The system acquires intervention information input by operators in real time through terminal devices and uses this information as real-time intervention data.

[0076] Furthermore, in one possible implementation of this application embodiment, the input unit 22 is specifically used for: Based on the retrieval enhancement generation technology, target medical knowledge that matches real-time intervention data is retrieved from a preset medical knowledge base, and the target medical knowledge and real-time intervention data are concatenated to form an enhanced prompt context; In the large language model, reasoning is performed based on structured cue words and enhanced cue context to obtain the evolution trend of physiological parameters. The structured cue words contain medical logic constraint rules based on medical guidelines, which are used to constrain the reasoning direction of the large language model. The physiological parameter evolution trend is converted into parameter adjustment instructions. The physiological parameter evolution trend includes at least the direction, magnitude and rate of change of the vital signs parameters corresponding to the simulated human terminal.

[0077] Furthermore, in one possible implementation of this application embodiment, the structured prompt word includes multiple pathophysiological branch judgment nodes, and each branch judgment node has corresponding triggering conditions and physiological parameter change rules; Input unit 22 is specifically used for: In response to real-time intervention data meeting the triggering conditions corresponding to the target branch judgment node, the large language model outputs the evolution trend of physiological parameters along the parameter inference path corresponding to the target branch judgment node, where the target branch judgment node is any branch judgment node. In response to the fact that the occurrence time of real-time intervention data is later than the target time threshold corresponding to the target branch judgment node, the large language model outputs the physiological parameter evolution trend containing the vital sign deterioration parameter along the punitive inference path corresponding to the target branch judgment node. The physiological parameter change rule includes the parameter inference path and the punitive inference path.

[0078] Furthermore, in one possible implementation of this application embodiment, the encoding unit 23 is specifically used for: Based on a preset behavior coding analysis algorithm, the operator's operation behavior in the audio and video stream data is time-axis labeled according to a predefined behavior dictionary to obtain the labeling results; Extract the behavior timestamps corresponding to each of the multiple observation nodes from the annotation results, and calculate the intervention response time of each observation node based on the behavior timestamps. Among them, the multiple observation nodes include at least the reporting event node, the judgment event node, the decision event node, and the execution event node.

[0079] Furthermore, in one possible implementation of this application embodiment, the encoding unit 23 is also used for: Perform voice activity detection on the audio waveform in the audio and video stream data to obtain the time point when the operator issues the operation and the time point when the operation is repeated. The difference between the time of issuance and the time of reiteration will be used as the team collaboration indicator for the operator. The intervention response time is linked to and stored with team collaboration metrics.

[0080] Furthermore, in one possible implementation of this application embodiment, the generating unit 24 is specifically used for: Based on a preset medical knowledge base, at least one time window threshold is determined to match anesthesia abnormalities. The preset time threshold includes at least one time window threshold. The intervention response time of each observation node is compared with the corresponding time window threshold to obtain the comparison results, and the observation nodes corresponding to the intervention response time that is greater than the time window threshold are marked as break nodes. Based on the comparison results, an evaluation map in the form of a radar chart is generated, and video segments corresponding to the breakpoints are extracted from the audio and video stream data. The evaluation map and video segments are then embedded into a structured review report.

[0081] Furthermore, in one possible implementation of this application embodiment, the generating unit 24 is specifically used for: Video clips are extracted from audio and video stream data based on the type of breakpoint and the corresponding behavior timestamp.

[0082] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this application, and the principle is the same. Therefore, the embodiments of this application are not limited thereto.

[0083] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0084] Figure 3 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0085] like Figure 3 As shown, device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 302 or a computer program loaded from storage unit 308 into RAM (Random Access Memory) 303. RAM 303 may also store various programs and data required for the operation of device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O (Input / Output) interface 305 is also connected to bus 304.

[0086] Multiple components in device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of monitors, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] The computing unit 301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as the evaluation method for handling anesthesia adverse events. For example, in some embodiments, the evaluation method for handling anesthesia adverse events may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured by any other suitable means (e.g., by means of firmware) to perform the aforementioned assessment method for handling anesthetic adverse events.

[0088] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0092] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0093] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0094] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0095] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An assessment method for handling anesthetic adverse events, characterized in that, include: Acquire multimodal operational data of the operator during the handling of anesthesia adverse events, wherein the multimodal operational data includes at least audio and video stream data and real-time intervention data; The real-time intervention data is input into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the human simulator terminal. The parameter adjustment instructions are obtained by the large language model based on pre-configured medical logic constraint rules, inferring the evolution trend of the physiological parameters corresponding to the real-time intervention data. The audio and video stream data are subjected to temporal behavior encoding to obtain the intervention reaction time of at least one observation node in the process of handling anesthesia abnormal events. The observation node is an event node used to characterize the operator's operation behavior. The intervention response time is compared with a preset time threshold, and a structured debriefing report containing visual evaluation indicators is generated based on the comparison results.

2. The method according to claim 1, characterized in that, The acquisition of multimodal operational data by the operator during the handling of adverse anesthesia events includes: The video stream data of the operator during the handling of anesthesia adverse events is collected by video acquisition equipment, and environmental audio data is collected synchronously by audio equipment. The video acquisition equipment includes at least a panoramic acquisition device and a close-up acquisition device. The video stream data and the environmental audio data are merged to obtain the audio and video stream data; The intervention information input by the operator in real time through the terminal device is obtained and used as the real-time intervention data.

3. The method according to claim 1, characterized in that, The step of inputting the real-time intervention data into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the simulated human terminal includes: Based on the retrieval enhancement generation technology, target medical knowledge that matches the real-time intervention data is retrieved from a preset medical knowledge base, and the target medical knowledge and the real-time intervention data are concatenated to form an enhanced prompt context; In the large language model, reasoning is performed based on the structured prompt words and the enhanced prompt context to obtain the evolution trend of the physiological parameters. The structured prompt words contain medical logic constraint rules constructed based on medical guidelines, which are used to constrain the reasoning direction of the large language model. The physiological parameter evolution trend is converted into the parameter adjustment command, and the physiological parameter evolution trend includes at least the direction, magnitude and rate of change of the vital signs parameters corresponding to the simulated human terminal.

4. The method according to claim 3, characterized in that, The structured prompts contain multiple pathophysiological branch judgment nodes, and each branch judgment node has a corresponding triggering condition and physiological parameter change rules; In the large language model, the reasoning process based on the structured prompt words and the enhanced prompt context yields the following trends in the evolution of the physiological parameters: In response to the real-time intervention data satisfying the triggering condition corresponding to the target branch judgment node, the large language model outputs the evolution trend of the physiological parameters along the parameter inference path corresponding to the target branch judgment node, where the target branch judgment node is any branch judgment node; In response to the occurrence time of the real-time intervention data being later than the target time threshold corresponding to the target branch judgment node, the large language model outputs the physiological parameter evolution trend containing the vital sign deterioration parameter along the punitive inference path corresponding to the target branch judgment node. The physiological parameter change rule includes the parameter inference path and the punitive inference path.

5. The method according to claim 1, characterized in that, The step of performing temporal behavior encoding on the audio and video stream data to obtain the intervention reaction time of the operator at at least one observation node during the handling of anesthesia abnormal events includes: Based on a preset behavior coding analysis algorithm, the operator's operation behavior in the audio and video stream data is time-axis labeled according to a predefined behavior dictionary to obtain the labeling result; Extract the behavior timestamps corresponding to each of the multiple observation nodes from the annotation results, and calculate the intervention response time of each observation node based on the behavior timestamps. The multiple observation nodes include at least a notification event node, a judgment event node, a decision event node, and an execution event node.

6. The method according to claim 5, characterized in that, After calculating the intervention response time of each observation node based on the behavior timestamp, the method further includes: Perform voice activity detection on the audio waveform in the audio and video stream data to obtain the time point when the operator's operation behavior is issued and the time point when the operation behavior is repeated. The difference between the time point of issuance and the time point of reiteration is used as the team collaboration indicator for the operator. The intervention response time is associated with and stored in relation to the team collaboration metrics.

7. The method according to claim 1, characterized in that, The step of comparing the intervention response time with a preset time threshold and generating a structured debriefing report containing visual evaluation indicators based on the comparison results includes: At least one time window threshold is determined based on a preset medical knowledge base to match the anesthesia abnormality event, and the preset time threshold includes the at least one time window threshold. The intervention response time of each observation node is compared with the corresponding time window threshold to obtain the comparison result, and the observation node corresponding to the intervention response time greater than the time window threshold is marked as a break node. An evaluation map in the form of a radar chart is generated based on the comparison results, and video segments corresponding to the breakpoints are extracted from the audio and video stream data. The evaluation map and the video segments are then embedded into the structured review report.

8. The method according to claim 7, characterized in that, The evaluation map uses multiple observation nodes as evaluation dimensions, and the difference between the intervention response time of each observation node and the corresponding time window threshold is used as the evaluation score of each evaluation dimension. The video segment is extracted from the audio and video stream data according to the type of the break node and the behavior timestamp corresponding to the break node.

9. An assessment device for handling anesthetic adverse events, characterized in that, include: The acquisition unit is used to acquire multimodal operation data of the operator during the handling of anesthesia adverse events. The multimodal operation data includes at least audio and video stream data and real-time intervention data. The input unit is used to input the real-time intervention data into a large language model containing structured prompt words to obtain parameter adjustment instructions for driving the simulated human terminal. The parameter adjustment instructions are obtained by the large language model based on pre-configured medical logic constraint rules, inferring the evolution trend of physiological parameters corresponding to the real-time intervention data. The encoding unit is used to perform temporal behavior encoding on the audio and video stream data to obtain the intervention reaction time of at least one observation node of the operator in the process of handling anesthesia abnormal events, wherein the observation node is an event node used to characterize the operator's operation behavior. The generation unit is used to compare the intervention response time with a preset time threshold and generate a structured debriefing report containing visual evaluation indicators based on the comparison results.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

Images