Multi-modal data spatio-temporal coupling evaluation method and system for nurse practical operation examination

Abstract

The present application relates to the technical field of artificial intelligence and medical education, and discloses a multi-modal data space-time coupling evaluation method and system for nurse practical operation examination, which comprises the following steps: receiving an asynchronous video stream to establish a video timeline; extracting physical operation action anchor point frames and pairing them with offline physiological sampling time of the manikin; fitting an affine drift model based on the anchor points to map the physiological data to a unified timeline; performing AOI attention analysis, aseptic barrier destruction judgment and decision response time quantification on the operation events based on the unified timeline; combining a specialized operation directed graph to execute downstream therapeutic effect blocking and generate evaluation results when a violation occurs. The present application corrects the inherent crystal oscillator frequency deviation and random frame loss error of asynchronous devices without external hardware synchronization conditions, realizes the causal space-time precise alignment of multi-modal heterogeneous data, objectively quantifies the intervention time delay and violation actions, and makes the evaluation more consistent with the real clinical medical causal logic.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and medical education, specifically to a multimodal data spatiotemporal coupling evaluation method and system for nurses' practical skills assessment. Background Technology

[0002] In the medical education and training system, standardized verification of the training and assessment process is fundamental to ensuring the quality of nursing talent training. Current practical assessment schemes primarily rely on process verification mechanisms based on training node datasets. These mechanisms generate compliance evaluations by automatically processing structured academic data such as attendance records, rotation durations, and signature information. This approach reduces the subjectivity of manual review to some extent.

[0003] However, existing assessment methods have limitations in terms of data dimensions and causal logic. Current technologies are essentially process records based on administrative and academic dimensions, failing to effectively cover multimodal sensory data generated during practical assessments, such as multiple video streams, object interaction trajectories, first-person perspective micro-operation details, and physiological feedback waveforms from medical simulators. Due to the lack of integration of multimodal data, existing technologies cannot achieve precise coupling between video footage, physiological parameters, and operational events at the causal level. In specialist nurse assessment scenarios such as continuous renal replacement therapy in the intensive care unit and trauma emergency care, key indicators determining the assessee's clinical decision-making ability are often the response delay of making correct interventions after pathological changes occur, the specific video frame moment when the sterile barrier is breached, and whether the operational techniques conform to spatial geometric norms. The objective quantification of these core capabilities must rely on multimodal raw sensory data with precise time anchors, and existing single academic log evaluation systems cannot provide such traceable data quantification evidence.

[0004] Furthermore, in the actual deployment of Objective Structured Clinical Examination (OSCE) test stations, the widely adopted low-cost engineering approach typically involves multiple camera devices connected to the main control unit via independent USB links, lacking an external hardware timecode synchronization mechanism between the devices. Simultaneously, the physiological data of high-fidelity medical simulators can usually only be exported offline after the assessment, resulting in complete decoupling of the clock sources for audio / video streams and physiological data streams. To address the data asynchrony issue, existing technologies either compromise by supporting only single-modal video playback and abandoning multimodal data alignment, or rely heavily on expensive real-time hardware synchronization solutions. These costly hardware synchronization solutions are not only highly invasive and costly to deploy, but also difficult to scale up, failing to meet the combined needs of specialist nurses in their daily large-scale practical assessments for low-cost hardware and high-precision causal evaluation.

[0005] Therefore, there is an urgent need for an evaluation method and system that can achieve accurate spatiotemporal alignment of multimodal data and automatic verification of operational compliance without dedicated hardware synchronization. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal data spatiotemporal coupling evaluation method and system for nurses' practical skills assessment. The technical problem solved by this invention is that existing nursing practical skills assessment methods struggle to unify video footage, operational events, and physiological parameters of medical simulators collected during practical operations onto the same timeline. Without an external hardware clock synchronization mechanism, clock drift and random frame drops between asynchronous camera devices and medical simulators can cause temporal misalignment of heterogeneous data, making it difficult to objectively quantify decision-making intervention delays and trace violations such as breaches of sterile barriers.

[0007] To address the above problems, the present invention provides the following technical solution:

[0008] The first aspect of this invention provides a multimodal data spatiotemporal coupling evaluation method for nurses' practical skills assessment, comprising:

[0009] In response to the assessment start command, the system receives video streams from multiple camera devices, extracts video frame sequences containing image frames, assigns video timestamps to the image frames, and establishes a video timeline.

[0010] Acquire offline physiological data packets containing internal clock sampling times from the output of a high-fidelity medical manikin;

[0011] Identify physical events in the video frame sequence according to predefined physical displacement events, and perform action peak detection on the video frame sequence corresponding to the physical events to locate the anchor frame of physically observable operation actions;

[0012] Extract the physiological sampling time corresponding to the physical event from the offline physiological data packet, and pair the video timestamp of the anchor frame of the physically observable operation action with the physiological sampling time as an anchor pair;

[0013] Based on the anchor point pair fitted with an affine drift model, the internal clock sampling time is mapped to a unified time axis with the video time axis as the reference.

[0014] On the unified time axis, the video frame interval and physiological sampling subset corresponding to the operation event are determined, and the operation event is subjected to AOI attention and operation chain coupling relationship determination, sterile barrier determination and decision response delay determination to generate evaluation results.

[0015] Preferably, assigning a video timestamp to the image frame includes:

[0016] Record the system startup time and establish the video timeline based on the system startup time;

[0017] The video stream is received in an asynchronous, non-blocking manner;

[0018] The system monotonic clock state at the current moment when the image frame is written to physical memory is captured, the system monotonic clock state is used as the video timestamp, and the video timestamp is bound and stored with the image frame;

[0019] When random frame drops occur in the video stream, the position of the subsequent arriving image frame in the video timeline is determined based on the video timestamp corresponding to the subsequent arriving image frame.

[0020] Preferably, the peak motion detection includes:

[0021] Visual analysis is performed on the video frame sequence to extract at least one feature quantity from the spatial displacement amplitude of key points of the hand bones and the optical flow variation features between consecutive image frames in the video frame sequence;

[0022] Search for local maxima of the feature quantity;

[0023] When the local maximum reaches a set dynamic threshold, the video frame corresponding to the local maximum is used as the anchor frame of the physically observable operation action. The dynamic threshold is used to determine whether the local maximum constitutes an operation action peak.

[0024] Preferably, the determination of the coupling relationship between AOI attention and the operation chain includes:

[0025] Obtain the region of interest for a predefined assessment scenario;

[0026] The head pose feature vector is extracted from the image frame to determine the approximate landing point of the attention orientation, the region of interest to which the approximate landing point of the attention orientation belongs is determined, and the dwell time is accumulated to form a dwell time vector.

[0027] Obtain the operation events that occur during the current operation phase, and bind the dwell time vector to the operation events as attention and behavior data pairs;

[0028] Obtain the deviation rules contained in the predefined deviation set, wherein the deviation rules include operational events and thresholds determined by assessment specifications, expert experience calibration results, or historical qualified sample statistical results;

[0029] An attention mismatch marker is generated when the attention and behavior data pair satisfies any of the deviation rules.

[0030] Preferably, before extracting the head pose feature vector, the method further includes:

[0031] Establish a polygon mask for the region of interest corresponding to the region of interest;

[0032] Output the guidance command and record the start and end timestamps corresponding to the guidance command;

[0033] Based on the start and end timestamps, a sequence of image frames containing the candidate's head region is extracted from the video stream. Transitional image frames at the beginning and end of the head posture deflection process in the sequence of image frames containing the candidate's head region are removed. The beginning and end time intervals are 0.5 seconds to 1.0 seconds.

[0034] Extract image frames from the intermediate time period to generate a calibration reference frame sequence;

[0035] The calibration reference frame sequence is input into the head pose estimation model to generate a reference head pose feature vector.

[0036] Preferably, the sterility barrier determination includes:

[0037] Extract the glove outline mask and sterile area mask from the image frame;

[0038] Perform a pixel-by-pixel inversion operation on the sterile area mask to generate a complement mask for the non-sterile area;

[0039] Obtain the intersection region of the glove outline mask and the complement mask of the non-sterile region, and extract the pixel area of ​​the connected components in the intersection region as the out-of-bounds area;

[0040] When the out-of-bounds area exceeds the out-of-bounds area threshold determined according to the spatial resolution of the image frame, and the continuous state meets the preset continuous determination condition, a sterile barrier destruction signal is triggered.

[0041] The preset persistence determination conditions include at least one of the consecutive frame count condition and duration condition.

[0042] Preferably, the affine drift model includes a scale factor and an intercept parameter;

[0043] Fitting the affine drift model includes:

[0044] The anchor point pairs are used as candidate anchor point pairs. Fitting verification is performed on the candidate anchor point pairs to generate fitting residuals. Candidate anchor point pairs whose fitting residuals exceed the residual threshold are eliminated. The remaining candidate anchor point pairs are used as valid anchor point pairs. The residual threshold is used to characterize the upper limit of the allowed time alignment deviation of the candidate anchor point pairs.

[0045] When there are multiple valid anchor point pairs, perform a multi-anchor point affine drift model fitting operation.

[0046] When the number of effective anchor point pairs is one, the scale factor is set to one, and the intercept parameter is calculated based on a single effective anchor point pair, and a constant-level time translation compensation operation is performed.

[0047] When the number of valid anchor point pairs is zero, the strong coupling scoring between the offline physiological data packet and the video timeline is not performed.

[0048] Preferably, the decision response delay determination includes:

[0049] Read time series of physiological parameters from the offline physiological data package mapped to the unified time axis, the time series of physiological parameters including at least one of blood pressure, blood oxygen and heart rate;

[0050] When the time series of physiological parameters meets the physiological parameter threshold combination and sampling window corresponding to the clinical assessment event, the starting sampling point of exceeding the limit is determined as the pathological inflection point;

[0051] Obtain the set of effective intervention events corresponding to the pathological inflection point, and locate the effective intervention start frame in the video frame sequence that corresponds to the set of effective intervention events;

[0052] Based on the video timestamp corresponding to the effective intervention start frame, the difference between the time when the pathological inflection point is mapped to the unified time axis is calculated to extract the decision response delay;

[0053] When the decision response delay exceeds the decision delay threshold determined by the professional nurse practical assessment standards, a decision lag deduction item is generated.

[0054] Preferably, after generating the evaluation result, the method further includes:

[0055] Obtain a predefined directed graph of specialty operations, which includes operation nodes, directed edges, efficacy score permission flags corresponding to the operation nodes, and original physiological efficacy scores corresponding to the operation nodes.

[0056] When it is determined based on the evaluation results that any of the operation nodes triggers the sterile barrier destruction signal, all downstream operation nodes starting from the operation node are extracted as a set of downstream operation nodes. The set of downstream operation nodes includes the operation node and all downstream operation nodes reachable from the operation node along the directed edge.

[0057] Set the efficacy score permission flag corresponding to each operation node in the downstream operation node set to False, and block the transmission of the original physiological efficacy score corresponding to each operation node in the downstream operation node set.

[0058] Based on the evaluation results, the video timestamps corresponding to the violations are extracted as deduction anchors. The violations include the sterile barrier breach signal. A deduction list containing the deduction anchors is generated.

[0059] Video clips are extracted from the deduction list, and the video clips are combined with the annotation layer to generate a structured review video file. Multiple deduction lists are aggregated to generate a group capability heatmap.

[0060] A second aspect of the present invention provides a multimodal data spatiotemporal coupling evaluation system for nurses' practical skills assessment, used to execute the method, the evaluation system comprising a hardware layer and a logic processing layer;

[0061] The hardware layer includes a main control terminal and a multi-channel camera device;

[0062] The main control terminal is used to acquire offline physiological data packets containing internal clock sampling times output by the high-fidelity medical simulator;

[0063] The logic processing layer is deployed on the main control terminal, and the logic processing layer includes a multimodal perception terminal layer, an edge computing processing layer, an intelligent evaluation layer, and a teaching feedback layer;

[0064] The multimodal sensing terminal layer is used to receive video streams collected by the multi-camera device;

[0065] The edge computing processing layer is used to extract video frame sequences containing image frames from the video stream, assign video timestamps to the image frames, establish a video timeline, identify physical events according to predefined physical displacement events and locate anchor frames of physically observable operation actions, fit an affine drift model, and map the internal clock sampling time to a unified timeline with the video timeline as the reference.

[0066] The intelligent evaluation layer is used to perform multimodal spatiotemporal coupled evaluation and generate evaluation results;

[0067] The teaching feedback layer is used to generate a deduction list, structured review video files, and group ability heatmaps based on the evaluation results.

[0068] This invention provides a multimodal data spatiotemporal coupling evaluation method and system for nurses' practical skills assessment. It has the following beneficial effects:

[0069] 1. This invention extracts anchor frames of physically observable operational actions from video sequences and pairs them with homologous physiological sampling moments in offline physiological data packets. It then fits an affine drift model for cross-clock domain mapping. In asynchronous multi-channel data collection lacking external hardware synchronization signals, this method can correct inherent crystal oscillator frequency offset and random frame loss errors between the camera device and the medical mannequin substrate. This achieves alignment of heterogeneous multimodal data on the causal time axis, thus providing a fundamental data foundation for accurately quantifying various assessment delays while reducing the cost of dedicated hardware deployment at examination stations.

[0070] 2. This invention calculates the decision-response delay between pathological inflection points and effective intervention actions on a unified time axis, and uses an image segmentation algorithm to verify the intersection area of ​​the glove contour mask and non-sterile areas frame by frame. It then combines this with a predefined directed graph of specialized procedures to determine the blocking effect on efficacy scores. This mechanism transforms traditional examiner experience-based evaluation into specific time difference values ​​and spatial geometric constraints. Furthermore, upon detecting a breach of the sterile barrier, it automatically truncates the simulated physiological improvement scores resulting from downstream operations at the violation node, making the system's scoring process more consistent with real clinical medical causal logic.

[0071] 3. This invention extracts spatial Euler angles using a head pose estimation model to calculate the approximate landing point of attention orientation, statistically analyzes the dwell time within a predefined region of interest, and links it to specific operational events. Simultaneously, it automatically edits and associates related video clips based on deduction anchor points, and aggregates them to generate a heatmap of group abilities. This solution can identify candidates' lack of contextual awareness or insufficient focus before the operation without introducing additional eye-tracking hardware. The final output of time-stamped structured videos and statistical data provides objective data for subsequent review of practical action details and adjustment of teaching focus. Attached Figure Description

[0072] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0073] Figure 2 This is a schematic diagram of the spatiotemporal coupling principle of the present invention;

[0074] Figure 3 This is a schematic diagram illustrating the coupling relationship between AOI attention and the operation chain in this invention;

[0075] Figure 4 This is a flowchart illustrating the evaluation logic of specialist nurses in this invention.

[0076] Figure 5 This is a schematic diagram of the teaching feedback interface of the present invention;

[0077] Figure 6 This is a schematic diagram of the fitting and degradation compensation of the event anchor point driven affine drift model of the present invention.

[0078] Figure 7 This is a schematic diagram of the predefined specialist operation directed graph and scoring mask blocking of the present invention;

[0079] Figure 8 This is a comparison chart of error distribution under different time axis mapping strategies of the present invention. Detailed Implementation

[0080] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] See attached document Figure 1 This invention provides a multimodal data spatiotemporal coupling evaluation system for nurses' practical skills assessment. The system includes a hardware layer and a logic processing layer.

[0082] The hardware layer includes a main control terminal, a multi-camera system, and a high-fidelity medical mannequin. The main control terminal uses an industrial computer or a computer device with processing power. The multi-camera system consists of five independent USB cameras connected to the main control terminal via a USB hub. There is no external hardware timecode synchronization signal between the five independent USB cameras and the main control terminal.

[0083] The five independent USB camera devices include a first-channel panoramic camera, a second-channel head-mounted first-person perspective stabilized camera, a third-channel monitor close-up camera, a fourth-channel sterile area overhead close-up camera, and a fifth-channel mannequin / tube close-up camera. The first-channel panoramic camera is fixed in the assessment scene to capture the candidate's head area and overall operational view. The second-channel head-mounted first-person perspective stabilized camera, as a single USB camera device, captures the candidate's first-person perspective operational view and does not serve as the data source for the overall head posture reference. The third-channel monitor close-up camera records the monitor display and provides video evidence for verifying the visibility of pathological inflection points and for structured debriefing. The high-fidelity medical mannequin is equipped with an independent internal clock, configured to output offline physiological data packets containing the internal clock sampling times after the assessment.

[0084] The logic processing layer is deployed on the main control terminal and includes a multimodal perception terminal layer, an edge computing processing layer, an intelligent evaluation layer, and a teaching feedback layer. The multimodal perception terminal layer receives video streams from multiple camera devices. The edge computing processing layer assigns video timestamps to video frames and maps the internal clock sampling times of offline physiological data packets to a unified timeline using an event-anchored affine drift model. The intelligent evaluation layer, combined with the unified timeline, performs AOI attention and operation chain coupling relationship determination, sterility barrier determination, and decision response delay determination, and performs logical scoring verification based on a predefined directed graph of specialized operations. The teaching feedback layer generates a deduction list with timestamped anchors and associated video clips based on the output of the intelligent evaluation layer, and aggregates the deduction lists of all examinees to generate a group ability heatmap.

[0085] This invention provides a multimodal data spatiotemporal coupling evaluation method for nurses' practical skills assessment, comprising the following steps:

[0086] S100, in response to the assessment start command, the main control terminal receives video streams collected by five independent USB camera devices, records the system start time, and establishes an initial relative time axis for the video frame sequence based on the start time; in the preferred embodiment where random frame loss exists, the main control terminal reads the monotonic clock time when the image frame is written to memory, uses the actual writing time as the video timestamp, and uses the video timestamp as the time reference for subsequent cross-clock domain mapping;

[0087] S200: At the beginning of the practical assessment, the main control terminal uses the first panoramic camera fixed in the assessment scene to collect images of the candidate's head area. It extracts facial key points and head rotation Euler angles through a pure head pose estimation model, and projects the head pose onto the image plane to estimate the approximate landing point of the candidate's attention orientation based on the head orientation.

[0088] During the assessment process, the S300 main control terminal extracts head posture feature vectors frame by frame, determines the region of interest based on the approximate landing point of the attention orientation, and accumulates the dwell time in the monitor parameter area, the aseptic operation work area, and other areas to form a dwell time vector. The main control terminal binds the dwell time vector with the operation events in the same operation stage to form a correspondence between attention allocation and operation behavior, and simultaneously uses image segmentation algorithm to extract the glove outline mask and the aseptic area mask in the top-down close-up camera image of the fourth aseptic area.

[0089] After the S400 assessment, the main control terminal imports the offline physiological data package generated by the high-fidelity medical mannequin, extracts the anchor frames of physically observable operation actions from the video stream sequence, and extracts the physiological sampling points corresponding to the same physical event from the offline physiological data package, and pairs the two to form anchor pairs; the main control terminal fits an affine drift model based on one or more anchor pairs, and maps the internal clock sampling time in the offline physiological data package to a unified time axis.

[0090] The S500, based on a unified timeline, extracts pathological inflection points from the main control terminal and locates the corresponding effective intervention start frame for the examinee in the video stream; the main control terminal calculates the decision response delay between the pathological inflection point and the effective intervention start frame, and compares the decision response delay with a preset decision delay threshold; when the decision response delay exceeds the preset decision delay threshold, a decision lag deduction item is generated.

[0091] S600: The main control terminal queries a predefined directed graph of specialty operations. When an operation node triggers a signal that the sterile barrier is breached, the main control terminal applies a scoring mask to all downstream operation nodes after that operation node, blocking the transmission of efficacy scores for all downstream operation nodes and allowing only the accumulation of process violation scores. The main control terminal outputs a deduction list with timestamp anchors and automatically edits video segments of the corresponding frame intervals according to the deduction list.

[0092] S700, the teaching feedback layer aggregates the deduction list of all candidates, and generates a group ability heatmap with the operation step nodes in the predefined directed graph of professional operation as the horizontal axis, the violation type as the vertical axis, and the frequency of deduction at each node as the heat value; among them, the teaching feedback layer separately counts the distribution of deductions for attention mismatch markers to identify teaching blind spots.

[0093] The following section provides a detailed explanation of each processing stage and operational logic of the technical solution described above, taking into account specific application scenarios and underlying data processing procedures.

[0094] See attached document Figure 2 The asynchronous acquisition and time base initialization phase specifically includes the following steps:

[0095] In response to the assessment start command, the S101 main control terminal establishes an asynchronous data transmission link with the five independent USB camera devices. The main control terminal calls the operating system's monotonic clock service to record the system startup time. ,in This represents the monotonic clock reading recorded by the main control terminal at the instant it receives the assessment start command, and serves as the zero-point reference for constructing the video frame relative to the time axis. The monotonic clock reading is used to establish a unified time base for the main control terminal, unaffected by system network time synchronization, system time rollback, or wall clock time jumps.

[0096] S102, the main control terminal concurrently receives video streams from five independent USB camera devices using an asynchronous non-blocking method, and constructs the corresponding video frame sequence. },in Indicates the first Frame video images, The integer is non-negative. The master control terminal establishes the initial relative time axis based on the master control monotonic time axis using the following formula:

[0097] ;

[0098] in, Indicates the system startup time. This represents the theoretical frame rate of the camera device. Indicates the first The initial relative timestamp of the video frame is derived from the theoretical frame rate. The five independent USB camera devices generate image frames independently using their internal hardware crystal oscillators, and there is no need to transmit external hardware timecode synchronization signals between the main control terminal and the five independent USB camera devices.

[0099] S103, during normal system operation, the concurrent transmission of five video data streams consumes a significant amount of general-purpose serial bus bandwidth. When the instantaneous data throughput approaches the hardware bus bandwidth limit, data backlog can easily occur in the underlying communication buffer, leading to the dropping of some image frames during transmission. To reduce the impact of such frame drops on the temporal continuity of the video sequence, the master control terminal obtains the completion time of writing each image frame in the video frame sequence into the master control terminal's physical memory at the operating system level. Specifically, when the master control terminal executes the memory write callback in the underlying driver, it captures the current system monotonic clock state and converts the system monotonic clock state into a video timestamp for that image frame. ,in Indicates the first The actual time when a frame of video image is written on the monotonic timeline of the main control terminal.

[0100] S104, the main control terminal will send the video timestamp Additional data is recorded in the corresponding image frame of the video frame sequence. The application layer uses a monotonic clock for recording, which avoids time jumps or rollback interference triggered by the system network time synchronization protocol and maintains the unidirectional incrementality of timing calculations. The main control terminal binds an independent video timestamp to each frame in the video frame sequence. And combined with the system startup time Establish a video timeline with frame drop resistance. When no frame drops occur, the video timestamp... relative timestamp The deviation is less than the preset deviation threshold, where, This represents the initial relative timestamp obtained from theoretical derivation; when random frame loss occurs, the main control terminal uses... This refers to the position of the frame on the video timeline.

[0101] The main control terminal obtains the main control monotonic timestamp from the underlying layer, thus enabling the video frame sequence... The time stamps are independent of the preset theoretical frame rate of the camera device. When random frame drops occur due to bandwidth limitations in the data transmission link, subsequent image frames received by the main control terminal can still be assigned time stamps reflecting the actual time they were written into memory. This mechanism helps to unify the time base of the five asynchronous video sequences generated by the five independent USB camera devices within the main control terminal, providing timing data support for subsequent cross-clock domain spatiotemporal reconstruction with offline physiological data packets.

[0102] To address the random frame drop phenomenon caused by hardware bandwidth limitations in asynchronous video stream acquisition, the main control terminal in this embodiment is configured with underlying timestamp acquisition and timing correction logic to establish a fault-tolerant relative timeline, specifically including the following steps:

[0103] S105, the main control terminal will initialize the relative timestamp With video timestamp Corresponding storage. When concurrent transmission from five independent USB camera devices causes random frame drops, the data of adjacent frames is stored. The difference reflects the actual physical time elapsed between two frames; subsequent image frames arriving at the master control terminal... Determine its position on the video timeline to avoid timeline shrinkage distortion caused by relying solely on theoretical frame rate derivation.

[0104] S106, the main control terminal will send the video timestamp The storage is bound to video image frames. The binding methods include building a structure object in memory that contains image pixel data and timestamp attributes, or using metadata fields in the audio / video container format to record the video timestamp. In this way, the main control terminal provides a video time reference for subsequent offline physiological data packet cross-clock domain mapping in a low-cost USB asynchronous acquisition environment.

[0105] See attached document Figure 3 The AOI spatial calibration stage prior to the assessment includes the following steps:

[0106] S201, during the system initialization phase, the main control terminal receives the spatial layout configuration parameters of the practical assessment scenario and defines the physical space with specific clinical business attributes in the assessment environment as the Region of Interest (AOI). The main control terminal establishes a polygon mask for the AOI corresponding to the AOI through at least one of the following methods: manual selection, importing preset coordinates, object detection algorithm recognition, or image segmentation algorithm recognition. This AOI polygon mask is then stored in the image coordinate system corresponding to the first panoramic camera. In one optional application scenario, combined with the critical care nursing practical assessment specifications, the predefined AOI can be configured to include a monitor area and a sterile operation work area. The monitor area corresponds to the physical space where the device screen displaying the patient's vital signs data is located, and the sterile operation work area corresponds to the physical table where the examinee performs interventional actions such as punctures or dressing changes.

[0107] S202, after the main control terminal completes system startup, it triggers the guided calibration process. In conventional computer vision applications, establishing the spatial mapping relationship between the camera and the physical work area usually relies on an external calibration board to solve the 3D extrinsic parameter matrix. To simplify the configuration of the examination room deployment environment, the main control terminal is configured to establish a spatial reference using a topological mapping mechanism based on the fixed physiological pose of the human head. The topological mapping mechanism does not require the placement of an external calibration board at the examination site; the camera intrinsic parameter matrix can be provided by the device's factory parameters, offline calibration parameters, or system initialization pre-configured parameters. The main control terminal outputs guidance instructions through the display or speaker, prompting the examinee to maintain a natural upright posture at the set operating position and to sequentially turn their head towards each predefined region of interest.

[0108] S203, when the examinee directs their head toward a specific area of ​​interest as instructed, the spatial orientation of their head forms a fixed spatial topological association with that physical area. The main control terminal invokes the first panoramic camera, which is fixedly set in the assessment scene, and uses concurrent threads to continuously acquire images of the examinee's head area and image data that can be used to extract facial key points. In this embodiment, the first panoramic camera serves as the data acquisition source for pure head pose estimation, used to generate a baseline head pose feature vector; the second head-mounted first-view stabilized camera is used to acquire the examinee's first-view operation footage and provides first-view evidence in the structured debriefing video, but is not used as a data source for global head pose estimation.

[0109] S204, the main control terminal records the start and end timestamps of the guidance command issuance and termination. Based on these timestamps, it extracts a sequence of image frames containing the examinee's head region from the continuously acquired video stream of the first panoramic camera. To eliminate dynamic blur and head posture transition interference caused by the examinee's head orientation change, the main control terminal removes transitional image frames at the beginning and end of the image frame sequence within a set time period, extracting image frames in the middle time period of a stable orientation state, and using these as the calibration reference frame sequence for this specific region of interest. In the specific parameter configuration, considering the physiological response of normal adults to head orientation change and neck rotation in response to auditory commands, the value range of the set time period can be set from 0.5 seconds to 1.0 seconds. By setting this time threshold, the video sequence is hard-cropped at both ends to avoid including non-target orientation images in the intermediate state of deflection within the reference calculation scope.

[0110] The main control terminal establishes a mapping between the identification information of each region of interest and the corresponding calibration reference frame sequence, and saves it in physical memory. This includes the calibration reference frame sequence, the polygon mask of the region of interest, and the coordinates of the subsequent header pointer landing point. All of these correspond to the image coordinate system of the first panoramic camera or the scene reference coordinate system mapped from that image coordinate system. This guidance process eliminates the need for absolute measurement of the physical three-dimensional coordinate system. By recording the examinee's facial visual data in a standard operating posture, it provides underlying data support for the main control terminal to subsequently extract the reference head posture feature vector.

[0111] The calculation of baseline head pose features specifically includes the following steps:

[0112] S205, the main control terminal sequentially reads the calibration reference frame sequence corresponding to each region of interest from physical memory. The main control terminal performs preprocessing operations on each frame of the calibration reference frame sequence. In specific implementation, the main control terminal calls the face detection algorithm to locate the bounding box of the facial region in the image, crops the image based on the bounding box, and scales the cropped facial image to the set spatial resolution according to the input requirements of the model network structure, for example, scaling the image size to 224×224 pixels. The main control terminal normalizes the pixel channel values ​​of the image to a specified floating-point range, specifically by dividing the pixel value by 255 to map it to the continuous range [0,1], or by performing Z-score standardization based on the channel mean and standard deviation obtained from a large-scale image dataset. For the underlying face bounding box detection logic, those skilled in the art can use a general cascaded classifier or a multi-task convolutional neural network algorithm, which are well-known technologies in the field and will not be elaborated here.

[0113] S206, the main control terminal inputs the preprocessed facial image tensor into the head pose estimation model for forward inference calculation. This head pose estimation model is configured to output three-dimensional Euler angles based on facial key points or head pose image features. The three-dimensional Euler angles include yaw, pitch, and roll angles. The head pose estimation model can be implemented using convolutional neural networks, Media Pipe Pose, Media Pipe Face Mesh, or equivalent algorithms capable of outputting head pose Euler angles. Before deployment, the model can be trained using a set of face pose samples with head pose annotations, or a general head pose estimation model trained on a publicly available dataset can be used directly.

[0114] S207, the main control terminal encapsulates the three-dimensional numerical values ​​output by the attitude regression head into a mathematical feature vector describing the spatial pointing state. Let the set of image frames included in the currently processed calibration reference frame sequence be... For any one of the preprocessed images The head pose feature vector output by the model's forward inference is represented as:

[0115] ;

[0116] in, Indicates the first The head yaw angle corresponding to the frame image. Indicates the first The head pitch angle corresponding to the frame image. Indicates the first The head roll angle corresponding to the frame image; It is a positive integer, and , This represents the total number of image frames contained in the calibration reference frame sequence.

[0117] S208. Due to physiological factors such as breathing or muscle rhythms, the human head inevitably experiences slight tremors when maintaining a stable head orientation. This can easily lead to numerical fluctuations in a single frame image during network inference. To reduce errors caused by such high-frequency environmental noise, the main control terminal performs an arithmetic mean aggregation calculation on all generated head pose feature vectors in the calibration reference frame sequence corresponding to a specific region of interest, generating a reference head pose feature vector representing the spatial orientation of that region of interest.

[0118] The formula for calculating the arithmetic mean aggregation is expressed as follows:

[0119] ;

[0120] in, This indicates the three-dimensional spatial reference datum when the candidate stably points their head toward a specific region of interest (AOI); This indicates the first frame in the calibration reference frame sequence. Frame to the The head pose feature vectors of the frames are summed; the values ​​of AOI include at least the monitor parameter area. and aseptic working area The main control terminal calculates respectively. and The two are then stored in a configuration table in memory to serve as the coordinates of the subsequent header pointing to the landing point. Spatial reference for determining the AOI region to which it belongs.

[0121] See attached document Figure 3 The specific steps for calculating the coordinates of the head pointing to the landing point, quantizing the dwell time in the AOI region, and coupling the attention and operation chain include:

[0122] S301, the main control terminal acquires the real-time image of the examinee's head region captured by the first panoramic camera and inputs it into a pure head pose estimation model for feature extraction. In specific implementation, the main control terminal calls the face detection module to crop the region of interest on the face and scales it to a preset dimension as model input. The facial key points extracted by the pure head pose estimation model include at least the corners of the eyes, the tip of the nose, and the chin. Based on the distribution and spatial displacement relationship of the extracted facial key points on the two-dimensional plane, the main control terminal, combined with a general head 3D deformation model, uses the Perspective-n-Point pose estimation algorithm to calculate the pitch angle, yaw angle, and roll angle of the head relative to the coordinate system of the first panoramic camera, and simultaneously calculates the displacement parameters of the head relative to the coordinate system of the first panoramic camera; based on the displacement parameters, the main control terminal determines the spatial position of the examinee's head center point, nose tip point, or facial 3D model reference point in the coordinate system of the first panoramic camera.

[0123] The three spatial angles mentioned above together constitute the three-dimensional Euler angle data describing the examinee's real-time head posture. The pure head posture estimation model can be implemented using Media Pipe Pose, MediaPipe Face Mesh, or equivalent algorithms, without requiring eye-tracking hardware. When the first panoramic camera cannot stably extract the examinee's facial key points, the main control terminal can perform auxiliary estimation or interpolation compensation for the head orientation based on human skeletal key points, head and shoulder contour direction, head detection box orientation, scene target recognition results from the second head-mounted first-view stabilized camera, or historical posture trajectories from adjacent frames.

[0124] S302, the main control terminal, based on the perspective projection geometric model, back-projects the three-dimensional head posture onto the two-dimensional image plane to estimate the coordinates of the head's pointing point. Specifically, the main control terminal retrieves the camera intrinsic parameter matrix configured during the system initialization phase, constructs a ray projection equation using the perspective projection geometry model, and calculates the spatial intersection point of a ray extending from the candidate's head center point, nose tip point, or facial 3D model reference point along the head orientation vector with a set two-dimensional reference plane. The two-dimensional reference plane is set as the image plane of the first-channel panoramic camera, or the assessment scene reference plane mapped from the coordinate system of the first-channel panoramic camera image. The main control terminal defines the coordinate data of this spatial intersection point as the coordinates of the head pointing towards the landing point. ,in This represents the approximate point of attention orientation obtained based on head orientation, not the actual eye fixation point in the context of eye tracking. In this embodiment, attention allocation evaluation is based on the head orientation surrogate, without requiring the actual eye fixation point as a necessary input.

[0125] S303, the main control terminal calls the predefined region of interest (ROI) polygon mask of the assessment scenario in memory. In this embodiment, the predefined ROI polygon mask includes at least the monitor parameter area corresponding to the monitor screen border. and the aseptic working area corresponding to the edge contour of the aseptic cloth. The coordinates of the head pointing to the landing point are calculated frame by frame. The main control terminal uses the ray-crossing test algorithm to determine... Whether it is contained within the boundary of a specific polygon mask, so as to realize the frame-by-frame determination of the AOI region to which the head points to the landing point.

[0126] S304, the main control terminal performs a quantitative calculation of dwell time based on the frame-by-frame AOI region determination results. The time window for a specific operation phase is determined by at least one of the following methods: a predefined directed graph of specialized operations, video action recognition results, device logs, mannequin event logs, or manual annotation by the examiner. Within the time window of a specific operation phase, the main control terminal accumulates the time for different head pointing and landing point states. To obtain an accurate accumulation step size, the main control terminal extracts the video timestamp of the currently processed video frame. and the video timestamp of the previous adjacent video frame. By subtracting the values, we can obtain the physical period duration corresponding to the video frame.

[0127] ;

[0128] in, This indicates the physical duration of the video frame. This indicates the video timestamp of the currently processed video frame. This represents the timestamp of the previous adjacent video frame.

[0129] When the header of a certain video frame points to the coordinates of the landing point Falling into the monitor parameter area At that time, the main control terminal accumulates the frame period duration to... When the head points to the landing point coordinates Falling into the aseptic working area At that time, the main control terminal accumulates the frame period duration to... When the head points to the landing point coordinates If the frame does not fall within any predefined AOI region, the main control terminal will increment the frame period duration to [the specified value]. At the end of this operation phase, the main control terminal outputs a vector of cumulative dwell time:

[0130] ;

[0131] in, This represents the vector of cumulative stay duration. This indicates the candidate's understanding of the monitor parameter area. The cumulative length of stay This indicates the candidate's familiarity with the aseptic work area. The cumulative length of stay This indicates the cumulative time a candidate spends in other areas when their head is pointing towards the target area but does not fall into the critical assessment target area.

[0132] S305, the main control terminal determines the operation events that occur during the current operation phase based on at least one of the following: video motion recognition results, hand key point trajectory, instrument target detection results, mannequin event log, equipment operation log, or examiner manual annotation. and the operation event Mapping to a predefined directed graph of specialty operations The corresponding operation node in The main control terminal will vectorize the dwell time. Operational events identified within the same operational phase Binding, generating attention and behavior data pairs The main control terminal will combine attention and behavioral data. The input is a predefined set of deviations for matching. The predefined set of deviations includes at least the following two types of deviation rules:

[0133] First, in high-risk operational events If in the preset observation window before the event occurs If so, it is determined that the candidate did not check the monitor parameters before the high-risk operation. Insufficient observation;

[0134] Second, in the core stage of aseptic operation, if If so, the candidate is deemed to be in a sterile operating area. Insufficient focus. Among these, and The criteria are determined by assessment standards, expert experience evaluation results, or statistical results of historical qualified samples, and can be configured separately for different specialty operation items.

[0135] When attention and behavioral data are compared When any deviation rule is met, the master control terminal generates an attention mismatch flag. ,in This indicates the candidate's attention allocation and current operational events. There is a mismatch; the main control terminal will Corresponding operation events Operation Nodes Video frame interval, video timestamp The points are then linked to the deduction anchor points and stored in the evaluation data pool.

[0136] See attached document Figure 4 The specific steps for detecting aseptic barrier breaches during the assessment process include:

[0137] S306, during the preset aseptic operation process, the main control terminal calls the fourth-channel aseptic area overhead close-up camera to continuously collect video stream data including the examinee's hands and the aseptic operating table. The main control terminal performs preprocessing operations on the collected video frames, cropping and scaling the original image to a set input dimension, such as adjusting the image size to 512×512 pixels, and normalizing the pixel channel values ​​to a specified floating-point range, for example, by subtracting the statistical mean of the channel pixels and dividing by the standard deviation, in order to construct a three-dimensional RGB image tensor that conforms to the input standard of deep learning models.

[0138] S307, the main control terminal inputs the preprocessed image tensor into the instance segmentation model with loaded network weights for forward inference computation. This instance segmentation model employs a deep convolutional neural network architecture based on region proposal. The model's internal hierarchical structure includes a feature extraction backbone network, a feature pyramid module, a region candidate network, and a mask generation branch. The feature extraction backbone network is configured to extract the spatial geometric and semantic features of the image layer by layer; the feature pyramid module is connected to the backbone network and is used to fuse multi-scale feature maps; the region candidate network generates candidate target bounding boxes based on the fused feature maps; the mask generation branch consists of a fully convolutional network and is configured to perform pixel-level classification computation within the candidate target bounding boxes.

[0139] The initial business results output by the forward inference of this model are the probability tensors corresponding to the area where the sterile gloves worn by the examinee are located, and the probability tensor corresponding to the coverage area of ​​the sterile drape. To meet the input requirements of subsequent logical operations, the main control terminal uses a preset confidence threshold (e.g., 0.5) to binarize the above probability tensors, generating the corresponding discrete binary image, that is, obtaining the final glove contour mask. and sterile area mask .

[0140] In one alternative implementation, the instance segmentation model can be trained using a clinical practice visual sample set containing annotations of medical gloves and sterile area boundaries, and configured to output a glove contour mask. Mask of sterile area .

[0141] S308, the main control terminal obtains the glove outline mask corresponding to the current video frame. Mask of sterile area In the underlying memory, morphological Boolean logic operations are performed on the aforementioned binary mask. The main control terminal then processes the sterile area mask. Perform a pixel-by-pixel inversion operation to generate a complement mask representing the non-sterile contamination space. .

[0142] Subsequently, the main control terminal calculates the glove contour mask. With complement mask The intersection of the two sets of elements is determined by the following core condition:

[0143] ;

[0144] in, This represents the glove outline mask. Indicates the mask for the sterile area. This represents the complement mask for non-sterile regions. This represents the area of ​​the intersection connected region formed after the glove outline enters the non-sterile area. This indicates the preset threshold for the area exceeding the boundary.

[0145] In practical implementation, considering that the underlying image segmentation network often has edge noise or outlier pixels when extracting target edges, strictly adopting the logic that a non-empty pixel intersection indicates a violation could easily lead to misjudgment by a single noisy pixel. To enhance the system's fault tolerance in complex clinical lighting environments, the main control terminal extracts the pixel area of ​​the connected components within the intersection region after calculating it. At that time, the main control terminal establishes that the aseptic boundary crossing judgment condition is met. The value can be set between 20 and 50 pixels depending on the spatial resolution of the input image.

[0146] To avoid misjudgments caused by single-frame segmentation noise, the main control terminal also checks whether the sterile boundary crossing judgment condition is continuously satisfied for a preset number of frames. Or whether the duration has reached the preset duration threshold. When the area threshold condition is met, and at least one of the consecutive frame count condition or duration condition is met, the main control terminal confirms the sterile barrier breach signal. .

[0147] S309, When the above-mentioned judgment condition is met, the main control terminal determines that the candidate's glove outline crosses the geometric boundary of the aseptic operation area, constituting a violation of the aseptic principle. In this assessment scoring rule, if the projected area of ​​the hand after wearing sterile gloves crosses the boundary of the sterile area and meets the preset continuous judgment condition, it is considered a sterile barrier breach event. The preset continuous judgment condition includes at least one of a continuous frame count condition or a duration condition. In response to this morphological conflict, the main control terminal triggers a sterile barrier breach signal. ,in This is a Boolean violation signal indicating a breach of the sterile barrier. The sterile barrier breach signal will be triggered by the main control terminal. The first video frame that meets the continuous determination criteria or the peak frame of the out-of-bounds area that caused the violation, and the video timestamp. and corresponding operation nodes Data binding is performed, and the structured violation event is stored in the system's evaluation data pool as the basic visual data support for subsequent execution of the scoring mask. The scoring mask is as follows:

[0148] for Each operation node in ,make ;in, Indicates starting from the current operation node The set of all downstream operation nodes reachable from the starting point along the directed dependency edges. Indicates the operation node The corresponding simulated human physiological improvement data cannot be converted into therapeutic efficacy scores. for Any operation node in it.

[0149] See attached document Figure 6 The event anchor-driven phase in the cross-clock domain mapping of offline physiological packets specifically includes the following steps:

[0150] S401, after the practical assessment task is completed, the main control terminal imports the offline physiological data package generated by the high-fidelity medical mannequin from the data storage module. This offline physiological data package records various vital sign data of the mannequin throughout the assessment process, and includes the sampling time of the mannequin's internal clock. ,in This represents the original sampling time under the internal clock of the high-fidelity medical simulator. Simultaneously, the main control terminal acquires multiple video streams and video frame sequences asynchronously collected by the system during the assessment period. .

[0151] S402, the main control terminal identifies physically observable operation action anchor frames in the video frame sequence. The main control terminal performs frame-by-frame parsing of the first or second video stream to detect predefined physical displacement events. In this embodiment, the predefined physical displacement events are selected from the following groups of actions: chest compression displacement apex, CRRT flow rate knob adjustment start, vascular access clamping / opening / closing, and airway device insertion / removal. These physical displacement events possess significant physical characteristics, making them suitable as calibration sources for subsequent multimodal spatiotemporal alignment. For each identified anchor frame... The main control terminal records the corresponding video timestamp. .

[0152] S403, for motion anchor frame In the specific recognition process, the main control terminal calls a visual analysis algorithm to capture the operational actions in the video stream. In one implementation, the main control terminal inputs consecutive video frames one by one into a preset pose estimation model. This pose estimation model adopts a deep convolutional neural network architecture with multi-resolution feature fusion. The main control terminal crops and scales the input image to a tensor of a set dimension (e.g., a 256×256×3 RGB image tensor).

[0153] Internally, image data first undergoes a downsampling backbone network to extract shallow features, then enters a fusion module composed of multiple parallel feature extraction branches with different resolutions. These branches interact with each other at multiple scales through strided convolution and bilinear interpolation upsampling. Finally, the model's prediction branch outputs a two-dimensional spatial probability heatmap consistent with the number of keypoints. The main control terminal searches for pixel maxima in this heatmap to extract the two-dimensional spatial coordinates of hand skeletal keypoints (such as the fingertip of the index finger and the palm), and calculates their spatial displacement amplitude between adjacent video frames. The action anchor point recognition model can be implemented using a hand skeletal keypoint detection model, a pose estimation model, or a dense optical flow algorithm; its output is used to determine the video frame where the peak of the action occurs.

[0154] In another implementation, the master control terminal uses a dense optical flow algorithm to directly calculate the pixel motion vector field between consecutive video frames to obtain optical flow variation characteristics. The master control terminal searches for local maxima of this displacement amplitude or optical flow variation in the time series. When the extracted amplitude value reaches a set dynamic threshold, it detects the motion peak during the operation. The master control terminal then locates the specific video frame where this motion peak is located as a candidate anchor frame. .

[0155] S404, to ensure strong coupling between visual actions and physiological results in the time dimension, the main control terminal further performs time-series analysis on the offline physiological data packets. The main control terminal monitors the time series of specific physiological parameters or mannequin event channels in the offline physiological data packets, identifying physiological sampling points that originate from physically observable operational actions. For example, in chest compression events, it identifies instantaneous increases in compression depth; in CRRT flow rate knob adjustment events, it identifies flow rate adjustment sampling points in the mannequin or equipment logs; and in vascular access clamping / opening / closing events, it identifies corresponding access state change sampling points. The sampling time of the mannequin's internal clock at the location of this physiological sampling point is obtained. Subsequently, the main control terminal uses the operation event occurrence log sequence as the retrieval interval and employs dynamic time warping or cumulative absolute difference minimization methods to coarsely match the video action peak sequence with the physiological sampling mutation sequence.

[0156] After causal verification, the main control terminal will finally determine the anchor frame. Physiological sampling time corresponding to the same physical event in the offline physiological package Strict pairing, constructed as anchor pairs During the anchor pairing process, the master control terminal performs event type consistency checks, temporal causality checks, and residual consistency checks on candidate anchor points. When the residual of a candidate anchor point pair in the affine drift model fitting exceeds a preset residual threshold, the master control terminal removes the candidate anchor point pair and does not consider it a valid anchor point pair. The establishment of this anchor point pair provides benchmark data support for subsequent time alignment of heterogeneous data under conditions where external hardware timecode synchronization signals are lacking.

[0157] See attached document Figure 6 The specific steps for the affine drift model construction and adaptive degradation compensation stage are as follows:

[0158] S405, the main control terminal acquires the structured anchor point pairs extracted through cross-clock domain matching:

[0159] ;

[0160] in, Indicates the first Anchor point pair, Indicates the number of valid anchor pairs. Indicates the first The video timestamp corresponding to each video anchor frame. Indicates the first The sampling time of each physiological anchor point under the simulated human internal clock.

[0161] In asynchronous data acquisition environments lacking external synchronization hardware, there is an inherent physical difference in the crystal oscillator frequency between the camera equipment and the mannequin's motherboard. As the assessment duration accumulates, this physical difference causes an approximately linear timing drift between the video timeline and the mannequin's internal clock.

[0162] To calibrate this time error, the main control terminal uses multiple extracted anchor point pairs as a reference to fit an affine drift model for time alignment in memory. The main control terminal extracts the video timestamps from each anchor point pair. Compared with the corresponding original simulated human internal sampling time Solving for the scale factor in affine transformations With intercept parameter Under normal conditions where multiple valid anchor point pairs are extracted, the master control terminal uses the least squares method to fit the linear relationship between the two time series. During the fitting process, the master control terminal iteratively filters valid anchor point pairs based on the fitting residuals, removing anchor point pairs whose residuals exceed a preset residual threshold as abnormal anchor points, and re-solves for the scale factor using the retained valid anchor point pairs. With intercept parameter .

[0163] The main control terminal uses the number of valid anchor point pairs retained after removing abnormal anchor points as the final number of valid anchor point pairs; when there are multiple valid anchor point pairs retained, a multi-anchor point affine drift model is performed; when there is only one valid anchor point pair retained, the main control terminal will... Set to 1, and calculate using this single anchor point. This causes the offline physiological data packets to undergo constant-level time shift compensation. The confidence level of this downgraded compensation result is lower than that of the multi-anchor fitting result, and it is recorded in the structured evidence triplet. When the number of valid anchor pairs retained is zero, the master terminal does not perform strong coupling scoring between the offline physiological data packets and the video timeline, but only outputs a low confidence prompt or transfers to the manual review process.

[0164] S406, when the master control terminal obtains the scale factor based on at least one valid anchor point pair With intercept parameter At that time, the main control terminal uses the established affine drift model to perform time axis compensation on the offline physiological data packets generated by the high-fidelity medical simulator.

[0165] The main control terminal iterates through all sampled data records in the offline physiological data package. Let the sampling time of the original simulated human internal clock corresponding to a certain record be... The main control terminal converts it to a unified time axis using the following standard affine mapping formula:

[0166] ;

[0167] In this compensation formula, The timeline representing the alignment of offline physiological data packets with the video frame sequence after affine drift model compensation; This represents the clock drift scaling factor caused by crystal oscillator differences; This parameter represents the time offset between the internal clock of the simulator and the time axis of the main control terminal. This represents the original sampling time under the internal clock of the high-fidelity medical simulator.

[0168] The main control terminal calibrates discrete physiological sampling points with drift errors to a unified time axis by performing the above linear mapping operation. superior.

[0169] S407, the main control terminal, based on a unified data foundation that completes time mapping, performs structured encapsulation processing on the underlying heterogeneous data. For the pre-set practical assessment of specialist nurses, the main control terminal extracts the physiological sampling data that falls within the mapped time interval based on the time span occupied by the operation in the video stream, and constructs a structured evidence triplet in the underlying storage module. The logical structure formula of this triplet is as follows:

[0170] .

[0171] S408, the main control terminal assigns corresponding business judgment attributes to specific elements within the constructed structured evidence triple. Indicates the first The structured evidence objects corresponding to each clinical procedure; Indicates the first The clinical procedure is performed within the start and end frame intervals of the video frame sequence, where The starting video frame, To terminate a video frame; Represents a unified timeline All physiological sample subsets falling within the corresponding time range of this video frame interval; Indicates the system's response to the first... The confidence score is given by the spatiotemporal mapping results of the clinical operation. To ensure the computability of the confidence score, the master terminal determines the number of effective anchor points when fitting the affine drift model. and the residual variance calculated by the least squares method Perform quantitative calculations. The specific calculation formula can be:

[0172] ;

[0173] in, and All are positive weighting coefficients. Indicates the number of valid anchor pairs. This represents the variance of the residuals in the affine drift model fit. A higher number of effective anchor pairs and a smaller residual variance result in a higher confidence score. The main control terminal achieves multimodal data binding between discrete causal actions and physiological feedback at the software computation level by outputting structured evidence triples containing the above elements.

[0174] See attached document Figure 4 The specific steps for the response delay extraction and quantization stage on a unified time axis include:

[0175] S501, when an effective mapping relationship has been established between offline physiological data packets and the video timeline, the main control terminal uses the previously constructed unified timeline. Extracting the pathological inflection points of preset clinical assessment events In practical assessment scenarios, pathological inflection points Mapping to a unified timeline The offline physiological data packets are determined and meet preset combinations of blood pressure, blood oxygen, or heart rate thresholds. For example, when systolic blood pressure is lower than a preset blood pressure threshold and the duration reaches a preset sampling window, the main control terminal determines the unified timeline time mapped from the threshold start sampling point as [the specified time]. .

[0176] The main control terminal can also call the close-up camera of the third-channel monitor. Nearby video clips are used to verify whether the corresponding abnormal vital signs are displayed on the monitor screen, serving as video evidence for decision lag assessment and structured debriefing; video footage from the close-up camera on the third monitor does not replace offline physiological data packets as pathological inflection points. The main data source.

[0177] S502, the main control terminal is on a unified timeline The above search corresponds to the pathological inflection point Effective intervention actions by specialist nurses. The main control terminal pre-stores pathological event types and sets of effective intervention events. The mapping relationship between them, where Indicates the current pathological inflection point One or more effective intervention events that match the pathological event type The set of events. For example, for pathological events such as decreased blood pressure, the set of effective intervention events. This may include checking vascular access, reducing ultrafiltration rate, clamping relevant tubing, performing fluid resuscitation, or initiating emergency response procedures; for pathological events related to decreased blood oxygenation, an effective set of intervention events is required. This may include checking the airway device, adjusting the oxygen flow rate, re-secured the airway device, or performing suctioning.

[0178] The main control terminal retrieves the structured evidence triples constructed in the aforementioned steps:

[0179] ;

[0180] If an operation event associated with the current pathological inflection point is successfully matched within the preset assessment time window. And the operation event Belongs to the set of effective intervention events Then, the video frame interval in the triplet is analyzed. The main control terminal will determine the video frame representing the initial action of this effective intervention as... And obtain the video timestamp of the starting video frame on the timeline of the main control terminal. .

[0181] S503, the main control terminal performs time-axis differential calculations to calculate the candidate's decision response delay. If the main control terminal fails to retrieve a set of valid intervention events within the preset assessment time window. Operation events The corresponding structured evidence triple indicates that the candidate omitted this operation, generating a deduction item for the omitted operation, and then the event is recorded. Recorded as an outlier exceeding the preset maximum tolerance time. When successfully acquired... The mathematical formula for calculating the decision response delay is as follows:

[0182] ;

[0183] in, Indicates decision response delay. This indicates the video timestamp of the effective intervention start frame on the main control terminal's timeline. This indicates the moment when the pathological inflection point is mapped onto a unified time axis; because and All data are based on a unified time reference in the main control terminal. This differential calculation eliminates the inherent crystal clock drift error in the independent acquisition process of asynchronous hardware devices, ensuring the fairness of the system's evaluation in the time dimension.

[0184] S504, the main control terminal calculates the decision response delay. Combined with preset decision delay threshold Determine the points deduction. If The main control terminal then generates a decision lag deduction item. and will With pathological inflection point Effective intervention in the starting frame Decision response delay And the corresponding video frame interval binding. In this embodiment, The assessment criteria for specialist nurses' practical skills, as entered during the system initialization phase, are determined accordingly; in the assessment of ICU specialist nurses' CRRT operation and hemodynamic crisis management... It can be set to 60 seconds.

[0185] See attached document Figure 4 The specific steps for the semantic judgment stage regarding decision-making delays, the aggregation of violations, and the blocking of information include:

[0186] S505: The main control terminal aggregates the evaluation results corresponding to the current operation node into a set of violation events. The set of violation events includes at least the penalty points for decision delay. Note the mismatch markers and signals of sterile barrier disruption .in, Used to indicate the timeout of intervention response after the occurrence of a pathological inflection point. Used to represent attention and behavior data pairs Falling into a predefined offset set Used to represent a glove outline mask Passing through the sterile area mask The defined sterile boundary.

[0187] S506, the main control terminal responds to sterile barrier breach signals. The triggered state is a directed graph of predefined specialty operations. China and Israel current node All downstream operation nodes starting from Apply a scoring mask and apply... Each operation node in , and its corresponding Set to False; when Operation nodes in satisfy At that time, the system ignores this operation node when calculating the final score. The efficacy score gained from the corresponding physiological improvement data of the simulated human is only allowed to accumulate for process violation scores and corresponding deductions. This step does not issue end-stage pathological parameter packages to the high-fidelity medical simulated human, nor does it change the simulated human's physiological feedback logic; it only blocks efficacy scores at the scoring logic level. In this embodiment, the sterile barrier disruption signal... After being triggered, Each operation node of Keep it set to False until the end of this assessment task.

[0188] The S507, after the assessment task is successfully completed, aggregates and persistently stores the data in the evaluation data pool. The main control terminal integrates structured evidence triples. Note the mismatch markers Aseptic barrier disruption signal Delayed decision-making deduction item Decision response delay In addition to multimodal event alignment records, a comprehensive evaluation report is constructed. When no effective mapping relationship is established between the offline physiological data packets and the video timeline, the comprehensive evaluation report records the low-confidence state of the corresponding assessment stage and marks the relevant video segments and offline physiological data as objects for manual review. The main control terminal stores the packaged comprehensive evaluation report in the system database and displays it visually on the terminal interface.

[0189] Through the above steps, this embodiment of the invention achieves multimodal heterogeneous data cleaning, clock domain alignment, and coupled evaluation at the business logic level at the software architecture level, thereby improving the objectivity of nursing practice assessment.

[0190] See attached document Figure 7 The specific steps for configuring the directed graph and scoring mask for predefined specialty operations include:

[0191] The S601 main control terminal, based on clinical specialty operation standards, constructs a predefined directed graph of specialty operations in memory to represent standard operating procedures. .in, This represents the set of operation step nodes included in this assessment task. This represents the set of directed dependencies between nodes in an operation; a chain of nodes... Pointing to node The directed edge represents the node For nodes The predecessor node. Operation events Operation step nodes in a directed graph Establish a mapping relationship between them, and It only represents a set of directed dependencies and is not used to represent operation events.

[0192] S602, the main control terminal is a predefined directed graph of specialized operations. Each node in Related Boolean variables . Used to represent nodes Whether the corresponding simulated human physiological improvement data can be converted into therapeutic effect scores. At the start of the assessment, the main control terminal will check all nodes... Initialize to True.

[0193] S603, when the main control terminal is at a certain operation node A signal indicating disruption of the sterile barrier was detected at the location. At that time, the main control terminal extracts the data from the operation node. All downstream operation nodes starting from and to Each operation node in , and its corresponding Set to False. Indicates the operation node Starting from the set of directed dependencies All nodes that can be reached by directed edges in the array. for Any operation node in it.

[0194] S604, when Any operation node in satisfy At that time, the main control terminal ignores this operation node when calculating the final score. The therapeutic effect score resulting from the corresponding simulated human physiological improvement data is only retained for the process violation score and the decision-making delay score. Note the mismatch markers and signals of sterile barrier disruption The corresponding deduction records.

[0195] See attached document Figure 7 The specific steps for blocking the transmission of therapeutic effects include:

[0196] S605, the main control terminal is based on the information obtained in the previous steps. The logic for blocking the transmission of therapeutic effects is implemented, in which... Represents a directed graph of predefined specialty operations. Any operation node in; when belong And due to the signal of sterility barrier disruption Set as At that time, the main control terminal executes the logic to block the transmission of efficacy scores at that node. In clinical nursing procedures, improvements in physiological indicators are based on standardized operational procedures. If the examinee is operating at the node... Signals of sterile barrier disruption triggered at the site Even if all downstream operating nodes Objectively, this triggered an improvement in the vital signs of the high-fidelity medical simulator, affecting all downstream operation nodes. The corresponding physiological efficacy score is also not included in the total score. The main control terminal calculates the node score using the following piecewise function. Final therapeutic effect score:

[0197] ;

[0198] in, This indicates that the main control terminal uses a physiological sampling subset from the structured evidence triples. The original physiological efficacy score is calculated by the internal quantitative scoring function. The internal quantitative scoring function calculates the original physiological efficacy score based on at least one or more of the following factors: whether the target physiological indicator recovers to the normal range within a preset time window, the extent of recovery, the speed of recovery, and the duration of stability after recovery. Represents a node The final efficacy score after scoring masking. Through the above-mentioned blocking mechanism, the system implements the clinical evaluation rule at the scoring logic level that "the efficacy score is not increased due to subsequent improvement in vital signs after the aseptic barrier is breached."

[0199] See attached document Figure 5 The structured video generation stage specifically includes the following steps:

[0200] After completing the multimodal spatiotemporal coupling evaluation and scoring blocking logic, the S606 main control terminal initiates the structured video generation program. This provides the examiner with traceable review data, based on a unified timeline. Extract data slices containing key events. The main control terminal traverses the system evaluation data pool, filtering out items with decision-making delays that incur deductions. Note the mismatch markers Aseptic barrier disruption signal Or therapeutic effect transmission blocking markers Operation action nodes.

[0201] After identifying the target node, the main control terminal retrieves the structured evidence triples bound to the aforementioned node:

[0202] ;

[0203] Analyze the video frame intervals in the triplet. The target video segment is extracted from the original continuous assessment video stream in the storage module. For the same deduction anchor point, the main control terminal uses a unified timeline. Simultaneously extract video segments from five video feeds within the corresponding time range, and generate single-stream or multi-grid review videos according to a preset format. In a preferred embodiment, the target video segment is extracted from 3 seconds before the deduction anchor point to 3 seconds after the deduction anchor point.

[0204] S607: The main control terminal reconstructs the heterogeneous evaluation and analysis data into a visualization graphics layer and performs temporal alignment and fusion with the target video segments. The main control terminal extracts discrete physiological sampling subsets contained within the triples. Render it in memory as a timeline that follows a uniform timeline. An extended two-dimensional physiological waveform layer. Simultaneously, the main control terminal will... , , , , Attention and behavioral data Metadata such as aseptic operation boundary crossing detection boxes are encapsulated as text or geometric annotation layers. The main control terminal operates based on a unified timeline. The physiological waveform layer and annotation layer are overlaid on the base frame of the target video segment to achieve the mapping and display of action execution, attention state and physiological feedback in the same screen space.

[0205] The S608, driven by the main control terminal, performs compression output on the superimposed multi-track data, generating the final structured review video file. The main control terminal indexes the generated short videos at each node and links them to the comprehensive evaluation report, storing them in the system's backend database for retrieval by the front-end terminal. Each structured review video file is associated with the deduction list. At least one deduction anchor point is associated in the above-mentioned structured video to ensure that the evaluation results are traceable and verifiable. Through the generation and output of the above-mentioned structured video, this embodiment of the invention establishes an intuitive presentation form of multimodal data analysis results and realizes a data closed loop in the practical assessment and evaluation business process.

[0206] The specific steps for the feedback phase of teaching include:

[0207] S701, the teaching feedback layer reads the deduction lists of multiple candidates by assessment batch. The deduction list includes deductions for decision-making delays. Note the mismatch markers Aseptic barrier disruption signal and decision response delay .

[0208] S702, the teaching feedback layer uses a predefined directed graph of specialized operations. The operation steps are plotted on the horizontal axis, and the violation types are plotted on the vertical axis. The frequency of points lost for each violation type at each operation step is counted to generate a heatmap of group capabilities. Violation types include at least decision delay, attention mismatch, breach of sterile barrier, and treatment efficacy blockade.

[0209] S703, Teaching Feedback Layer Note that mismatch markers should be statistically analyzed separately to obtain data from different operational stages. Insufficient observation Lack of focus and The distribution results show an excessively high percentage. The teaching management system identifies teaching blind spots based on these distribution results and pushes corresponding structured review video clips to the teaching feedback interface.

[0210] S704, the teaching feedback layer will include a heat map of group capabilities, The distribution statistics, deduction list, and structured review video files are linked together to create an index that allows examiners to trace back from high-frequency deduction points in the group ability heatmap to specific candidates, specific timestamps, and specific video clips.

[0211] Specific application examples:

[0212] This embodiment uses CRRT (Continuous Renal Replacement Therapy) and sudden hypotension management as assessment scenarios, and combines specific data flow and numerical calculations to illustrate the multimodal data alignment and evaluation process of this system in an asynchronous hardware environment.

[0213] During the initial assessment phase, the main control terminal uses a monotonic clock to record the system startup time. (Defined as relative time zero point 0.0s). The five-channel USB camera is set to a frame rate of 30FPS, corresponding to a theoretical frame interval of 33.3ms. Due to bus bandwidth limitations, random frame drops occur. The high-fidelity mannequin's internal motherboard uses an independent crystal oscillator, and its recording time scalar is set to... .

[0214] During the assessment, the main control terminal extracted two physical displacement anchor point frames through visual analysis: Anchor point 1, which corresponds to the connection of the venous catheter, and the visual algorithm identified the peak frame of the hand tightening action at the interface. The system's monotonic clock reads the actual timestamp of the frame written to memory as follows: Retrieving the simulated human logs, the corresponding time of the stress mutation sampling point is... Anchor point 2, i.e., the operation of adjusting the ultrafiltration rate knob, detects the end frame of knob rotation. Its actual timestamp The corresponding internal settings of the simulator were changed at the following time. .

[0215] Based on formula The main control terminal substitutes the above anchor points to solve for the scale factor. With intercept : ; Solving This value reflects a clock frequency offset of approximately 555 ppm between the analog crystal oscillator and the main control terminal. After establishing a unified timeline mapping model, any simulated human physiological sampling time can be determined. All can be passed Map to the video timeline.

[0216] Event of connecting extracorporeal circulation The duration is 15 seconds. The main control terminal extracts the Euler angles of the head frame by frame and calculates the ray landing point. Statistical analysis shows that the ray points to the monitor's parameter area. Frame cumulative duration Pointing to the aseptic operation area Cumulative duration Other areas The system retrieves the deviation rules for this operation node: if Set threshold Substitute the values ​​into the calculation: The system meets the judgment criteria and generates an attention mismatch flag. And associate it with the corresponding deduction item.

[0217] During the aseptic dressing change of the central venous catheter, the image segmentation network outputs a glove mask. With sterile cloth mask In a certain video frame, the system detected the glove outline and its complement in the non-sterile area. Pixel intersections occur. Calculate the area of ​​the connected components at the intersection. Pixels larger than the preset out-of-bounds area threshold The pixel, and this state continues to meet the condition for the next 16 consecutive frames. The system triggers a sterile barrier breach signal. The subsequent query operation is a directed graph. , will the current node The location of the efficacy approval mark for all downstream nodes (including pipeline flushing, blood drainage, etc.): .

[0218] The mapped unified timeline At this point, the simulated human offline data packet showed that the systolic blood pressure dropped below 80 mmHg, serving as a pathological inflection point. The system retrieves corresponding valid intervention events, and the video stream records the nurse's actions. The ultrafiltration rate was reduced at that time. The decision response delay was calculated. The result is less than the set decision delay threshold. No decision-making lag penalty was triggered. In the subsequent stage, the simulated human's blood pressure rose to 110 mmHg, corresponding to an original physiological improvement score of 15 points. This was because the preceding stage had already triggered a sterile barrier disruption signal. and When set to False, the system executes the efficacy score blocking logic, and outputs the final score for that node as follows: This scoring system ensures that the principle of aseptic technique is prioritized in the evaluation process.

[0219] In the system effectiveness verification, practical assessment data from 200 critical care nurses were selected for a comparative experiment. The control group system was configured with an external hardware timecode generator for hardware synchronization between multiple cameras and the simulated human, and scoring was achieved by manually reviewing physiological curves and monitoring video recordings. The experimental group adopted the system architecture of this invention, without external synchronization signal input, and performed software compensation processing through concurrent execution of multiple asynchronous USB video stream acquisitions and an affine drift model.

[0220] See attached document Figure 8 This figure illustrates the clock drift calibration effect during the assessment process. The horizontal axis represents the assessment time (minutes), the left vertical axis represents the relative clock alignment error (milliseconds), and the right vertical axis represents the cumulative number of dropped frames. The discrete dots in the figure indicate the original clock error before model compensation. As the assessment time accumulates to 45 minutes, due to differences in hardware crystal oscillator frequencies, the original timestamp residual shows a linear accumulation trend, with a maximum offset of approximately 1800 milliseconds. The stepped dashed lines in the figure record the cumulative random frame drops caused by limited USB bus bandwidth, totaling over 200 frames. The solid lines in the figure represent the calibration residual curve after compensation using the affine drift model of this embodiment. The data shows that, under the condition of random frame drop interference, the compensated residual curve remains near the zero baseline, and the absolute error of the system's multimodal alignment throughout the entire cycle is controlled within ±18 milliseconds. This error level is lower than the time response threshold of conventional manual clinical judgment, meeting the time accuracy requirements for coupled evaluation of operational actions and physiological data.

[0221] In terms of system performance statistics, the control group, constrained by dedicated synchronization equipment and manual review processes, experienced a longer average analysis output time per workstation, and a false negative rate of 12.5% ​​for transient aseptic operation boundary violations. After adopting this system, the experimental group, based on the intersection calculation of directed graphs and morphological masks, reduced the false negative rate for aseptic operation boundary violations to 1.5%. Simultaneously, by utilizing asynchronous decoupling and automated scoring data aggregation, the output latency of evaluation results for a single assessment sample was shortened to less than 3 seconds, verifying the reliability and data processing efficiency of the solution in the business logic execution phase.

Claims

1. A multimodal data spatiotemporal coupling evaluation method for nurses' practical skills assessment, characterized in that: include: In response to the assessment start command, the system receives video streams from multiple camera devices, extracts video frame sequences containing image frames, assigns video timestamps to the image frames, and establishes a video timeline. Acquire offline physiological data packets containing internal clock sampling times from the output of a high-fidelity medical manikin; Identify physical events in the video frame sequence according to predefined physical displacement events, and perform action peak detection on the video frame sequence corresponding to the physical events to locate the anchor frame of physically observable operation actions; Extract the physiological sampling time corresponding to the physical event from the offline physiological data packet, and pair the video timestamp of the anchor frame of the physically observable operation action with the physiological sampling time as an anchor pair; Based on the anchor point pair fitted with an affine drift model, the internal clock sampling time is mapped to a unified time axis with the video time axis as the reference. On the unified time axis, the video frame interval and physiological sampling subset corresponding to the operation event are determined, and the operation event is subjected to AOI attention and operation chain coupling relationship determination, sterile barrier determination and decision response delay determination to generate evaluation results.

2. The method according to claim 1, characterized in that, Assigning the video timestamp to the image frame includes: Record the system startup time and establish the video timeline based on the system startup time; The video stream is received in an asynchronous, non-blocking manner; The system monotonic clock state at the current moment when the image frame is written to physical memory is captured, the system monotonic clock state at the current moment is used as the video timestamp, and the video timestamp is bound and stored with the image frame; When random frame drops occur in the video stream, the position of the subsequent arriving image frame in the video timeline is determined based on the video timestamp corresponding to the subsequent arriving image frame.

3. The method according to claim 1, characterized in that, The peak motion detection includes: Visual analysis is performed on the video frame sequence to extract at least one feature quantity from the spatial displacement amplitude of key points of the hand bones and the optical flow variation features between consecutive image frames in the video frame sequence; Search for local maxima of the feature quantity; When the local maximum reaches a set dynamic threshold, the video frame corresponding to the local maximum is used as the anchor frame of the physically observable operation action. The set dynamic threshold is used to determine whether the local maximum constitutes an operation action peak.

4. The method according to claim 1, characterized in that, The determination of the coupling relationship between AOI attention and the operation chain includes: Obtain the region of interest for a predefined assessment scenario; The head pose feature vector is extracted from the image frame to determine the approximate landing point of the attention orientation, the region of interest to which the approximate landing point of the attention orientation belongs is determined, and the dwell time is accumulated to form a dwell time vector; Obtain the operation events that occur during the current operation phase, and bind the dwell time vector to the operation events as attention and behavior data pairs; Obtain two types of deviation rules contained in the predefined deviation set, wherein the deviation rules include the operation event and the threshold determined by the assessment specifications, expert experience calibration results or historical qualified sample statistical results; An attention mismatch marker is generated when the attention and behavior data pair satisfies any of the deviation rules.

5. The method according to claim 4, characterized in that, Before extracting the head pose feature vector, the method further includes: Establish a polygon mask for the region of interest corresponding to the region of interest; Output the guidance command and record the start and end timestamps corresponding to the guidance command; Based on the start and end timestamps, a sequence of image frames containing the candidate's head region is extracted from the video stream. Transitional image frames at the beginning and end of the head posture deflection process in the sequence of image frames containing the candidate's head region are removed. The beginning and end time intervals are 0.5 seconds to 1.0 seconds. Extract image frames from the intermediate time period to generate a calibration reference frame sequence; The calibration reference frame sequence is input into the head pose estimation model to generate a reference head pose feature vector.

6. The method according to claim 1, characterized in that, The aseptic barrier determination includes: Extract the glove outline mask and sterile area mask from the image frame; Perform a pixel-by-pixel inversion operation on the sterile area mask to generate a complement mask for the non-sterile area; Obtain the intersection region of the glove outline mask and the complement mask of the non-sterile region, and extract the pixel area of ​​the connected components in the intersection region as the out-of-bounds area; When the out-of-bounds area exceeds the out-of-bounds area threshold determined according to the spatial resolution of the image frame and the continuous state meets the preset continuous determination condition, a sterile barrier destruction signal is triggered. The preset persistence determination conditions include at least one of the consecutive frame count condition and duration condition.

7. The method according to claim 1, characterized in that, The affine drift model includes a scale factor and an intercept parameter; Fitting the affine drift model includes: The anchor point pairs are used as candidate anchor point pairs. Fitting verification is performed on the candidate anchor point pairs to generate fitting residuals. Candidate anchor point pairs whose fitting residuals exceed the residual threshold are eliminated. The remaining candidate anchor point pairs are used as valid anchor point pairs. The residual threshold is used to characterize the upper limit of the allowed time alignment deviation of the candidate anchor point pairs. When there are multiple valid anchor point pairs, perform a multi-anchor point affine drift model fitting operation. When the number of effective anchor point pairs is one, the scale factor is set to one, and the intercept parameter is calculated based on the single effective anchor point pair, and a constant-level time translation compensation operation is performed. When the number of valid anchor point pairs is zero, strong coupling scoring between offline physiological data packets and the video timeline is not performed.

8. The method according to claim 1, characterized in that, The decision response delay determination includes: Read physiological parameter time series from the offline physiological data package mapped to the unified time axis, the physiological parameter time series including at least one of blood pressure, blood oxygen and heart rate; When the time series of physiological parameters meets the physiological parameter threshold combination and sampling window corresponding to the clinical assessment event, the starting sampling point of exceeding the limit is determined as the pathological inflection point; Obtain the set of effective intervention events corresponding to the pathological inflection point, and locate the effective intervention start frame in the video frame sequence that corresponds to the set of effective intervention events; The decision response delay is extracted by performing a difference calculation based on the video timestamp corresponding to the effective intervention start frame and the time when the pathological inflection point is mapped onto the unified time axis; When the decision response delay exceeds the decision delay threshold determined by the professional nurse practical assessment standards, a decision lag deduction item is generated.

9. The method according to claim 6, characterized in that, After generating the evaluation result, the method further includes: Obtain a predefined directed graph of specialty operations, which includes operation nodes, directed edges, efficacy score permission flags corresponding to the operation nodes, and original physiological efficacy scores corresponding to the operation nodes. When it is determined based on the evaluation results that any of the operation nodes triggers the sterile barrier destruction signal, all downstream operation nodes starting from the operation node are extracted as a set of downstream operation nodes. The set of downstream operation nodes includes the operation node and all downstream operation nodes reachable from the operation node along the directed edge. Set the efficacy score permission flag corresponding to each operation node in the downstream operation node set to False, and block the transmission of the original physiological efficacy score corresponding to each operation node in the downstream operation node set; Based on the evaluation results, the video timestamps corresponding to the violations are extracted as deduction anchors. The violations include the sterile barrier breach signal. A deduction list containing the deduction anchors is generated. Video clips are extracted from the deduction list, and the video clips are combined with the annotation layer to generate a structured review video file. Multiple deduction lists are aggregated to generate a group capability heatmap.

10. A multimodal data spatiotemporal coupled evaluation system for nurses' practical skills assessment, characterized in that: The method for performing any one of claims 1 to 9 includes a hardware layer and a logic processing layer; The hardware layer includes a main control terminal and a multi-channel camera device; The main control terminal is used to acquire offline physiological data packets containing internal clock sampling times output by the high-fidelity medical simulator; The logic processing layer is deployed on the main control terminal, and the logic processing layer includes a multimodal perception terminal layer, an edge computing processing layer, an intelligent evaluation layer, and a teaching feedback layer; The multimodal sensing terminal layer is used to receive video streams collected by the multi-camera device; The edge computing processing layer is used to extract video frame sequences containing image frames from the video stream, assign video timestamps to the image frames, establish a video timeline, identify physical events according to predefined physical displacement events and locate anchor frames of physically observable operation actions, fit an affine drift model, and map the internal clock sampling time to a unified timeline with the video timeline as the reference. The intelligent evaluation layer is used to perform the multimodal spatiotemporal coupling evaluation and generate evaluation results; The teaching feedback layer is used to generate a deduction list, structured review video files, and group ability heatmaps based on the evaluation results.

Images