Operating room nurse osce multi-modal intelligent assessment method and system for three-dimensional target

Abstract

The application discloses a kind of " attitude-knowledge-skill " three-dimensional target-oriented operating room nurse OSCE multi-modal data intelligent evaluation method and system.The core of this method is: by deploying camera, microphone, motion and physiological sensor and other multi-source equipment in OSCE examination station, the behavior, voice, operation and physiological data of examinee under simulated operation scene are synchronously collected;The system first presets the basic weight of three-dimensional evaluation dimension according to the type of examination task, and then through the unique dynamic weight self-adaptive mechanism, the performance state of examinee is analyzed in real time, when specific states such as " skill bottleneck ", " knowledge deficiency " or " attitude stress " are identified, the weight of each dimension is dynamically adjusted, the intelligentization and individualization of evaluation strategy are realized;Subsequently, the weighted features are analyzed in depth using the multi-modal fusion model based on attention mechanism, and finally the fine quantification score, diagnostic feedback report and individualized improvement suggestion of the three dimensions of attitude, knowledge and skill are output.

Description

Technical Field

[0001] This invention relates to the field of intelligent assessment technology in medical education, and in particular to an automated assessment method and system for objective structured clinical assessment (OSCE) of operating room nurses based on multimodal data fusion and artificial intelligence. Background Technology

[0002] Objective Structured Clinical Examination (OSCE) has been widely used to assess the clinical competence of operating room nurses. It comprehensively examines nurses' practical skills by simulating a series of clinical scenarios. Traditional OSCE assessment models heavily rely on the examiner's subjective observation and on-site scoring, a human-driven approach with significant limitations. First, assessment results are easily influenced by the examiner's personal experience, fatigue level, and subjective preferences, making it difficult to standardize scoring criteria and ensuring fairness and consistency. Second, traditional assessment forms often only provide general scores, failing to capture and analyze the specific behavioral details of nurses during procedures, such as the precision of hand movements, emotional state during communication, and the logic of emergency decision-making. This results in coarse feedback and limited teaching guidance value.

[0003] With technological advancements, some research has begun to explore the use of video analytics or single-type sensor data (such as motion sensors) to aid in assessments. However, these methods are typically limited to simple analysis of the "skills" dimension, failing to achieve a comprehensive assessment of multi-dimensional abilities such as "attitudes" and "knowledge." They often create data silos, lacking deep fusion and correlation analysis of different modalities (such as video, audio, and physiological signals), and are unable to simulate the complex cognitive process of human examiners making comprehensive judgments. More importantly, most existing technology-assisted assessment systems employ fixed assessment models and weights, failing to dynamically adjust according to the different characteristics of the assessment task or the candidate's real-time performance, resulting in rigid assessment strategies that are ill-suited to the complex and ever-changing demands of real-world operating room scenarios.

[0004] Therefore, the field of OSCE assessment for operating room nurses urgently needs an intelligent assessment scheme that can achieve multi-dimensional, automated, and refined analysis, and has dynamic adaptability, in order to overcome the subjectivity and inefficiency of traditional methods and the limitations of existing technology-assisted schemes, so as to truly achieve objective, accurate, and efficient assessment that aligns with the three-dimensional goals of "attitude-knowledge-skills". Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for fully automated, refined, and quantifiable intelligent assessment of the "attitude-knowledge-skills" three-dimensional goals of operating room nurses. The core of this invention lies in constructing an intelligent system capable of perceiving the scene, understanding the state, and dynamically adjusting the assessment strategy. Through deep fusion of multimodal data and a unique dynamic weight adaptive mechanism, it simulates and surpasses the assessment capabilities of human experts, not only outputting objective scores but also providing diagnostic feedback with high educational value.

[0006] To achieve the above objectives, the specific technical solution adopted by the present invention is as follows: An intelligent multimodal data assessment method for operating room nurses' OSCE (Operating System Care) oriented towards the three-dimensional goals of "attitude-knowledge-skills" includes the following five interrelated, sequentially executed steps in its core process: S1: Multimodal data acquisition and synchronization In a physically constructed, highly simulated OSCE test station (such as a sterile instrument table or a simulated operating room), a heterogeneous, multi-source sensor network is deployed. This network specifically includes: Visual perception unit: Employs an RGB-D camera (such as Microsoft Azure Kinect) to not only capture color video streams for behavior analysis, but also accurately track the three-dimensional spatial relationship between the operator, instruments, and sterile areas through depth information.

[0007] Auditory perception unit: Deploys a microphone array to capture all voice interactions between nurses and simulated patients, surgeons, and scrub nurses, and uses beamforming technology to enhance the audio of the target speaker.

[0008] Motion sensing unit: The nurse wears a data glove with an integrated inertial measurement unit (IMU) on her hands to accurately capture the six-degree-of-freedom spatial motion trajectory, attitude, acceleration and angular velocity of the hands and handheld instruments at a high frequency (≥100Hz).

[0009] Physiological sensing unit: Nurse wears a wrist-worn ECG / skin conductance sensor to continuously monitor their ECG signals and skin conductance activity, which is used to quantify their stress level and emotional state.

[0010] Attention perception unit: Using a head-mounted or screen-based eye tracker, nurses' gaze focus, fixation duration, and saccade path are tracked to analyze their attention allocation strategies.

[0011] All of the aforementioned sensor modules are connected to a central synchronizer, which uses hardware triggering or High Precision Network Time Protocol (NTP) to stamp each frame of data with a uniform and precise microsecond-level timestamp, ensuring the consistency of subsequent multimodal data fusion on the timeline.

[0012] S2: Task-aware weight initialization The system has a pre-built "Assessment Task-Weight" mapping library. Before the assessment begins, the administrator assigns the task type to each assessment station (e.g., "Routine Instrument Delivery," "Intraoperative Massive Hemorrhage Emergency Management," "Preoperative Communication with Anxious Patients"). Based on this type, the system automatically loads the pre-defined three-dimensional base weight vector `Wbase = {Wa, Wk, Ws}` for that task type. An example of the pre-defined strategy is shown below: The "Routine Instrument Transfer" task focuses on the skill and precision of the operation, with preset weights of `Ws=0.6, Wk=0.3, Wa=0.1`.

[0013] The task "Emergency Management of Intraoperative Massive Hemorrhage" emphasizes the accurate application of emergency plan knowledge and teamwork and composure under high pressure, with preset weights of `Wk=0.4, Wa=0.4, Ws=0.2`.

[0014] The task of “Communicating with Anxious Patients Before Surgery” focuses on humanistic care and communication skills, with preset weights of `Wa=0.7, Wk=0.2, Ws=0.1`.

[0015] This `Wbase` serves as the initial baseline for evaluation and will be dynamically adjusted in subsequent steps.

[0016] S3: Fine-grained behavioral feature extraction This step processes the synchronous data stream acquired by S1 in parallel, using a pre-trained AI model to extract quantifiable behavioral features: Video feature extraction: Using open-source libraries such as OpenPose or MediaPipe, or by training a custom CNN model, extract the 2D / 3D keypoint coordinates of the operator's whole body and hands from RGB video in real time.

[0017] Based on the keypoint sequence, calculate the smoothness of operation (such as the jerk of joint movement, i.e., the derivative of acceleration), the amplitude of movement, etc.

[0018] Using object detection models such as YOLO or Faster R-CNN, surgical instruments, sterile dressings, and sterile area boundaries are identified. By calculating the spatial relationship between instruments and sterile areas, the number of aseptic principle violations (such as non-sterile parts of instruments crossing the boundary of sterile areas) is automatically counted.

[0019] Using facial expression recognition models trained on FER2013 or AffectNet datasets, we analyzed nurses' facial movement units and extracted the probability values ​​of expressions such as tension, focus, and smiling as features.

[0020] Audio feature extraction: Use an ASR engine such as Google Speech-to-Text or the open-source model Whisper to convert speech streams into text in real time.

[0021] The text is matched with preset key process steps, standardized reporting terms, and a medical terminology database to assess the accuracy and completeness of the reported content.

[0022] Simultaneously, signal processing is performed on the original audio to extract acoustic features such as fundamental frequency (reflecting intonation), speech rate (syllables / second), spectral center of gravity, and pause frequency, quantifying the calmness and friendliness of the communication.

[0023] Motion feature extraction: After filtering (such as Kalman filtering) and transforming the coordinates of the IMU data, the average speed, acceleration, and total trajectory length of the hand movement are calculated (to measure the efficiency of movement).

[0024] Perform a Fast Fourier Transform (FFT) on the acceleration signal to analyze its power in the high-frequency range (e.g., 8-12Hz), which serves as a quantitative indicator of operational jitter.

[0025] Physiological and eye-movement feature extraction: R-wave detection is performed on electrocardiogram signals to calculate time-domain (e.g., SDNN) and frequency-domain (e.g., LF / HF power ratio) indices of heart rate variability (HRV) as an objective quantification of psychological stress levels.

[0026] Process eye-tracking data to generate fixation sequence and heat map, calculate the first fixation time, total fixation duration and number of fixations of nurses on areas of interest (AOI) such as key instruments, drug labels, and vital signs monitors, and evaluate their observation efficiency and vigilance.

[0027] S4: Dynamic weight adaptive adjustment This step is crucial for achieving intelligent and personalized assessment. The system incorporates a real-time state recognition machine, whose input is the feature stream continuously output from step S3. The state recognition machine has clearly defined pre-defined decision logic: "Skill Bottleneck" status: When the system detects that the length of the hand movement trajectory significantly exceeds the average value (too many redundant movements) and the high-frequency shaking power exceeds the preset threshold, it determines that the nurse is facing difficulties in operational skills.

[0028] "Knowledge Deficiency" Status: When the key operation steps identified by ASR are incorrect or missing, or when video analysis finds that the number of violations of core aseptic principles exceeds the threshold, it is judged as a lack of solid professional knowledge or improper application.

[0029] "Attitude stress" state: When the LF / HF ratio of HRV increases sharply (sympathetic nerve excitation) and the synchronous audio characteristics show abnormally fast speech rate, increased fundamental frequency and abnormal pauses, it is judged as an emotional stress state caused by tension.

[0030] Once the state recognition machine determines that the examinee has entered a certain state, the dynamic weight adjustment module immediately adjusts `Wbase` according to a predefined strategy and generates `Wfinal`. If identified as a "skill bottleneck": the weight of `Ws` is increased (e.g., by 0.2), while the weights of `Wk` and `Wa` are decreased accordingly.

[0031] If the condition is determined to be "knowledge deficient": the weight of `Wk` is increased (e.g., by 0.2), and the system will focus on its compliance with the process in the subsequent time period.

[0032] If the condition is classified as "attitude stress", the weight of `Wa` is significantly increased (e.g., by 0.3). This assesses the individual's emotional regulation ability, and minor skill errors can be tolerated to a certain extent.

[0033] S5: Multimodal Fusion and Intelligent Evaluation Feedback All features extracted in S3 (forming a high-dimensional feature vector) are concatenated with the dynamic `Wfinal` weight vector obtained in S4 and input together into a pre-trained multimodal Transformer fusion model.

[0034] The model's attention mechanism, guided by `Wfinal`, automatically learns and assigns higher attention scores to features more relevant to the current high-weight dimensions. For example, when `Wa` has a high weight, the model pays more attention to audio acoustic features and facial expression features; when `Ws` has a high weight, the model pays more attention to motion trajectories and jitter features.

[0035] The model ultimately outputs a composite vector, which is then decoded by subsequent fully connected layers as follows: 1. Three-dimensional quantitative scoring: `ScoreA` (attitude), `ScoreK` (knowledge), `ScoreS` (skills), and a comprehensive score based on `Wfinal` weighting.

[0036] 2. Diagnostic Feedback Report: The report not only lists the score, but also clearly points out the problem, such as: "At T=12:35, the transfer of the sharp object was not carried out in the neutral zone, violating the safety principle (deduction of points in the knowledge dimension)", and automatically associates it with the corresponding video clip timestamp.

[0037] 3. Personalized Improvement Suggestions: Based on the diagnostic results, the system automatically matches and recommends resources from the teaching resource database. For example, it may recommend "Advanced Suturing Techniques Video Tutorials" to those experiencing "skill bottlenecks" and "Operating Room Stress Management and Communication Virtual Simulation Module" to those experiencing "attitude stress."

[0038] Through the closed-loop execution of the above five steps, this invention achieves a comprehensive, objective, accurate, and highly instructive intelligent assessment of the capabilities of operating room nurses.

[0039] The above technical solution can bring about the following three outstanding technical effects: 1. This invention represents a leap from "static assessment" to "dynamic perception assessment," significantly improving the accuracy and personalization of evaluations. Traditional OSCE assessments use fixed scoring sheets, making it impossible to adjust the assessment focus based on the examinee's real-time state. This invention, through its unique "dynamic weight adaptive mechanism," enables the system to "read between the lines" like a human expert. Specifically, by analyzing multimodal data streams in real time (such as motion trajectory jitter, voice tremor, and heart rate variability), the system proactively identifies whether the examinee is in a state of "skill bottleneck," "knowledge deficiency," or "attitude stress," and dynamically adjusts the assessment weights of the three dimensions of attitude, knowledge, and skills accordingly. For example, when the system detects severe hand tremors and redundant motion trajectories through IMU data, it determines it as a "skill bottleneck" state and automatically increases the weight of the skill dimension, concentrating assessment resources on accurately diagnosing operational deficiencies. This data-driven dynamic weight adjustment ensures that each assessment is a "personalized diagnosis" for a specific examinee in a specific state, rather than a uniform "assembly line judgment," greatly improving the accuracy of assessment results and the relevance of teaching guidance.

[0040] 2. This invention overcomes the limitations of single-modal data by achieving a comprehensive quantitative assessment of complex three-dimensional capabilities through deep fusion. Existing technologies for assisting assessment often rely on single data types such as video or motion, making it difficult to fully capture nurses' comprehensive abilities in real-world scenarios. This invention constructs a complete intraoperative behavioral data profile by deploying a heterogeneous sensor network (RGB-D camera, microphone array, IMU data gloves, physiological sensors, and eye trackers) and performing millisecond-level time synchronization. More importantly, it employs a multimodal Transformer fusion model based on an attention mechanism. This model can intelligently associate features from different modalities guided by dynamic weights. For example, when the system needs to assess "teamwork attitude," the model simultaneously considers the standardization of language in audio data, eye contact and body orientation in visual data, and emotional stability reflected in physiological data, assigning higher weights to these cross-modal features through an attention mechanism. This deep fusion technology achieves, for the first time, the objective quantification of subjective dimensions such as "attitude" and the precise analysis of the interaction between "knowledge-skills-attitude," solving the long-standing problem of comprehensive multidimensional competency evaluation in nursing education.

[0041] 3. A closed-loop system of "assessment-diagnosis-intervention" has been constructed, transforming the assessment site into an efficient teaching scenario. Traditional assessments often stop at providing scores, with feedback that is crude and delayed. This invention achieves "both scoring and diagnosis" through refined feature extraction and intelligent report generation. The system not only outputs three-dimensional scores but also generates diagnostic reports containing timestamps of specific behavioral segments, such as "At T=12:35, the instrument transfer trajectory exhibits three unnecessary loops (skill dimension, insufficient motion economy)." More importantly, the system can automatically link to a teaching resource library based on the diagnostic results, pushing customized improvement plans to examinees, such as pushing a "microsurgical suturing stability training" virtual simulation module to examinees with large operational jitters. This technological effect transforms the assessment process itself into an efficient learning process that accurately identifies weaknesses and provides immediate solutions, achieving a fundamental shift from simple "assessment" to "assessment-teaching integration," greatly improving training effectiveness. Attached Figure Description

[0042] Figure 1 This is a diagram of the overall architecture of the patented system of this invention. Figure 2 This is a diagram showing the overall method steps of the present invention. Detailed Implementation The following will refer to the appendices in the embodiments of the present invention. Figure 1 and Figure 2The technical solutions in the embodiments of the present invention are clearly and completely described herein. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1: Assessment of Emergency Management of Sudden Intraoperative Massive Hemorrhage 1. Scene Setup and Initialization Assessment task: The "patient" suffers a sudden massive hemorrhage during surgery.

[0044] System defaults: Based on the task type, the system initializes the three-dimensional base weights as `Wbase = {Wa=0.4, Wk=0.4, Ws=0.2}`, emphasizing emergency knowledge and teamwork attitude.

[0045] 2. Data Acquisition and Feature Extraction The candidate (nurse A) quickly retrieved the suction device, but her movements were hurried.

[0046] Sensor data: IMU: The hand movement speed is extremely fast, and the trajectory has slight redundancy.

[0047] Microphone: When reporting to the doctor, the speaker speaks quickly and in a rising tone, but the content is accurate ("Blood pressure dropped, 90 / 50 mmHg, fluids are being rapidly administered").

[0048] Physiological sensors: Heart rate jumped from 75 bpm to 110 bpm, and the HRV LF / HF ratio increased significantly.

[0049] Camera: Facial expression tense.

[0050] 3. Dynamic weight adaptive adjustment (core advantage) State Recognition: The system analyzes feature streams in real time. Although the motion data has flaws, the speech content is accurate. Most importantly, physiological data (high heart rate, abnormal HRV) and audio features (high speech rate, high pitch) form a strong correlation. The system comprehensively determines that the candidate is in a state of "attitude stress" rather than "skill bottleneck" or "knowledge deficiency".

[0051] Weight Adjustment: Based on a predefined strategy, the system dynamically increases the weight of `Wa` from 0.4 to `0.6`, while correspondingly decreasing the weights of `Wk` and `Ws`. This means that the system is now more focused on emotion management and teamwork than on the perfection of the actions.

[0052] 4. Multimodal fusion evaluation and feedback Evaluation results: Score A (Attitude): 85 points. Comment: "Able to maintain clear reporting and effective communication under pressure. However, physiological indicators show high levels of stress, requiring improved psychological adjustment." ScoreK (Knowledge): 95 points. Comments: "Familiar with emergency procedures, and the reported content is accurate." `ScoreS` (Skill): 78 points. Comment: "The smoothness of operation was affected by tension, and there were redundant actions." Feedback Report: The report highlighted its communication advantages under high pressure and expressed understanding for the tension. The system also recommended the "Psychological Resilience Training in High-Pressure Situations" module.

[0053] The advantage of this embodiment is that it demonstrates the system's ability to dynamically perceive and provide humanized assessment. Instead of completely rejecting the system due to operational flaws caused by the candidate's nervousness, it accurately identified the core issue as "attitude stress" and adjusted the assessment focus, providing fair and constructive feedback—something that fixed-rule assessment systems cannot achieve.

[0054] Example 2: Routine Instrument Transfer Assessment 1. Scene Setup and Initialization Assessment task: To pass various surgical instruments to the surgeon.

[0055] The system defaults to a base weight of `Wbase = {Wa=0.1, Wk=0.3, Ws=0.6}`, with a focus on evaluating operational skills.

[0056] 2. Data Acquisition and Feature Extraction Candidate (Nurse B) passes instruments.

[0057] Sensor data: IMU: The hand movement trajectory is smooth and economical, with low tremor power.

[0058] Camera (object recognition): Identifies that the name of the transmitted instrument matches the doctor's request.

[0059] Microphone: No verbal confirmation was given during the transmission (violating the "vote counting" system).

[0060] Eye tracker: Allows for timely switching of the line of sight between the instrument table and the surgical field.

[0061] 3. Dynamic weight adaptive adjustment State recognition: The system found that the skill characteristics (movement trajectory, shaking) were excellent and the knowledge characteristics (correct identification of equipment) were correct, but the attitude characteristics (lack of verbal communication) had a clear deficiency.

[0062] Weighting Adjustment: Due to the stable performance of the skills and knowledge dimensions, the system does not require significant weighting adjustments. The assessment will continue to be based on `Wbase`, but the deduction weight for the "communication attitude" sub-item will remain unchanged.

[0063] 4. Multimodal fusion evaluation and feedback Evaluation results: `ScoreS` (Skill): 95 points. Comment: "Stable, accurate, and economical operation." ScoreK (Knowledge): 90 points. Comment: "Accurate recognition of equipment." Score A (Attitude): 70 points. Comment: "There were 3 instances where verbal confirmation was not performed during the transfer of equipment, posing a safety hazard." Feedback Report: The report, through video playback, precisely marked the three times when verbal confirmation was not obtained. The system recommends learning the micro-course "Operating Room Safety Communication Standards".

[0064] The advantage of this embodiment is that it demonstrates the system's ability to make sophisticated diagnostics. Even in skills-based assessments, the system can still capture easily overlooked but crucial safety attitude issues (without verbal confirmation) through multimodal data (audio + video) and provide evidence-based feedback to guide improvement.

[0065] Example 3: Assessment of communication with patients experiencing preoperative anxiety 1. Scene Setup and Initialization Assessment task: To comfort an anxious patient who is about to undergo surgery.

[0066] System preset: base weight `Wbase = {Wa=0.7, Wk=0.2, Ws=0.1}`, core assessment of humanistic care and communication attitude.

[0067] 2. Data Acquisition and Feature Extraction The candidate (nurse C) communicates with the "patient".

[0068] Sensor data: Microphone: The voice was gentle and used empathetic statements ("I understand you must be nervous...").

[0069] Camera: Always maintain a smile and eye contact, and lean slightly forward.

[0070] Microphone (ASR): When explaining the preoperative fasting time, it was incorrectly stated as "8 hours" (it should actually be 12 hours).

[0071] Physiological sensors (simulated patient): showed that the "patient's" tension decreased after communication.

[0072] 3. Dynamic weight adaptive adjustment State recognition: The system recognized that the patient's attitudinal characteristics (tone of voice, facial expression, posture) were excellent and the communication was effective (the "patient's" physiological indicators improved). However, at the same time, there was a serious bias in the knowledge characteristics (incorrect delivery of key information).

[0073] Weight Adjustment: The system has determined that a "knowledge deficiency" status has been detected. Although this task emphasizes attitude, knowledge errors are related to patient safety and cannot be ignored. Therefore, the system dynamically increases the weight of `Wk` from 0.2 to `0.4`, amplifying the impact of the penalty for knowledge errors.

[0074] 4. Multimodal fusion evaluation and feedback Evaluation results: Score A (Attitude): 95 points. Comment: "Excellent empathy, good communication style, and able to effectively alleviate patient anxiety." ScoreK (Knowledge): 65 points. Comment: "The preoperative fasting period was incorrectly stated, which is a serious oversight in knowledge." `ScoreS` (Skills): Does not involve major point deductions.

[0075] Feedback Report: The report first highly praised the communicator's attitude, then clearly pointed out the factual errors and directly linked to the corresponding chapter in the "Preoperative Preparation Guidelines" document. The report emphasized: "Excellent communication is fundamental, but accurate knowledge is the bottom line for ensuring patient safety." The advantage of this embodiment is that it demonstrates the system's comprehensive ability to weigh factors in complex scenarios. It does not overlook factual errors due to a candidate's excellent communication skills, but rather dynamically adjusts weights to ensure that the assessment results both encourage strengths and unequivocally point out key weaknesses, guiding candidates towards well-rounded development.

[0076] Through the above three embodiments, the core advantages of the present invention are highlighted: 1. Dynamic Adaptability: The system weights (`Wfinal`) are not fixed, but dynamically adjusted based on the real-time identified candidate status (stress, bottleneck, deficiency), making the assessment more intelligent and fair.

[0077] 2. Comprehensiveness of assessment: Through multimodal data fusion, the system can simultaneously conduct in-depth analysis of "attitudes, knowledge, and skills" and discover complex problems that cannot be found in single-dimensional assessments.

[0078] 3. Value of the teaching closed loop: Each assessment result is accompanied by specific, data-supported diagnoses and personalized improvement suggestions, transforming assessment from an "endpoint" into "a link in the learning journey," greatly improving training effectiveness.

[0079] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal intelligent assessment method for OSCE (Operating System Care) of operating room nurses, oriented towards the three-dimensional goals of "attitude-knowledge-skills," characterized in that... Includes the following steps: S1: Multimodal Data Acquisition and Synchronization: In the constructed OSCE simulation test station, multiple heterogeneous sensor modules are deployed and run synchronously, including: an RGB-D camera for capturing overall operational behavior and facial expressions; a microphone array for acquiring voice interaction data with surgical team members; an inertial measurement unit (IMU) or data glove for accurately tracking the spatial motion trajectory of hands and instruments; a heart rate / skin conductance sensor for monitoring stress physiological state; and an eye tracker for analyzing attention allocation; all sensors are synchronized through a unified timecode generator to form a timestamped multimodal data stream; S2: Task-Aware Weight Initialization: The system predefines multiple typical OSCE assessment task scenarios and configures an initial three-dimensional weight vector `Wbase = {Wa, Wk, Ws}` for each scenario, where Wa, Wk, and Ws represent the basic weights of the attitude, knowledge, and skill dimensions, respectively. When a candidate enters a specific examination station, the system automatically loads the corresponding `Wbase` according to the task scenario. Specifically, for routine equipment delivery tasks, Ws > Wk > Wa is set; for emergency handling tasks, the weights of Wa and Wk are set to be greater than or equal to Ws; for doctor-patient communication tasks, Wa has the highest weight. S3: Fine-grained behavioral feature extraction: Parallel processing of the multimodal data stream in S1 to extract quantized features: Video features: A deep learning-based human pose estimation model was used to extract key points of the operator's body and calculate the operation smoothness index; an object detection model was used to identify the relative position of instruments and sterile areas and count the number of violations of sterility principles; a facial motion unit analysis model was used to extract facial expression features such as tension and focus. Audio features: The speech is converted into text using an automatic speech recognition engine and compared with a preset keyword library to evaluate the accuracy and completeness of the reported content; at the same time, acoustic features such as fundamental frequency, speech rate, and spectral centroid are extracted from the original audio signal to quantify the tone and calmness of the communication. Motion characteristics: Calculate the average acceleration, trajectory length, and jitter power of hand movements from IMU or data glove data to evaluate the accuracy and stability of the operation; Physiological and eye-tracking characteristics: Calculate the low-frequency to high-frequency power ratio in heart rate variability as an indicator of stress level; analyze fixation sequence and heat map of eye-tracking data to assess its attention allocation efficiency for key instruments and monitoring equipment; S4: Dynamic Weight Adaptive Adjustment: A state machine is constructed, whose input is the real-time feature vector extracted in S3, and whose output is the judgment of the candidate's current state. The state includes at least "skill bottleneck", "knowledge deficiency", and "attitude stress". The state machine has pre-set judgment logic, for example: when the hand movement tremor power exceeds the threshold and the operation fluency is lower than the threshold, it is judged as the "skill bottleneck" state; when the number of errors in the key process steps of speech recognition exceeds the threshold, it is judged as the "knowledge deficiency" state; when the heart rate variability index is abnormal and occurs simultaneously with abnormal speech rate and vibrato in the audio features, it is judged as the "attitude stress" state. According to the judged state, the system adjusts the evaluation weight in real time and generates the final weight `Wfinal`. The adjustment strategy is: when in the "skill bottleneck" state, increase Ws; when in the "knowledge deficiency" state, increase Wk; when in the "attitude stress" state, increase Wa. S5: Multimodal Fusion and Intelligent Evaluation Feedback: The multi-dimensional features extracted in S3 are concatenated with the `Wfinal` weight vector obtained in S4 and input into a pre-trained multimodal Transformer fusion model. This model utilizes its attention mechanism, guided by `Wfinal`, to weight the features of different modalities and time steps based on their importance, and finally outputs a comprehensive vector. This vector is decoded into: quantitative scores for three dimensions: attitude (ScoreA), knowledge (ScoreK), and skills (ScoreS); a diagnostic report containing specific deduction points, corresponding timestamps, and behavioral segments; and a personalized learning path suggestion linked to a specific chapter or simulation training module in the teaching resource database.

2. The method according to claim 1, characterized in that, The dynamic weight adjustment in step S4 is a continuous process. During the assessment, the system continuously evaluates the state and updates the weights based on the real-time feature flow. The `Wfinal` is a dynamic vector that changes over time.

3. The method according to claim 1, characterized in that, The sensor network in step S1 supports plug-and-play and modular deployment, and can flexibly adjust the type, number and location of sensors according to the spatial layout of the specific assessment site and the evaluation focus.

4. The method according to claim 1, characterized in that, The weight strategy library in step S2 can be edited and updated by the administrator according to the actual assessment outline to adapt to different training focuses and evaluation standards.

5. The method according to claim 1, characterized in that, The multimodal Transformer fusion model in step S5 is trained in a supervised manner using historical assessment data and expert scores. Its loss function is constructed as a weighted combination of a regression task for three-dimensional target scores and a classification task for key behavioral labels.

6. A system for implementing the method of any one of claims 1-5, characterized in that, include: The intelligent examination station hardware unit is a physically constructed simulated operating room environment that integrates the sensor network, data acquisition card, synchronization triggering device, and high-performance computing gateway described in claim 1. Data aggregation and preprocessing server: responsible for receiving raw data streams from various examination stations, and completing data cleaning, format unification, timestamp alignment, and secure storage; Feature computation and state recognition engine: Loaded with various pre-trained AI models, used to execute steps S3 and S4 of claim 1 in parallel, and output behavioral features and dynamic weights in real time; Core evaluation and feedback generator: Built-in multimodal Transformer fusion model, executes step S5 of claim 1, and generates the final evaluation results and report; Interactive user portal: Provides examiners with a real-time monitoring dashboard and scoring assistance interface, provides candidates with a personal report viewing interface and learning resource portal, and provides administrators with system configuration and data analysis dashboards.

7. The system according to claim 6, characterized in that, The system also includes a distributed data storage and analysis platform for long-term storage of multimodal data, evaluation results and model parameters of all assessment processes, and provides a data mining interface for group capability analysis, training effectiveness evaluation and self-iterative optimization of the evaluation model.

8. The system according to claim 6, characterized in that, The interactive user portal embeds key behavioral video clips automatically captured and marked by the system into the feedback report generated by the candidate. Candidates can directly click on the timestamp to replay their own operation video, audio, and synchronized physiological data curves at a specific moment.

Images