A psychological assessment method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a psychological evaluation method and device, electronic equipment and storage medium. The method synchronously collects facial video stream, eye movement data, voice signal and respiratory physiological signal and other data of a target patient; the facial video stream, eye movement data, voice signal and respiratory physiological signal are respectively subjected to feature extraction to generate corresponding expression features, eye movement features, semantic acoustic features and physiological features; the expression features, eye movement features, semantic acoustic features and physiological features are subjected to feature fusion based on preset multi-modal fusion weights to generate target fusion features, the eye movement features being given the highest fusion weight to effectively suppress the disguisability of facial expression; a preoperative emotional score is generated based on the target fusion features, a psychological evaluation result is generated based on the preoperative emotional score and a preset risk stratification rule, seamless mapping from bottom layer signal to clinical decision support is realized, and thus the recognition accuracy of disguised calm patients and the clinical reliability of preoperative psychological evaluation are improved.

Description

Technical Field

[0001] This application relates to the field of medical information technology, and in particular to a psychological assessment method, device, electronic device, and storage medium. Background Technology

[0002] Preoperative psychological assessment is an important part of perioperative medicine. Clinical studies have shown that preoperative anxiety and depression can activate the hypothalamic-pituitary-adrenal axis, leading to increased blood pressure, increased risk of arrhythmia, and impaired metabolism of anesthetic drugs. It can also increase the risk of intraoperative awareness and result in delayed wound healing and prolonged hospital stay after surgery.

[0003] Currently, commonly used preoperative psychological assessment methods in clinical practice mainly include subjective questionnaires and monomodal emotion recognition technology. Subjective questionnaires rely on patient self-reporting, and patients may provide inaccurate answers due to deliberate concealment or feigned composure, and real-time dynamic assessment is not possible. Monomodal emotion recognition technology mainly relies on facial key points and facial motion unit intensity to recognize facial expressions; however, facial expressions can be consciously controlled, and micro-expressions typically last less than 500 milliseconds, making them difficult to capture. These technologies still suffer from poor accuracy in preoperative psychological assessment and fail to accurately reflect the patient's psychological state. Summary of the Invention

[0004] In view of this, embodiments of this application provide a psychological assessment method, device, electronic device, and storage medium to solve the problem of poor accuracy in preoperative psychological assessment in related technologies.

[0005] According to a first aspect of this application, a psychological assessment method is provided, comprising: During the preoperative assessment period, multimodal raw data of the target patient are collected simultaneously; the multimodal raw data includes at least facial video stream, eye movement data, speech signals and respiratory physiological signals; Feature extraction was performed on facial video streams, eye-tracking data, speech signals, and respiratory physiological signals to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features. Based on preset multimodal fusion weights, facial expression features, eye movement features, semantic acoustic features, and physiological features are fused to obtain the target fused features; eye movement features are assigned the highest fusion weight. Preoperative emotion scores are generated based on target fusion features; Psychological assessment results are generated based on preoperative emotional scores and pre-defined risk stratification rules.

[0006] Optionally, the method further includes: Obtain the current anesthesia plan; Based on the results of the psychological assessment, the current anesthesia protocol was adjusted, and a target anesthesia protocol was determined.

[0007] Optionally, based on preoperative emotional scores and preset risk stratification rules, the generated psychological assessment results include: Obtain preset risk stratification thresholds and spoofed calmness markers; spoofed calmness markers are used to indicate that the target patient's facial expressions are inconsistent with their internal anxiety. The preoperative emotional score was compared with the preset risk stratification threshold to obtain the comparison results; and the initial risk level was determined based on the comparison results. In the presence of a deceptive calmness marker, the initial risk level is increased to become the target risk level. Based on the target risk level, a psychological assessment result is generated that includes the risk level and a prompt for feigning composure.

[0008] Optionally, the preoperative emotional assessment includes facial expression assessment and eye movement assessment; the method also includes: When the facial expression score corresponds to the calm category and the eye movement score corresponds to the anxiety category, a fake calmness marker is generated.

[0009] Optionally, the preoperative emotional assessment may also include a semantic assessment, and the method may further include: When the semantic score is in the calm or avoidance category and the eye movement score is in the anxiety category, generate a fake calmness tag.

[0010] Optionally, after extracting features from the facial video stream, eye-tracking data, speech signals, and respiratory physiological signals to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features, the method further includes: Obtain individual baseline multimodal features of target patients during the resting baseline period, and establish an individual emotional fingerprint model based on the individual baseline multimodal features; Based on the corresponding baseline parameters of each feature in the corresponding facial expression features, eye movement features, semantic acoustic features, physiological features, and personal emotional fingerprint model, the corresponding relative rate of change is calculated, and the corresponding relative rate of change is fused based on the preset multimodal fusion weights.

[0011] Optionally, after generating a preoperative emotion score based on target fusion features, the method further includes: Obtain multimodal raw data of the family members of the target patient's accompanying family members; A time-series correlation analysis was performed on the multimodal raw data of the target patient and the multimodal raw data of the family members of the target accompanying family members to calculate the anxiety influence coefficient of the target accompanying family members transmitting anxiety to the target patient; The anxiety impact coefficient is compared with the preset anxiety impact threshold. If the anxiety impact coefficient is greater than the preset anxiety impact threshold, the preoperative emotional score is increased. Based on the increased preoperative emotional score and the preset risk stratification rules, a psychological assessment result is generated.

[0012] Optionally, after generating a preoperative emotion score based on target fusion features, the method further includes: The preoperative emotional score, eye movement characteristics, physiological characteristics, and basic patient information of the target patient are input into the preset time-series prediction model, which outputs the intraoperative stress response prediction score, the predicted value of anesthesia induction dosage, and the intraoperative awareness risk level.

[0013] Optionally, feature extraction is performed on the facial video stream, eye-tracking data, speech signals, and respiratory physiological signals to obtain corresponding expression features, eye-tracking features, semantic acoustic features, and physiological features, including: Facial motion unit intensity, expression category, and micro-expression duration are extracted from facial video streams as expression features; Pupil diameter, pupil contraction latency, fixation point trajectory, fixation avoidance area, spontaneous blink frequency, and microjump frequency are extracted from eye movement data as eye movement features. Emotional vocabulary intensity, topic avoidance behavior, speech rate, pause patterns, and speech tremor frequency are extracted from speech signals as semantic acoustic features. Respiratory rate, respiratory depth, baseline skin conductance value, and skin conductance response amplitude are extracted from respiratory physiological signals as physiological characteristics.

[0014] Optionally, based on preset multimodal fusion weights, facial expression features, eye movement features, semantic acoustic features, and physiological features are fused to obtain the target fused features; eye movement features are assigned the highest fusion weight; including: Normalization was performed on facial expression features, eye movement features, semantic acoustic features, and physiological features to obtain facial expression modal features, eye movement modal features, semantic acoustic modal features, and physiological modal features; Based on preset multimodal fusion weights, facial expression modality features, eye movement modality features, semantic acoustic modality features, and physiological modality features are fused to obtain the target fused features.

[0015] Optionally, based on preset multimodal fusion weights, facial expression modal features, eye-tracking modal features, semantic acoustic modal features, and physiological modal features are fused to obtain the target fused features, including: Obtain quality assessment scores corresponding to facial expression modality features, eye movement modality features, semantic acoustic modality features, and physiological modality features; Based on the quality assessment score, the preset multimodal fusion weights are dynamically adjusted to obtain the target fusion weights corresponding to each modal feature; Based on the target fusion weights corresponding to each modal feature, facial expression modal features, eye movement modal features, semantic acoustic modal features, and physiological modal features are fused to obtain the target fusion features.

[0016] Optionally, based on target fusion features, the generated preoperative emotion score includes: The target fusion features are input into a preset emotion classifier, which outputs the probability distribution of emotion categories; the emotion categories include calm, mild anxiety, moderate anxiety, and severe anxiety. The target fusion features are input into a preset sentiment rating regression model, and the initial rating is output. The initial score is calibrated based on the probability distribution of emotion categories to obtain the preoperative emotion score.

[0017] According to a second aspect of this application, a psychological assessment device is provided, comprising: The data acquisition module is used to simultaneously acquire multimodal raw data of the target patient during the preoperative assessment period; the multimodal raw data includes at least facial video stream, eye movement data, speech signals and respiratory physiological signals; The feature extraction module is used to extract features from facial video streams, eye-tracking data, speech signals, and respiratory physiological signals to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features. The feature fusion module is used to fuse facial expression features, eye movement features, semantic acoustic features, and physiological features based on preset multimodal fusion weights to obtain the target fused features; eye movement features are assigned the highest fusion weight. The scoring generation module is used to generate preoperative emotion scores based on target fusion features; The assessment result determination module is used to generate psychological assessment results based on preoperative emotional scores and preset risk stratification rules.

[0018] According to a third aspect of this application, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement any of the aforementioned psychological assessment methods.

[0019] According to a fourth aspect of this application, a computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the above-described psychological assessment methods.

[0020] Compared with related technologies, the technical solution provided in this application has the following beneficial effects: The psychological assessment method provided in this application simultaneously collects multimodal raw data from the target patient, including facial video stream, eye movement data, speech signals, and respiratory physiological signals, during the preoperative assessment period, providing a data foundation for accurate psychological assessment. By extracting features from the facial video stream, eye movement data, speech signals, and respiratory physiological signals respectively, corresponding facial expression features, eye movement features, semantic acoustic features, and physiological features are generated, achieving quantitative representation of multidimensional emotional information. Based on preset multimodal fusion weights, facial expression features, eye movement features, semantic acoustic features, and physiological features are fused to generate target fusion features, with eye movement features assigned the highest fusion weight, effectively suppressing the masqueradability of facial expressions. A preoperative emotional score is generated based on the target fusion features, and a psychological assessment result is generated based on the preoperative emotional score and preset risk stratification rules, achieving a seamless mapping from underlying signals to clinical decision support. This transforms preoperative psychological assessment from a passive assessment relying on patient self-assessment and single facial expression recognition into an active assessment combining multimodal perception, weighted fusion, objective scoring, and risk stratification, significantly improving the accuracy of identifying patients feigning sedation and the clinical reliability of preoperative psychological assessment. Attached Figure Description

[0021] Figure 1 This is a flowchart of a psychological assessment method provided in one embodiment of this application; Figure 2 This is a flowchart of feature extraction provided in one embodiment of this application; Figure 3 This is a flowchart illustrating the determination of target fusion features provided in one embodiment of this application; Figure 4 This is a flowchart of obtaining a preoperative emotional score provided in one embodiment of this application; Figure 5 This is a flowchart of generating psychological assessment results provided in one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a psychological assessment device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0023] As described in the background section, the technical solution of this application aims to address the problem that the preoperative psychological assessment process relies on patient subjective self-assessment questionnaires and single-modal facial expression recognition, which leads to the inability to accurately identify the patient's psychological state and the susceptibility of assessment results to human manipulation. The following description, in conjunction with the accompanying drawings, will illustrate this point.

[0024] Optionally, the psychological assessment method in this application can be performed by a computer device. This computer device can be a standalone server, a terminal device (such as a tablet computer or a dedicated assessment terminal), or a processing module integrated into an anesthesiology information system. The computer device includes at least one processor and a memory, the memory storing computer program instructions, which the processor executes to implement the method steps of this application.

[0025] In one specific application scenario, the computer device is deployed in the anesthesiology outpatient clinic or preoperative waiting area, and collects multimodal data of patients through connected RGB cameras, microphones, respiratory sensors and skin conductance sensors.

[0026] To make the above-mentioned objectives, features and beneficial effects of this application more apparent and understandable, the technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 This is a flowchart of a psychological assessment method provided in one embodiment of this application.

[0028] Please refer to Figure 1 The psychological assessment method provided in this application includes: S1: During the preoperative assessment period, multimodal raw data of the target patient are collected simultaneously; the multimodal raw data includes at least facial video stream, eye movement data, speech signals and respiratory physiological signals.

[0029] It should be noted that the preoperative assessment period refers to a specific time window after the target patient enters the anesthesiology outpatient clinic, preoperative waiting area, or preoperative preparation room, and before entering the operating room. Optionally, the duration of this time window can be flexibly set according to the clinical scenario, such as 3 to 5 minutes, or even longer, 10 to 30 minutes. The specific duration can be adjusted according to the required assessment accuracy and clinical workflow. Collecting multimodal data within this period can best reflect the patient's true psychological state as surgery approaches, providing clinically valuable raw data for subsequent assessments.

[0030] Synchronous acquisition refers to the parallel acquisition of different types of multimodal raw data on the same timeline, and assigning them a unified timestamp. Since the sampling frequencies and data types of different modalities vary—for example, video is typically 30 frames per second, speech is typically 16 kHz, and respiratory physiological signals are typically 10-32 Hz—synchronization does not require all modalities to be acquired at the exact same sampling frequency. Instead, it requires that the data from each modality be temporally aligned. That is, after being identified by a unified timestamp, facial expressions, eye movements, speech content, and physiological responses acquired at the same time can be associated with the same point in time. Through synchronous acquisition, this application can capture the temporal correlation and consistency between different modalities. For example, it can detect the cross-modal temporal coupling relationship of judging whether the pupils are continuously dilated while the patient says "I'm not nervous," thus providing a temporally aligned data foundation for subsequent detection of feigned sedation.

[0031] The facial video stream is a continuous sequence of images captured by an RGB camera or an RGB-IR binocular camera, used for subsequent extraction of facial expression features and micro-expression features. Preferably, the camera is installed at the top edge of the consultation screen or directly in front of the preoperative assessment terminal, 0.6 to 1.2 meters away from the patient's face, with the lens height roughly level with the eyes to ensure unobstructed facial features. In standard expression analysis scenarios, a video sampling rate of 30 frames per second is suitable; for scenarios requiring the capture of micro-expressions lasting less than 500 milliseconds, a sampling rate of 60 frames per second or higher is preferred.

[0032] Eye-tracking data is acquired via an iris / pupil detection module using the same RGB / IR camera, or via a dedicated near-infrared eye tracker. The data includes temporal variations in pupil diameter, pupil contraction latency, fixation point coordinates, distribution of the fixation avoidance zone, spontaneous blink frequency, and micro-jump frequency. Before acquisition, optional 5-point or 9-point fixation calibration can be performed to improve the accuracy of pupil localization and fixation point calculation. Simultaneously, ambient light intensity is recorded and pupil illumination correction is performed to eliminate the influence of ambient light on pupil diameter measurement results.

[0033] The speech signal is a continuous audio signal acquired through a directional microphone or a microphone built into the terminal, used for subsequent extraction of semantic and emotional features and acoustic features. The microphone sampling rate is no less than 16kHz, and the microphone is installed 0.3 to 1.0 meters away from the patient's mouth, and can be used with noise reduction algorithms to filter out ambient background noise. The acquired speech signal is transcribed into text using Automatic Speech Recognition (ASR) technology, allowing the semantic and emotional recognition model to extract features such as the intensity of emotional words, topic avoidance behavior, and narrative structure anomalies; on the other hand, it is directly subjected to acoustic analysis to extract prosodic features such as speech rate, pause patterns, and the frequency of voice tremor.

[0034] Respiratory physiological signals are acquired through at least one of the following methods: thoracic-abdominal breathing belt, photoplethysmography (PPG) thoracic motion estimation algorithm, millimeter-wave radar, or depth camera. These signals include indicators such as respiratory rate, coefficient of variation of respiratory intervals, respiratory depth, and breath-holding or shallow, rapid breathing patterns.

[0035] When using contact-based solutions such as breathing harnesses, the sensor can be fixed to the surface of the patient's clothing on the chest or abdomen; when using non-contact solutions such as depth cameras, the depth camera can be aimed at the region of interest on the chest or abdomen at a distance of 0.5 to 2.0 meters. All physiological signals are aligned with the video stream, audio signal, and eye-tracking data using a unified timestamp.

[0036] In an optional embodiment, the multimodal raw data may also include skin conductance signals, which are collected via skin conductance sensors, wristbands, or finger clip electrodes, measuring indicators such as skin conductance baseline, skin conductance response amplitude, response latency, and recovery time of the skin on the index / middle finger pads, palmar side, or wrist. At least 30 to 60 seconds of resting baseline data are collected before data acquisition for subsequent personalized calibration.

[0037] In a preferred embodiment, the specific process of synchronous acquisition is as follows: First, all sensors are activated and time synchronization calibration is performed to ensure that the timestamp reference of each modality data is consistent; then, multimodal data is continuously acquired for 3 to 5 minutes during the preoperative assessment period, of which the first 30 to 60 seconds is the resting baseline acquisition phase, during which the patient remains calm, faces the camera and breathes naturally, and the subsequent phase is the standardized question-and-answer and fixation task acquisition phase, during which the patient needs to answer preset questions and fixate on the surgery-related or neutral images presented on the screen; finally, after the acquisition is completed, all modality data are packaged according to a unified timestamp to generate a multimodal time series dataset for use by the subsequent feature extraction module.

[0038] S2: Extract features from facial video streams, eye-tracking data, speech signals, and respiratory physiological signals to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features.

[0039] It should be noted that feature extraction refers to the process of extracting quantitative indicators that reflect the emotional state of the target patient from the raw multimodal signals. Facial expression features can be features used to characterize facial emotional expression; eye movement features can be features used to characterize visual attention patterns and pupillary responses; semantic acoustic features can be features used to characterize language content and vocal rhythm; and physiological features can be features used to characterize the activity of the autonomic nervous system. Optionally, since the raw data of different modalities have different data structures and information dimensions, this application uses feature extraction algorithms adapted to each modality for processing. Specifically, facial expression features can be extracted from facial video streams; eye movement features can be extracted from eye movement data; semantic acoustic features can be extracted from speech signals; and physiological features can be extracted from respiratory physiological signals. Furthermore, all extracted features are identified and associated based on a unified timestamp to ensure the alignment of features in the temporal dimension during subsequent multimodal fusion.

[0040] Figure 2 This is a flowchart of feature extraction provided in one embodiment of this application. As one implementation, step S2 specifically includes, for example: S21: Extract facial motion unit intensity, expression category, and micro-expression duration from the facial video stream as expression features; S22: Extract pupil diameter, pupillary contraction latency, fixation point trajectory, fixation avoidance area, spontaneous blink frequency, and microjump frequency from eye movement data as eye movement features; S23: Extract emotional vocabulary intensity, topic avoidance behavior, speech rate, pause patterns and speech tremor frequency from speech signals as semantic acoustic features; S24: Extract respiratory rate, respiratory depth, baseline skin conductance value, and skin conductance response amplitude from respiratory physiological signals as physiological characteristics.

[0041] It should be noted that, regarding facial expression features, facial action unit intensity is the intensity value of each AU output by a pre-trained action unit detection model. For example, increased activity of the corrugator supercilii (AU4) is associated with anxiety. Expression category is the result of calmness or anxiety output by the expression classification model. Microexpression duration is the length of the time window during which the AU intensity rapidly increases and then recovers, typically less than 500 milliseconds. Regarding eye movement features, pupil diameter and contraction latency reflect the reaction speed of the autonomic nervous system; fixation trajectory and fixation avoidance area reflect avoidance behavior to surgery-related stimuli; spontaneous blink frequency and microjugation frequency reflect acute stress. Regarding semantic acoustic features, emotional lexical intensity is the text emotional intensity value extracted by a deep semantic emotion recognition model; topic avoidance behavior refers to the patient actively changing the topic to avoid surgery-related questions; speech rate, pause patterns, and voice tremor frequency are acoustic prosodic indicators reflecting anxiety. Regarding physiological features, respiratory rate and respiratory depth reflect changes in respiratory pattern; baseline skin conductance and skin conductance response amplitude reflect the level of sympathetic nervous system arousal.

[0042] Specifically, facial action unit intensity is obtained by regressing facial keypoint coordinates using a pre-trained deep learning model, outputting a continuous value from 0 to 1; expression category is determined by the probability distribution output by an expression classification model, taking the category with the highest probability; micro-expression duration is determined by detecting rapid rises and falls in facial action unit intensity within a time window of less than 500 milliseconds. The deep learning model can be an AU detector based on ResNet or a graph convolutional network, and the expression classification model can be EfficientNet.

[0043] Pupil diameter can be extracted from eye images using a pupil detection algorithm, and the contraction latency can be calculated by combining the stimulus presentation time point. The gaze point trajectory and gaze avoidance region are calculated using a gaze tracking algorithm to determine the screen gaze point coordinates and statistically analyze the percentage of gaze duration within a pre-defined surgically relevant region of interest. The number of blinks per unit time can be counted by detecting rapid decreases and recoveries in the eye's aspect ratio to determine the spontaneous blink frequency. Micro-jump frequency is obtained by detecting minute, high-speed eye movement events after filtering saccade components from high-frequency eye movement signals.

[0044] The intensity of emotional vocabulary was obtained by performing emotional intensity regression on the speech-to-text using a pre-trained semantic sentiment analysis model; topic avoidance behavior was determined by detecting whether patients' answers to surgery-related questions contained topic shifts, avoidance words, or refusal to answer; speech rate, pause patterns, and speech tremor frequency were obtained by segmenting effective speech segments using speech endpoint detection, and then statistically analyzing the number of words per unit time, pause duration distribution, and fundamental frequency jitter value.

[0045] Respiratory rate and respiratory depth are calculated using a respiratory peak detection algorithm to determine the number of respiratory cycles per unit time and the peak-to-trough amplitude of each breath. The baseline value of skin conductance and the amplitude of skin conductance response are obtained by low-pass filtering to obtain the steady-state baseline of the skin conductance signal, and the difference between the peak value and the baseline is detected within a time window of 1-5 seconds after the stimulus presentation.

[0046] In a preferred embodiment, after step S2, the method further includes: Obtain individual baseline multimodal features of target patients during the resting baseline period, and establish an individual emotional fingerprint model based on the individual baseline multimodal features; Based on the corresponding baseline parameters of each feature in the corresponding facial expression features, eye movement features, semantic acoustic features, physiological features, and personal emotional fingerprint model, the corresponding relative rate of change is calculated, and the corresponding relative rate of change is fused based on the preset multimodal fusion weights.

[0047] It should be noted that this technical feature defines a personalized assessment scheme based on individual baseline calibration, namely, emotional fingerprinting technology. Individuals exhibit significant physiological differences in emotional expression: some people naturally have larger pupil diameters at rest, some speak faster naturally, and some have naturally higher baseline skin conductance values. Using a uniform group threshold for assessment can easily misjudge individual physiological traits as emotional abnormalities, or misjudge genuine emotional abnormalities as normal. To address this issue, this application introduces a personal emotional fingerprint model to establish a personalized assessment baseline for each patient.

[0048] Specifically, the resting baseline period refers to the initial phase of preoperative assessment, such as the first 30 to 60 seconds, during which the target patient remains calm, faces the camera, breathes naturally, and does not engage in question-and-answer tasks. During this period, the patient's individual baseline multimodal characteristics in a calm state can be acquired, including baseline facial expressions, eye movement characteristics (e.g., resting pupil diameter), semantic-acoustic characteristics (e.g., normal speech rate), and physiological characteristics (e.g., resting respiratory rate and skin conductance baseline values). These baseline characteristics, combined, constitute the patient's individual emotional fingerprint model, reflecting the patient's individual-specific physiological expression patterns in a non-anxiety state.

[0049] After completing the resting baseline acquisition, the system enters the formal preoperative assessment phase, performing standardized question-and-answer and gaze tasks. At this point, the extracted real-time features of each modality are no longer directly used for fusion, but are first compared with the corresponding baseline parameters in the individual's emotional fingerprint model to calculate the relative rate of change of each feature.

[0050] The relative rate of change is calculated using the formula: (real-time feature value - baseline value) / baseline value, or real-time feature value / baseline value. For example, the relative rate of change of pupil diameter is (real-time pupil diameter - resting pupil diameter) / resting pupil diameter. A positive value indicates pupil dilation, corresponding to an anxiety signal. A negative value can indicate pupil constriction. Through this conversion, the originally individualized absolute value is mapped to the magnitude of change relative to the individual baseline, thereby eliminating the influence of individual physiological differences on the assessment results.

[0051] Finally, the calculated relative change rates of each modality are used as input, and feature fusion is performed according to the preset multimodal fusion weights. Since the relative change rates have been calibrated against individual baselines, the fused target features can more accurately reflect the patient's emotional fluctuation relative to their calm state, thereby achieving personalized and precise preoperative psychological assessment.

[0052] S3: Based on the preset multimodal fusion weights, facial expression features, eye movement features, semantic acoustic features and physiological features are fused to obtain the target fused features; eye movement features are given the highest fusion weight.

[0053] It should be noted that feature fusion refers to the process of combining features extracted from different modalities into a comprehensive feature vector according to certain weights. Optionally, since different modal features have different representational abilities of emotional states—for example, eye-tracking features are more difficult to fake and better reflect true anxiety states than facial expression features—this application assigns different fusion weights to each modal feature. During fusion, each modal feature is multiplied by its corresponding weight and then summed or subjected to other fusion operations to generate a target fusion feature containing multi-dimensional emotional information. This target fusion feature serves as the input for subsequently generating a preoperative emotional score.

[0054] It should be noted that this application assigns the highest fusion weight to eye-tracking features based on the physiological principle that eye-tracking responses are direct indicators of brain activity and are more difficult to consciously manipulate than facial expressions. Facial expressions can be consciously controlled by patients, while eye-tracking indicators such as pupillary responses, gaze patterns, and blink frequency are difficult to fake consciously. Therefore, in multimodal fusion assessment, eye-tracking features have higher reliability and resistance to faking compared to facial expression features, semantic-acoustic features, and physiological features. By assigning the highest fusion weight to eye-tracking features, this application can effectively suppress the influence of patient faking behavior on assessment results, improve the sensitivity of identifying patients who are feigning calm, and thus solve the assessment distortion problem caused by reliance on faking facial expressions in related technologies.

[0055] Figure 3 This is a flowchart illustrating the determination of target fusion features according to one embodiment of this application. As one implementation, step S3 specifically includes, for example: S31: Normalize the facial expression features, eye movement features, semantic acoustic features, and physiological features respectively to obtain facial expression modal features, eye movement modal features, semantic acoustic modal features, and physiological modal features; S32: Based on the preset multimodal fusion weights, the facial expression modality features, eye movement modality features, semantic acoustic modality features and physiological modality features are fused to obtain the target fused features.

[0056] It should be noted that, because different modal features have different dimensions and numerical ranges—for example, facial action unit intensity is a continuous value of 0-1, pupil diameter is measured in millimeters, and speech rate is measured in words per minute—direct weighted fusion would result in features with larger dimensions dominating the fusion result, leading to fusion distortion. Therefore, before performing weighted fusion, this application first normalizes each modal feature, mapping each feature to a unified numerical range, resulting in normalized modal features, namely, facial expression modal features, eye-tracking modal features, semantic acoustic modal features, and physiological modal features. Normalization can employ min-max normalization, Z-score normalization, or quantile-based normalization methods. After normalization, the normalized modal features are then weighted and fused according to preset multimodal fusion weights to generate the target fused feature.

[0057] As one implementation method, step S32 specifically includes, for example: S321: Obtain the quality assessment scores corresponding to facial expression modality features, eye movement modality features, semantic acoustic modality features, and physiological modality features; S322: Based on the quality assessment score, dynamically adjust the preset multimodal fusion weights to obtain the target fusion weights corresponding to each modal feature; S323: Based on the target fusion weights corresponding to each modal feature, the facial expression modal features, eye movement modal features, semantic acoustic modal features, and physiological modal features are fused to obtain the target fusion features.

[0058] It should be noted that the quality of data from different modalities may vary during actual data acquisition. For example, a patient tilting their head down may reduce the confidence level of facial feature extraction, excessive ambient light may decrease the signal-to-noise ratio of pupil detection, excessive microphone distance may cause blurred speech signals, and loose breathing belts may distort physiological signals. To address this issue, this application introduces a quality assessment mechanism: During or after feature extraction, a quality assessment score is calculated for each modal feature. This score can be determined comprehensively based on indicators such as signal-to-noise ratio, data completeness, acquisition confidence, and occlusion ratio. When the quality assessment score of a certain modality falls below a preset threshold, the system automatically reduces the fusion weight corresponding to that modality and correspondingly increases the weights of other modalities with better quality. Through this dynamic adjustment, this application can still ensure the reliability and robustness of the fusion results even when the data quality of some modalities is poor. After weight adjustment, the normalized modal features are weighted and fused according to the target fusion weights corresponding to each modality to generate the target fusion feature.

[0059] S4: Generate a preoperative emotion score based on target fusion features.

[0060] It should be noted that generating a preoperative emotional score refers to the process of converting the target fusion features obtained in the previous stage into a numerical value that can quantify the anxiety level of the target patient. This score is a continuous value, for example, from 0 to 100, with higher values ​​indicating higher levels of anxiety. By mapping high-dimensional fusion features to an intuitive score, this application can provide a concise and quantifiable basis for subsequent risk stratification and clinical decision-making.

[0061] Figure 4 This is a flowchart illustrating the process of obtaining a preoperative emotional score in one embodiment of this application. As one implementation, step S4 specifically includes, for example: S41: Input the target fusion features into the preset emotion classifier and output the probability distribution of emotion categories; the emotion categories include calm, mild anxiety, moderate anxiety and severe anxiety; S42: Input the target fusion features into the preset sentiment rating regression model and output the initial score; S43: The initial score is calibrated based on the probability distribution of emotion categories to obtain the preoperative emotion score.

[0062] It should be noted that using a regression model alone to directly output scores may result in unclear score boundaries and insufficient class discrimination; using a classifier alone to output categories cannot provide refined score values ​​for clinical reference. To address this issue, this application adopts a joint calibration scheme that combines classification and regression.

[0063] Specifically, the preset emotion classifier is a multi-classification model, such as Softmax regression, support vector machine, or multilayer perceptron. Its input is the target fusion feature, and its output is the probability distribution of four emotion categories, that is, the probability value of the target patient belonging to calm, mild anxiety, moderate anxiety, and severe anxiety, and the sum of the four probability values ​​is 1. For example, the output result may be: calm 0.1, mild anxiety 0.2, moderate anxiety 0.5, severe anxiety 0.2.

[0064] The pre-defined emotion rating regression model is a regression model, such as linear regression, support vector regression, or a deep regression network. Its input is also the target fusion feature, and its output is a continuous value from 0 to 100 as an initial score. This initial score reflects the regression model's direct numerical estimate of the patient's anxiety level.

[0065] The calibration process is as follows: The probabilities of each category output by the classifier are used as weights, and the preset representative scores corresponding to each category are weighted and summed to obtain the calibrated preoperative emotion score. A more refined calibration method can also be used: the initial score is used as a priori value, and Bayesian updates or score corrections are performed based on the probability distribution output by the classifier to obtain the final score, i.e., the preoperative emotion score.

[0066] In a preferred embodiment, after step S4, the method further includes: Obtain multimodal raw data of the family members of the target patient's accompanying family members; A time-series correlation analysis was performed on the multimodal raw data of the target patient and the multimodal raw data of the family members of the target accompanying family members to calculate the anxiety influence coefficient of the target accompanying family members transmitting anxiety to the target patient; The anxiety impact coefficient is compared with the preset anxiety impact threshold. If the anxiety impact coefficient is greater than the preset anxiety impact threshold, the preoperative emotional score is increased. Based on the increased preoperative emotional score and the preset risk stratification rules, a psychological assessment result is generated.

[0067] It should be noted that this technology limits the analysis of family-patient emotional linkages. In clinical practice, family members' anxiety can be transmitted to patients through words, expressions, and behaviors, forming a chain of anxiety contagion and further increasing the patient's preoperative anxiety level. However, the technology only focuses on the patient's own emotional state, ignoring the impact of family members' emotions on the patient, which may lead to an underestimation of the patient's true anxiety level.

[0068] Specifically, the target accompanying family member refers to the family member who accompanies the target patient into the preoperative assessment area. This application can simultaneously collect the multimodal raw data of the family member while collecting the patient's multimodal raw data. The collection method is the same as that of the patient, including facial video stream, eye movement data, voice signals, and respiratory physiological signals, with the collection duration being the same time window.

[0069] Temporal correlation analysis refers to the time-series correlation analysis of multimodal data from patients and their families to detect whether family members' anxiety precedes the patient's anxiety and to determine the transmission relationship between the two. Specific methods include: extracting anxiety-related characteristic time-series curves from the multimodal data of both patients and their families, such as pupil diameter change curves and skin conductance response curves; calculating the cross-correlation coefficient or Granger causality test statistic between the two curves; and determining the time delay and correlation strength between the family's anxiety signal and the patient's anxiety signal.

[0070] The anxiety impact coefficient is a quantitative indicator used to measure the strength and degree of transmission of family anxiety to the patient's anxiety level. This coefficient can be calculated based on time-series correlation analysis results, for example, by taking the peak value of the cross-correlation coefficient as the impact coefficient; its value can range from 0 to 1, with a larger value indicating a stronger impact of family anxiety on the patient's anxiety; or it can be obtained by normalization based on the F-statistic of the Granger causality test.

[0071] After obtaining the anxiety impact coefficient, it is compared with a preset anxiety impact threshold. If the anxiety impact coefficient is greater than the threshold, it indicates that the family's anxiety has a significant transmission effect on the patient. In this case, the patient's preoperative emotional score may be underestimated—part of the anxiety originates from family transmission rather than the patient's own anxiety. To correct this bias, this application increases the preoperative emotional score, for example, by multiplying it by an adjustment factor greater than 1, or by directly adding a preset correction score. Then, psychological assessment results are generated based on the increased preoperative emotional score and preset risk stratification rules.

[0072] Through the above mechanism, this application can identify and quantify the impact of family members' emotions on patients, and correct patients' emotional scores in the presence of obvious anxiety contagion, thereby more comprehensively reflecting the patient's psychological state in the real preoperative environment and avoiding assessment bias caused by ignoring environmental factors.

[0073] In a preferred embodiment, after step S4, the method further includes: The preoperative emotional score, eye movement characteristics, physiological characteristics, and basic patient information of the target patient are input into the preset time-series prediction model, which outputs the intraoperative stress response prediction score, the predicted value of anesthesia induction dosage, and the intraoperative awareness risk level.

[0074] It should be noted that this technical feature defines a full-link emotion prediction solution, which extends the preoperative assessment results to the prediction of intraoperative stress response, achieving a leap from current assessment to forward prediction.

[0075] The pre-trained temporal prediction model is a deep learning model whose input is a multi-dimensional feature vector and whose output is the predicted values ​​of multiple intraoperative indicators. This model is trained on a historical surgical dataset, with training samples including the patient's preoperative characteristics and their corresponding intraoperative records. The deep learning model can be a Long Short-Term Memory (LSTM) network, a Gated Recurrent Unit (GRU) model, or a Transformer model, etc. Preoperative characteristics can include at least emotion scores, eye movement features, physiological characteristics, and basic information. Intraoperative records can include at least the intensity of stress response, dosage of anesthetic drugs, and whether intraoperative awareness occurred.

[0076] Patient basic information, including age, gender, weight, ASA classification, type of surgery, previous anesthesia history, and medication history, is used to correct the model prediction results and improve the accuracy of the prediction.

[0077] The intraoperative stress response prediction score is a quantitative indicator used to predict the intensity of stress responses that patients may experience during surgery. A high stress response score can indicate the risk of sudden increases in blood pressure, arrhythmias, and large releases of stress hormones during surgery. Anesthesiologists can use this information to prepare vasoactive drugs in advance and enhance intraoperative monitoring.

[0078] Predicted induction doses refer to the dose of anesthetic drugs a patient is expected to require based on their preoperative emotional state, such as the induction dose of propofol or the booster dose of midazolam. Highly anxious patients typically require higher induction doses to achieve the same depth of sedation. This predicted value can provide anesthesiologists with a dosage reference, preventing insufficient dosage leading to intraoperative awareness or excessive dosage leading to circulatory depression.

[0079] Intraoperative awareness risk grading categorizes the risk level of a patient's potential to remain conscious and recall surgical events during surgery, such as low risk, medium risk, and high risk. Intraoperative awareness is a serious complication of anesthesia, and preoperative high anxiety is a significant risk factor. By outputting this risk grading, this application can help anesthesiologists identify high-risk patients and take preventative measures such as deepening anesthesia and using EEG monitoring.

[0080] In a preferred embodiment, the temporal prediction model employs an LSTM architecture, capable of capturing temporal change patterns during the preoperative assessment period, such as the trend of increasing anxiety levels over time, thereby more accurately predicting intraoperative stress responses. The model's output can be directly displayed on the anesthesia workstation interface, providing quantitative evidence for personalized adjustments to the anesthesia protocol.

[0081] Through the aforementioned end-to-end prediction mechanism, this application overcomes the limitation of related technologies that can only perform current assessments, transforming preoperative psychological assessment results into quantitative predictions of intraoperative risks, and realizing a complete technical closed loop from identification to prediction to prevention.

[0082] S5: Generate psychological assessment results based on preoperative emotional scores and preset risk stratification rules.

[0083] It should be noted that generating psychological assessment results refers to the process of mapping preoperative emotional scores into clinically meaningful assessment conclusions according to pre-defined risk stratification rules. Preoperative emotional scores are continuous numerical values ​​that only reflect the quantitative level of anxiety, but clinical decision-making requires a clear risk level classification. Therefore, this application pre-defines risk stratification rules, such as setting one or more risk thresholds, to divide continuous emotional scores into different risk intervals. Each risk interval can correspond to a risk level and a corresponding emotional state description, which can be emotional stability, mild anxiety, significant anxiety, or depression, etc. Through this mapping, this application transforms abstract score values ​​into clinically understandable and operable psychological assessment results, providing a basis for subsequent anesthesia plan adjustments and preoperative interventions.

[0084] Figure 5 This is a flowchart illustrating the generation of psychological assessment results in one embodiment of this application. As one implementation, step S5 specifically includes, for example: S51: Obtain the preset risk stratification threshold and the sedition calming marker; the sedition calming marker is used to indicate that the target patient's facial expression is inconsistent with their internal anxiety; S52: Compare the preoperative emotional score with the preset risk stratification threshold to obtain the comparison results; and determine the initial risk level based on the comparison results. S53: In the presence of a deceptive calmness marker, the initial risk level is increased to become the target risk level; S54: Based on the target risk level, generate a psychological assessment result that includes the risk level and a suggestion to feign calmness.

[0085] It should be noted that the feigned calmness marker is a binary indicator used to characterize whether a target patient exhibits a contradictory state of outward calmness and inner anxiety. When a patient displays feigned calmness, their true anxiety level is higher than what is presented by their outward expression. Determining the risk level solely based on emotion scores may lead to an underestimation of the risk.

[0086] Specifically, this application first obtains preset risk stratification thresholds, such as a first threshold of 40 points, a second threshold of 70 points, etc., and compares the preoperative emotional score with these thresholds to determine the corresponding initial risk level, such as low risk, medium risk, and high risk. Further, it can determine whether there are signs of feigned calmness: if so, the initial risk level is increased by one level, for example, from medium risk to high risk, or from low risk to medium risk; if not, the initial risk level remains unchanged. Finally, a psychological assessment result containing the target risk level and signs of feigned calmness is generated.

[0087] In one preferred embodiment, the preoperative emotional assessment includes facial expression assessment and eye movement assessment; the method also includes: When the facial expression score corresponds to the calm category and the eye movement score corresponds to the anxiety category, a fake calmness marker is generated.

[0088] It should be noted that facial expression scores are quantified from facial expression features extracted from facial video streams and are used to reflect the patient's outward emotional state; eye movement scores are quantified from eye movement features extracted from eye movement data. Since eye movements are difficult to control consciously, they can reflect the patient's true inner emotional state. When the facial expression score corresponds to the calm category, meaning the patient's facial expression is calm and shows no signs of anxiety, while the eye movement score corresponds to the anxiety category, meaning the patient's eye movement indicators such as pupil dilation, gaze avoidance, and rapid blinking show an anxious state, it indicates that the patient has a contradictory state of outward calmness and inner anxiety. In this case, a feigned calmness label is generated. This label is used for subsequent risk level correction.

[0089] As a preferred implementation, the preoperative emotional assessment may further include a semantic assessment. The method may also include generating a feigned calmness marker when the semantic assessment is in the calm or avoidance category and the eye movement assessment is in the anxiety category.

[0090] It should be noted that the semantic score is quantified from semantic acoustic features extracted from the speech signal, reflecting the patient's emotional state expressed through language. When a patient verbally expresses calming phrases such as "I'm not nervous" or "I'm fine," or exhibits behavior that avoids topics related to surgery, while the eye movement score shows an anxiety category, it also indicates a contradictory state of verbal concealment and internal anxiety. In this case, a feigned calmness marker is generated. This method complements the aforementioned facial-eye movement contradiction detection: facial-eye movement contradiction detection is suitable for patients who are expressionless but anxious internally, while semantic-eye movement contradiction detection is suitable for patients who verbally deny but are anxious internally. Both methods can be used simultaneously; a feigned calmness marker is generated whenever a contradiction is detected in either method, thereby improving the sensitivity and robustness of the detection.

[0091] In one preferred embodiment, the method further includes: Obtain the current anesthesia plan; Based on the results of the psychological assessment, the current anesthesia protocol was adjusted, and a target anesthesia protocol was determined.

[0092] It should be noted that the current anesthesia protocol refers to an initial anesthesia protocol based on routine patient information, such as age, weight, ASA classification, and type of surgery, without taking into account the patient's psychological state. This protocol can be obtained from the hospital information system or entered by the anesthesiologist.

[0093] Adjusting the current anesthesia protocol based on psychological assessment results refers to using the assessment results (including risk level, markers of feigned sedation, and descriptions of emotional state) as a basis for decision-making and making personalized adjustments to the current anesthesia protocol. The adjustment strategy can be pre-set as a rule mapping table. For example: when the psychological assessment result is low-risk, maintain the current anesthesia protocol or implement a standard anesthesia protocol; when the psychological assessment result is medium-risk, add sedative adjunctive drugs to the current anesthesia protocol; when the psychological assessment result is high-risk, add anti-anxiety drugs or adjust the dosage of induction drugs to the current anesthesia protocol; when the psychological assessment result contains markers of feigned sedation, prompt the anesthesiologist to strengthen intraoperative stress monitoring and appropriately increase the reserve of sedative drugs.

[0094] Determining the target anesthesia plan refers to outputting the adjusted anesthesia plan as the final execution plan to the anesthesia information system or presenting it to the anesthesiologist for confirmation and execution. Through this adjustment mechanism, this application can directly transform the results of preoperative psychological assessment into executable anesthesia decisions, achieving a complete closed loop from identification to intervention.

[0095] In a preferred embodiment, the adjustment strategy can be optimized based on clinical validation data. For example, a correlation model can be established between psychological assessment results and intraoperative stress response and anesthetic drug dosage. Based on the model's prediction results, the optimal adjustment range of the anesthesia plan can be automatically recommended, achieving precise and personalized anesthesia decision-making.

[0096] Figure 6 This is a schematic diagram of the structure of a psychological assessment device provided in one embodiment of this application. The following describes an embodiment of a psychological assessment device according to this application. Specifically, as shown... Figure 6 As shown, the device includes: The data acquisition module S61 is used to simultaneously acquire multimodal raw data of the target patient during the preoperative assessment period; the multimodal raw data includes at least facial video stream, eye movement data, speech signals and respiratory physiological signals; The feature extraction module S62 is used to extract features from facial video streams, eye-tracking data, speech signals, and respiratory physiological signals to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features. The feature fusion module S63 is used to fuse facial expression features, eye movement features, semantic acoustic features and physiological features based on preset multimodal fusion weights to obtain target fused features; eye movement features are given the highest fusion weight; The rating generation module S64 is used to generate preoperative emotion ratings based on target fusion features; The assessment result determination module S65 is used to generate psychological assessment results based on preoperative emotional scores and preset risk stratification rules.

[0097] In a preferred embodiment, the above-described apparatus further includes: The current protocol acquisition unit is used to acquire the current anesthesia protocol; The target anesthesia protocol determination unit is used to adjust the current anesthesia protocol and determine the target anesthesia protocol based on the results of psychological assessment.

[0098] As a preferred embodiment, the evaluation result determination module S65 includes: The information acquisition unit is used to acquire preset risk stratification thresholds and spoofed calmness markers; spoofed calmness markers are used to indicate that the target patient's facial expression is inconsistent with their internal anxiety. The initial risk determination unit is used to compare the preoperative emotional score with a preset risk stratification threshold to obtain the comparison result; and to determine the initial risk level based on the comparison result. The target risk determination unit is used to increase the initial risk level as the target risk level when there are disguised calm markers. The results determination unit is used to generate psychological assessment results that include the risk level and a suggestion to feign calmness, based on the target risk level.

[0099] In one preferred embodiment, the preoperative emotional score includes facial expression score and eye movement score; the device further includes a first marker generation unit, used for: When the facial expression score corresponds to the calm category and the eye movement score corresponds to the anxiety category, a fake calmness marker is generated.

[0100] In a preferred embodiment, the preoperative emotional score also includes a semantic score, and the device further includes a second marker generation unit for: When the semantic score is in the calm or avoidance category and the eye movement score is in the anxiety category, generate a fake calmness tag.

[0101] In a preferred embodiment, after extracting features from the facial video stream, eye-tracking data, speech signals, and respiratory physiological signals to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features, the device further includes: The model building unit is used to acquire the individual baseline multimodal features of the target patient during the resting baseline period, and to build an individual emotional fingerprint model based on the individual baseline multimodal features. The rate of change fusion unit is used to calculate the corresponding relative rate of change based on the corresponding baseline parameters of each feature in the corresponding facial expression feature, eye movement feature, semantic acoustic feature, physiological feature and personal emotion fingerprint model, and then fuse the corresponding relative rate of change based on the preset multimodal fusion weights.

[0102] In a preferred embodiment, after generating a preoperative emotion score based on target fusion features, the device further includes: The family information acquisition unit is used to acquire multimodal raw data of the target patient's accompanying family members. The correlation analysis unit is used to perform time-series correlation analysis on the multimodal raw data of the target patient and the multimodal raw data of the target accompanying family members, and to calculate the anxiety influence coefficient of the target accompanying family members transmitting anxiety to the target patient. The anxiety impact result unit is used to compare the anxiety impact coefficient with the preset anxiety impact threshold. If the anxiety impact coefficient is greater than the preset anxiety impact threshold, the preoperative emotional score is increased, and a psychological assessment result is generated based on the increased preoperative emotional score and the preset risk stratification rules.

[0103] In a preferred embodiment, after generating a preoperative emotion score based on target fusion features, the above-mentioned device further includes a prediction unit for: The preoperative emotional score, eye movement characteristics, physiological characteristics, and basic patient information of the target patient are input into the preset time-series prediction model, which outputs the intraoperative stress response prediction score, the predicted value of anesthesia induction dosage, and the intraoperative awareness risk level.

[0104] In a preferred embodiment, the feature extraction module S62 includes: The facial expression extraction unit is used to extract facial motion unit intensity, expression category, and micro-expression duration from the facial video stream as facial features. The eye movement extraction unit is used to extract pupil diameter, pupil contraction latency, fixation point trajectory, fixation avoidance area, spontaneous blink frequency, and micro-jump frequency from eye movement data as eye movement features. The semantic acoustic extraction unit is used to extract emotional word intensity, topic avoidance behavior, speech rate, pause patterns and speech tremor frequency from speech signals as semantic acoustic features. The physiological extraction unit is used to extract respiratory rate, respiratory depth, baseline skin conductance value, and skin conductance response amplitude from respiratory physiological signals as physiological characteristics.

[0105] In a preferred embodiment, the feature fusion module S63 includes: The normalization unit is used to normalize the facial expression features, eye movement features, semantic acoustic features, and physiological features respectively, to obtain facial expression modal features, eye movement modal features, semantic acoustic modal features, and physiological modal features; The fusion unit is used to fuse facial expression modality features, eye movement modality features, semantic acoustic modality features and physiological modality features based on preset multimodal fusion weights to obtain the target fused features.

[0106] In a preferred embodiment, the fusion unit includes: The evaluation score acquisition subunit is used to obtain the quality evaluation scores corresponding to facial expression modality features, eye movement modality features, semantic acoustic modality features, and physiological modality features; The target weight determination sub-unit is used to dynamically adjust the preset multimodal fusion weights based on the quality assessment score, so as to obtain the target fusion weights corresponding to each modal feature; The target fusion subunit is used to fuse facial expression modal features, eye movement modal features, semantic acoustic modal features, and physiological modal features based on the target fusion weights corresponding to each modal feature, to obtain the target fusion feature.

[0107] In a preferred embodiment, the scoring generation module S64 includes: The probability distribution determination unit is used to input the target fusion features into the preset emotion classifier and output the probability distribution of emotion categories; the emotion categories include calm, mild anxiety, moderate anxiety, and severe anxiety; The initial score is obtained by inputting the target fusion features into the preset sentiment rating regression model and outputting the initial score. The scoring determination unit is used to calibrate the initial score based on the probability distribution of emotion categories to obtain the preoperative emotion score.

[0108] The apparatus and method embodiments described above are based on the same inventive concept and are used to implement the psychological assessment method of any of the above embodiments.

[0109] Correspondingly, this application also discloses an electronic device, which includes a processor and a memory. The memory stores at least one instruction and at least one program. The processor loads and executes the at least one instruction and at least one program to implement the psychological assessment method of any of the above embodiments.

[0110] Correspondingly, this application also discloses a computer storage medium storing at least one instruction and at least one program, wherein the at least one instruction and at least one program are loaded and executed by a processor to implement the psychological assessment method as described in any of the above embodiments.

[0111] Correspondingly, this application also discloses a computer program product, including a computer program / instructions, which, when executed by a processor, implement the psychological assessment method as described in any of the above embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A psychological assessment method, characterized in that, The method includes: During the preoperative assessment period, multimodal raw data of the target patient are collected simultaneously; the multimodal raw data includes at least facial video stream, eye movement data, speech signals and respiratory physiological signals; Feature extraction is performed on the facial video stream, the eye movement data, the speech signal, and the respiratory physiological signal to obtain corresponding facial expression features, eye movement features, semantic acoustic features, and physiological features; Based on preset multimodal fusion weights, the facial expression features, eye movement features, semantic acoustic features, and physiological features are fused to obtain the target fused features; the eye movement features are assigned the highest fusion weight. Based on the target fusion features, a preoperative emotion score is generated; Based on the preoperative emotional score and the preset risk stratification rules, psychological assessment results are generated.

2. The psychological assessment method according to claim 1, characterized in that, The method further includes: Obtain the current anesthesia plan; Based on the results of the psychological assessment, the current anesthesia protocol is adjusted, and a target anesthesia protocol is determined.

3. The psychological assessment method according to claim 1, characterized in that, The psychological assessment results generated based on the preoperative emotional score and preset risk stratification rules include: Obtain a preset risk stratification threshold and a feigned calmness marker; the feigned calmness marker is used to indicate that the target patient's facial expression is inconsistent with their internal anxiety. The preoperative emotional score is compared with the preset risk stratification threshold to obtain the comparison result; and the initial risk level is determined based on the comparison result. In the presence of the aforementioned deceptive calming markers, the initial risk level is increased to become the target risk level. Based on the target risk level, a psychological assessment result is generated that includes the risk level and a suggestion to feign composure.

4. The psychological assessment method according to claim 1, characterized in that, The preoperative emotional assessment includes facial expression assessment and eye movement assessment; the method also includes: When the facial expression score corresponds to the calm category and the eye movement score corresponds to the anxiety category, the fake calmness tag is generated.

5. The psychological assessment method according to claim 3, characterized in that, The preoperative emotional score also includes a semantic score, and the method further includes: The fake calmness tag is generated when the semantic score is in the calmness or avoidance category and the eye movement score is in the anxiety category.

6. The psychological assessment method according to claim 1, characterized in that, After extracting features from the facial video stream, the eye-tracking data, the speech signal, and the respiratory physiological signal to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features, the method further includes: The individual baseline multimodal features of the target patient were acquired during the resting baseline period, and an individual emotional fingerprint model was established based on the individual baseline multimodal features. Based on the corresponding baseline parameters of each feature in the facial expression feature, eye movement feature, semantic acoustic feature, physiological feature, and personal emotion fingerprint model, the corresponding relative rate of change is calculated, and the corresponding relative rate of change is fused based on the preset multimodal fusion weight.

7. The psychological assessment method according to claim 1, characterized in that, After generating a preoperative emotion score based on the target fusion features, the method further includes: Obtain multimodal raw data of the family members of the target patient's accompanying family members; A time-series correlation analysis was performed on the multimodal raw data of the target patient and the multimodal raw data of the family members of the target accompanying family member to calculate the anxiety influence coefficient of the anxiety transmitted from the target accompanying family member to the target patient. The anxiety impact coefficient is compared with the preset anxiety impact threshold. If the anxiety impact coefficient is greater than the preset anxiety impact threshold, the preoperative emotional score is increased. Based on the increased preoperative emotional score and the preset risk stratification rule, the psychological assessment result is generated.

8. The psychological assessment method according to claim 1, characterized in that, After generating a preoperative emotion score based on the target fusion features, the method further includes: The preoperative emotional score, eye movement characteristics, physiological characteristics, and basic patient information of the target patient are input into a preset time-series prediction model, which outputs the intraoperative stress response prediction score, the predicted value of anesthesia induction dosage, and the intraoperative risk awareness level.

9. The psychological assessment method according to claim 1, characterized in that, The step of extracting features from the facial video stream, the eye-tracking data, the speech signal, and the respiratory physiological signal to obtain corresponding facial expression features, eye-tracking features, semantic acoustic features, and physiological features includes: The facial motion unit intensity, expression category, and micro-expression duration are extracted from the facial video stream as the expression features; Pupil diameter, pupil contraction latency, fixation point trajectory, fixation avoidance area, spontaneous blink frequency, and microjump frequency are extracted from the eye movement data as the eye movement features. Emotional vocabulary intensity, topic avoidance behavior, speech rate, pause patterns, and speech tremor frequency are extracted from the speech signal as semantic acoustic features. Respiratory rate, respiratory depth, baseline skin conductance value, and skin conductance response amplitude are extracted from the respiratory physiological signals as the physiological characteristics.

10. The psychological assessment method according to claim 1, characterized in that, The expression features, eye movement features, semantic acoustic features, and physiological features are fused based on preset multimodal fusion weights to obtain the target fused features; The eye-tracking features are assigned the highest fusion weight; including: The facial expression features, eye movement features, semantic acoustic features, and physiological features are normalized respectively to obtain facial expression modal features, eye movement modal features, semantic acoustic modal features, and physiological modal features; Based on the preset multimodal fusion weights, the facial expression modality features, the eye movement modality features, the semantic acoustic modality features, and the physiological modality features are fused to obtain the target fusion features.

11. The psychological assessment method according to claim 10, characterized in that, The step of fusing the facial expression modality features, eye-tracking modality features, semantic acoustic modality features, and physiological modality features based on the preset multimodal fusion weights to obtain the target fused features includes: Obtain the quality assessment scores corresponding to the facial expression modality features, the eye movement modality features, the semantic acoustic modality features, and the physiological modality features; Based on the quality assessment score, the preset multimodal fusion weights are dynamically adjusted to obtain the target fusion weights corresponding to each modal feature; Based on the target fusion weights corresponding to each modal feature, the facial expression modal features, the eye movement modal features, the semantic acoustic modal features, and the physiological modal features are fused to obtain the target fusion features.

12. The psychological assessment method according to claim 1, characterized in that, The generation of the preoperative emotion score based on the target fusion features includes: The target fusion features are input into a preset emotion classifier, which outputs an emotion category probability distribution; the emotion categories include calm, mild anxiety, moderate anxiety, and severe anxiety. The target fusion features are input into a preset emotion rating regression model to output an initial rating; The initial score is calibrated based on the probability distribution of the emotion categories to obtain the preoperative emotion score.

13. A psychological assessment device, characterized in that, The device includes: The data acquisition module is used to simultaneously acquire multimodal raw data of the target patient during the preoperative assessment period; the multimodal raw data includes at least facial video stream, eye movement data, speech signals and respiratory physiological signals; The feature extraction module is used to extract features from the facial video stream, the eye movement data, the speech signal, and the respiratory physiological signal, respectively, to obtain corresponding facial expression features, eye movement features, semantic acoustic features, and physiological features. The feature fusion module is used to fuse the facial expression features, eye movement features, semantic acoustic features, and physiological features based on preset multimodal fusion weights to obtain target fused features; the eye movement features are assigned the highest fusion weight. The scoring generation module is used to generate a preoperative emotion score based on the target fusion features; The assessment result determination module is used to generate psychological assessment results based on the preoperative emotional score and preset risk stratification rules.

14. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the psychological assessment method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the psychological assessment method as described in any one of claims 1 to 12.

Images