Psychological state assessment method and system based on multi-modal data and model fine tuning

Abstract

The invention provides a psychological state assessment method and system based on multi-modal data and model fine tuning, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the feature extraction of target multi-modal sensing data of a to-be-assessed individual, and obtaining modal initial features corresponding to different modals; performing tokenization processing on each modal initial feature, and splicing each modal initial feature after the tokenization processing to obtain a target token sequence; inputting the target token sequence into a comprehensive psychological state assessment model to obtain a psychological state assessment result of the to-be-assessed individual; the comprehensive psychological state evaluation model is obtained by performing fine adjustment on a pre-training Transform model with a multi-task psychological state prediction head based on a multi-modal psychological state clinical data set; each psychological state prediction task head is constructed according to different psychological state prediction tasks. According to the method, the accuracy, comprehensiveness and robustness of psychological state assessment are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for assessing psychological states based on multimodal data and model fine-tuning. Background Technology

[0002] Traditional methods for assessing mental state suffer from the following problems: Traditional methods (questionnaires, interviews) are subject to subjectivity and recall bias; traditional machine learning models or small deep learning networks using multimodal data have limited capacity and representation learning ability when processing complex, high-dimensional, and long-term multimodal physiological and behavioral data streams, making it difficult to capture deep abstract patterns and subtle nonlinear long-range dependencies between different modal features, thus limiting the accuracy, detail, and generalization ability of the assessment; while applying large AI models to multimodal physiological and behavioral data analysis overcomes the bottlenecks of traditional methods, it faces key technical challenges, such as converting heterogeneous data into a unified input format and how to efficiently and robustly fine-tune the model for specific mental state assessment tasks.

[0003] Therefore, there is an urgent need for a psychological state assessment method and system based on multimodal data and model fine-tuning to solve the above problems. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a method and system for assessing psychological state based on multimodal data and model fine-tuning.

[0005] This invention provides a method for assessing psychological states based on multimodal data and model fine-tuning, comprising: Feature extraction is performed on the target multimodal sensing data of the individual to be evaluated to obtain the initial modal features corresponding to different modalities; Each of the modal initial features is tokenized, and the tokenized modal initial features are concatenated to obtain the target token sequence. The target token sequence is input into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed, output by the comprehensive psychological state assessment model. The comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset. Each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks. Each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment tag.

[0006] According to the present invention, a psychological state assessment method based on multimodal data and model fine-tuning is provided, wherein feature extraction is performed on the target multimodal sensor data of the individual to be assessed to obtain initial modal features corresponding to different modalities, including: Based on a multimodal sensing front end, millimeter-wave radar data, RGB camera data, and event camera data corresponding to the individual to be evaluated are collected; Feature extraction was performed on the millimeter-wave radar data to obtain event sequences of heart rate variability parameters, time sequences of key morphological features of electrocardiograms, and time sequences of micromotor pattern features. Feature extraction is performed on the RGB camera data to obtain time series of facial motion unit intensity features, time series of head three-dimensional rotation angle features, and time series of body key point three-dimensional coordinate features. Feature extraction is performed on the event camera data to obtain time series of micro-expression category probability features, time series of saccade parameter features, and time series of pupil dynamic features; Based on the heart rate variability parameter event sequence, the electrocardiogram key morphological feature time sequence, the micro-motion pattern feature time sequence, the facial motion unit intensity feature time sequence, the head three-dimensional rotation angle feature time sequence, the body key point three-dimensional coordinate feature time sequence, the micro-expression category probability feature time sequence, the saccade parameter feature time sequence, and the pupil dynamic feature time sequence, the modal initial features corresponding to different modalities are obtained.

[0007] According to the present invention, a method for assessing psychological states based on multimodal data and model fine-tuning includes tokenizing the initial features of each modality and concatenating the tokenized initial features to obtain a target token sequence, comprising: Based on a preset analysis time window, the initial features of each modality are aligned with the time window to obtain a sequence of modal features with the same time window. Based on the modality type, the modality feature sequences are divided into corresponding time segments to obtain the time segment features of each modality feature sequence; The time segment features are tokenized to obtain multiple time segment token sequences corresponding to each tokenized modal initial feature; Each modality type embedding vector and time segment position embedding vector are added to each of the time segment token sequences to obtain each modality token sequence; wherein, the modality type embedding vector is used to distinguish the time segment token sequences of different modalities; the time segment position embedding vector is used to represent the time order information of each time segment token sequence within its respective time window; The global context token is concatenated with each of the modal token sequences to obtain the target token sequence; wherein the global context token is used to aggregate the encoded global information.

[0008] According to the present invention, a psychological state assessment method based on multimodal data and model fine-tuning is provided, wherein the comprehensive psychological state assessment model is trained through the following steps: Based on the psychological evaluation results of sample fragments corresponding to each modal token sequence in the sample token sequence, the psychological state evaluation label is constructed. Based on the psychological state assessment labels, each modal token sequence in the sample token sequence is labeled to obtain the multimodal psychological state clinical dataset; Based on the multimodal psychological state clinical dataset, the pre-trained Transformer model and the multi-task psychological state prediction head are progressively fine-tuned to obtain the comprehensive psychological state assessment model. The multi-task mental state prediction head includes at least an emotion classification head, an anxiety / depression severity regression head, a cognitive load classification head, and a personality trait regression head. The emotion classification head consists of a fully connected layer and a Softmax activation function, used to output the probability distribution of discrete emotion categories. The loss function of the emotion classification head is cross-entropy loss. The anxiety / depression severity regression head consists of a fully connected layer and a linear activation function, used to predict continuous severity scores of anxiety and / or depression. The loss function of the anxiety / depression severity regression head is mean squared error or Huber loss. The cognitive load classification head consists of a fully connected layer and a Softmax activation function, used to output the probability distribution of cognitive load levels. The loss function of the cognitive load classification head is cross-entropy loss. The personality trait regression head includes five parallel regression subheadings, each consisting of a fully connected layer and a linear activation function, used to output continuous scores for each dimension of the Big Five personality traits.

[0009] According to the present invention, a method for assessing mental state based on multimodal data and model fine-tuning is provided. The method involves progressively fine-tuning the pre-trained Transformer model and the multi-task mental state prediction head based on the multimodal mental state clinical dataset to obtain the comprehensive mental state assessment model, comprising: The encoder parameters of the pre-trained Transformer model are frozen, and the multi-task mental state prediction head is initially trained based on the multimodal mental state clinical dataset and the first learning rate to obtain the initially trained multi-task mental state prediction head. Unfreeze the top preset number of Transformer layers in the encoder of the pre-trained Transformer model whose encoder parameters are all frozen to obtain the pre-trained Transformer model after initial unfreezing. Based on the multimodal mental state clinical dataset and the second learning rate, the pre-trained Transformer model after initial thawing and the pre-trained multi-task mental state prediction head are jointly trained to obtain the pre-trained Transformer model and the pre-trained multi-task mental state prediction head after initial fine-tuning, respectively. Unfreeze all Transformer layers in the encoder of the pre-trained Transformer model after initial fine-tuning to obtain the fully unfrozen pre-trained Transformer model. Based on the multimodal mental state clinical dataset and the third learning rate, the fully unfrozen pre-trained Transformer model and the preliminarily fine-tuned multi-task mental state prediction head are jointly fine-tuned to obtain the comprehensive mental state assessment model. The first learning rate, the second learning rate, and the third learning rate decrease sequentially.

[0010] According to the present invention, a method for evaluating mental states based on multimodal data and model fine-tuning is provided. The total loss of the multi-task mental state prediction head is obtained by weighted summation of the losses of each mental state prediction task head in the multi-task mental state prediction head. The weight coefficients corresponding to each mental state prediction task head are obtained based on the homoscedastic uncertainty learning of the mental state prediction task head during the training process.

[0011] According to the present invention, a method for assessing mental state based on multimodal data and model fine-tuning, after inputting the target token sequence into a trained comprehensive mental state assessment model to obtain the mental state assessment result of the individual to be assessed output by the comprehensive mental state assessment model, the method further includes: Based on the psychological state assessment results, real-time monitoring data of the psychological state indicators of the individual to be assessed, a multi-dimensional psychological state assessment report, and historical change curves of the psychological state indicators are obtained. Based on the real-time monitoring data of the psychological state indicators, the multi-dimensional assessment report of the psychological state, and the historical change curves of the psychological state indicators, a comprehensive psychological state profile of the individual to be assessed is constructed.

[0012] This invention also provides a psychological state assessment system based on multimodal data and model fine-tuning, comprising: The multimodal data feature extraction module is used to extract features from the target multimodal sensing data of the individual to be evaluated, and obtain the initial modal features corresponding to different modalities. A physiological behavior tokenizer is used to tokenize each of the modal initial features and concatenate the tokenized modal initial features to obtain a target token sequence. The comprehensive psychological state assessment module is used to input the target token sequence into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed, output by the comprehensive psychological state assessment model. The comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset. Each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks. Each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment label.

[0013] The present invention provides a psychological state assessment method and system based on multimodal data and model fine-tuning. It extracts features from the multimodal sensor data of the individual to be assessed, obtains initial modal features, and tokenizes and concatenates them into a target token sequence. The target token sequence is then input into a trained comprehensive psychological state assessment model. This model is based on a multimodal clinical dataset and is fine-tuned from a pre-trained Transformer model that includes multi-task prediction heads constructed with different prediction tasks. This improves the accuracy, comprehensiveness and robustness of psychological state assessment. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the psychological state assessment method based on multimodal data and model fine-tuning provided by the present invention. Figure 2 A schematic diagram of the physiological behavior tokenizer provided by the present invention; Figure 3 This is a schematic diagram of the overall process of the psychological state assessment method based on multimodal data and model fine-tuning provided by the present invention; Figure 4 This is a schematic diagram of the structure of the psychological state assessment system based on multimodal data and model fine-tuning provided by the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0017] Traditional psychological assessment methods (such as questionnaires and interviews) suffer from subjectivity and recall bias. In recent years, objective assessment methods based on sensor data have received widespread attention, among which multimodal information fusion is considered an effective way to improve assessment performance because it can integrate complementary information from different physiological and behavioral channels.

[0018] However, even with multimodal data, traditional machine learning models or small deep learning networks designed for specific tasks often have limited capacity and representation learning ability when dealing with highly complex, high-dimensional, and long-term multimodal physiological behavior data streams. They struggle to fully capture deep, abstract patterns in the data, as well as subtle, nonlinear, long-range dependencies between modal features, thus limiting the accuracy, detail, and generalization ability of psychological state assessments to individual differences and situational changes.

[0019] Large-scale artificial intelligence models, especially pre-trained foundation models based on the Transformer architecture, have demonstrated powerful general knowledge learning and representation capabilities in fields such as natural language processing and computer vision. Applying these large-scale models to multimodal physiological and behavioral data analysis holds promise for overcoming the bottlenecks of traditional methods. However, the key technical challenges currently facing us include how to effectively transform heterogeneous multimodal physiological and behavioral data into a unified input format suitable for processing by large-scale models (i.e., "tokenization"), and how to efficiently and robustly fine-tune these large-scale pre-trained models for specific psychological state assessment tasks defined by psychiatric expertise.

[0020] To address the problems existing in the prior art, this invention designs a "PhysioBehavioral Tokenizer" that can effectively convert highly heterogeneous, multi-rate feature sequences from millimeter-wave radar (physiology), RGB cameras (macro-behavior), and event cameras (micro-behavior) into unified, information-rich token sequences suitable for input to large-scale Transformer models. Simultaneously, a core large-scale pre-trained Transformer model is constructed, and a set of parallel, lightweight, task-specific prediction heads for various clinically relevant psychological states (such as mood, anxiety / depression severity, cognitive load, and personality trait tendencies) are designed. Furthermore, a "Psychiatrist-curated Progressive" data-guided fine-tuning strategy is constructed. Fine-tuning (PCPF) efficiently and stably transfers and specializes the general representation capabilities of large pre-trained models to the multimodal mental state assessment task of this invention, while avoiding catastrophic amnesia. In addition, it deeply integrates the knowledge and experience of professional psychiatrists into the entire process of data acquisition protocol design, mental state labeling system construction, curation of high-quality labeled datasets, and clinical validation and interpretation of model output results, so as to ensure the accuracy, reliability and practical value of the assessment system. This solves the problems of insufficient model representation capabilities, immature data tokenization methods, lack of targeted fine-tuning strategies for large models and insufficient integration of psychiatric professional knowledge when using multimodal data and artificial intelligence for mental state assessment.

[0021] Figure 1 The flowchart of the psychological state assessment method based on multimodal data and model fine-tuning provided by this invention is shown below. Figure 1 As shown, this invention provides a method for assessing psychological states based on multimodal data and model fine-tuning, including: Step 101: Extract features from the target multimodal sensing data of the individual to be evaluated to obtain the initial modal features corresponding to different modalities.

[0022] In this invention, a collaborative multimodal sensing front-end (including millimeter-wave radar, RGB camera, and event camera with optimized parameters) and a high-precision synchronization module are used to collect physiological and behavioral data of the individual to be evaluated. This data covers multiple aspects; for example, data acquired through different devices such as millimeter-wave radar, RGB camera, and event camera can reflect the individual's physiological and behavioral characteristics from different perspectives.

[0023] In this invention, specialized feature extraction networks for each modality, such as convolutional neural networks, facial action unit (AU) detection networks, pose estimation networks, recurrent neural networks (for extracting micro-expression category probability time series), and graph neural networks (for extracting saccade parameter time series), can be used to extract features from the collected data of different modalities. Each network can be designed for the characteristics of specific modal data, enabling the extraction of high-level semantic feature sequences from the raw data. These high-level semantic feature sequences constitute the initial modal features corresponding to different modalities. For example, convolutional neural networks are used to extract features related to electrocardiograms, and AU detection networks are used to extract features related to facial action units. This step is fundamental to improving the comprehensiveness of the assessment because multimodal data can provide richer information, reflecting the individual's state from multiple dimensions.

[0024] Step 102: Tokenize each of the modal initial features and concatenate the tokenized modal initial features to obtain the target token sequence.

[0025] In this invention, a key module, namely the physiological behavior tokenizer, is constructed to process heterogeneous feature sequences of different modalities. These heterogeneous feature sequences include heart rate variability (HRV) time series, AU intensity sequences, micro-dynamic feature vector sequences, etc.

[0026] In this invention, the physiological behavior tokenizer includes sub-modules such as modality-specific embedding, temporal segmentation and encoding, and modality type and location encoding. Through the actions of these sub-modules, feature sequences from different modalities are converted into a unified input token sequence that can be processed by large Transformer models. For example, the modality-specific embedding sub-module performs specific embedding processing on the characteristics of different modal features, giving them a format suitable for subsequent model processing; the temporal segmentation and encoding sub-module divides and encodes the feature sequence along the temporal dimension to preserve temporal information; and the modality type and location encoding sub-module adds corresponding encoding information to features of different modalities and locations, enabling the model to distinguish features from different modalities and locations.

[0027] Furthermore, the initial features of each modality after tokenization are concatenated to obtain the target token sequence. This target token sequence integrates information from different modalities, providing a unified data format for subsequent input into the comprehensive psychological state assessment model for further processing.

[0028] Step 103: Input the target token sequence into the trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed output by the comprehensive psychological state assessment model; wherein, the comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset; each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks; each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment label.

[0029] In this invention, a pre-trained Transformer encoder model with strong general representation capabilities (e.g., BERT-Large, RoBERTa-Large, or the encoder part of Perceiver IO, with a parameter count exceeding 100M) is selected as the core feature learning and integration engine. These pre-trained models are trained on large-scale data and possess powerful feature extraction and integration capabilities, providing a solid foundation for subsequent psychological state assessment.

[0030] Furthermore, multiple lightweight, independent prediction heads for different mental state assessment tasks are connected in parallel above the top-level output of the Transformer encoder (such as [CLS]-tagged representations or pooling results of sequence outputs). These are known as mental state prediction task heads. These different mental state assessment tasks include emotion classification, anxiety / depression severity regression, cognitive load level assessment, and Big Five personality trait rating. Each prediction head is constructed according to a specific mental state prediction task and can output prediction results for the corresponding task.

[0031] During model fine-tuning, a progressive fine-tuning strategy guided by data curated by psychiatric experts can be employed to fine-tune the pre-trained model. First, the pre-trained model is fine-tuned using a large-scale, multimodal clinical dataset of mental states, constructed and annotated with high quality by a team of psychiatric experts. The fine-tuning process employs progressive unfreezing and tiered learning rates, prioritizing the training of the mental state prediction task head and the top layer of the model, then gradually expanding to the lower layers. This fine-tuning approach efficiently transfers knowledge while maintaining model stability. Simultaneously, psychiatric knowledge guides the design of the loss function and the weight balancing for multi-task learning, enabling the model to better adapt to mental state assessment tasks, resulting in a well-trained comprehensive mental state assessment model.

[0032] In this invention, the target token sequence obtained in step 102 is input into the trained comprehensive psychological state assessment model. The target token sequence is feature-learned and integrated through a pre-trained Transformer encoder. Then, each multi-task psychological state prediction head makes predictions for different psychological state prediction tasks based on the learned features. Finally, the psychological state assessment results of the individual to be assessed are output, including emotion classification results, anxiety / depression severity, cognitive load level, Big Five personality trait scores, etc.

[0033] After the model outputs a comprehensive assessment of an individual's various psychological states, combining model interpretability techniques (such as attention visualization and feature contribution analysis) with a psychiatric knowledge base provides a degree of explanation for the decision-making basis. This helps to understand the rationale behind the model's assessment results, increasing their credibility and comprehensibility. For example, attention visualization can reveal which parts of the input data the model focuses on, while feature contribution analysis clarifies the degree to which different features contribute to the assessment results.

[0034] Based on the above embodiments, the step of extracting features from the target multimodal sensing data of the individual to be evaluated to obtain the initial modal features corresponding to different modalities includes: Based on a multimodal sensing front end, millimeter-wave radar data, RGB camera data, and event camera data corresponding to the individual to be evaluated are collected; Feature extraction was performed on the millimeter-wave radar data to obtain event sequences of heart rate variability parameters, time sequences of key morphological features of electrocardiograms, and time sequences of micromotor pattern features. Feature extraction is performed on the RGB camera data to obtain time series of facial motion unit intensity features, time series of head three-dimensional rotation angle features, and time series of body key point three-dimensional coordinate features. Feature extraction is performed on the event camera data to obtain time series of micro-expression category probability features, time series of saccade parameter features, and time series of pupil dynamic features; Based on the heart rate variability parameter event sequence, the electrocardiogram key morphological feature time sequence, the micro-motion pattern feature time sequence, the facial motion unit intensity feature time sequence, the head three-dimensional rotation angle feature time sequence, the body key point three-dimensional coordinate feature time sequence, the micro-expression category probability feature time sequence, the saccade parameter feature time sequence, and the pupil dynamic feature time sequence, the modal initial features corresponding to different modalities are obtained.

[0035] In this invention, the multimodal sensing front end is composed of various types of sensors, including millimeter-wave radar, RGB camera and event camera.

[0036] Specifically, millimeter-wave radar can employ 79GHz FMCW (Frequency Modulated Continuous Wave) technology, with a 4GHz bandwidth, using a 4Tx / 8Rx MIMO (Multiple-Input Multiple-Output) configuration, achieving a frame rate of 500Hz. This configuration enables millimeter-wave radar to transmit and receive signals at higher frequencies, thereby acquiring more detailed target information, such as detecting minute movements and physiological characteristics of the individual being evaluated.

[0037] RGB cameras are equipped with a global shutter, enabling them to capture image data at a resolution of at least 1080p and a frame rate of 60fps. The global shutter ensures that all pixels are exposed simultaneously when capturing an image, avoiding image distortion caused by rolling shutter speeds, making it suitable for capturing rapidly changing facial expressions and body movements.

[0038] Event cameras, such as the DAVIS 346 or equivalent, offer an equivalent temporal resolution exceeding 1 kHz and a dynamic range exceeding 120 dB. Unlike traditional cameras, event cameras only output data when they detect changes in pixel brightness. This characteristic enables them to efficiently capture rapidly changing dynamic information, such as micro-expressions and saccades.

[0039] In this invention, all sensors achieve sub-millisecond synchronization through a synchronization mechanism, ensuring that the data collected by different sensors are highly consistent in time and accurately correspond to the same moment, providing a reliable time reference for subsequent feature extraction and analysis. For example, when analyzing a certain action or expression of an individual to be evaluated, physiological signals detected by millimeter-wave radar, facial images captured by an RGB camera, and dynamic changes recorded by an event camera can be acquired simultaneously, ensuring the integrity and accuracy of the data.

[0040] Furthermore, feature extraction is performed on different modes to obtain the initial modal features corresponding to each mode. Specifically, feature extraction is performed on millimeter-wave radar data to obtain the initial modal features, which include the event sequence of heart rate variability parameters, the time series of key morphological features of electrocardiogram, and the time series of micromotor pattern features.

[0041] The extraction process of heart rate variability (HRV) parameter event sequences involves calculating a set of HRV parameters in 5-second windows, including SDNN (standard deviation of normal sinus intervals), RMSSD (root mean square of the difference between adjacent normal sinus intervals), pNN50 (percentage of adjacent normal sinus intervals exceeding 50ms), LF (low-frequency power), HF (high-frequency power), and LF / HF (low-frequency to high-frequency power ratio). These parameters reflect the activity of the cardiac autonomic nervous system.

[0042] The process of extracting key morphological features time series from electrocardiogram (ECG): Key morphological features time series of ECG are extracted from millimeter-wave radar data, such as P wave / T wave amplitude and QTc interval. Among them, the P wave represents atrial depolarization and the T wave represents ventricular repolarization. Their amplitude and morphological changes can reflect the electrical activity of the heart. The QTc interval is the heart rate-corrected QT interval, which is used to assess the timing of ventricular repolarization and the electrical stability of the heart.

[0043] The process of extracting micromotor pattern characteristics from time series: By processing millimeter-wave radar data, the time series of micromotor pattern characteristics of the individual to be assessed are quantified, such as the energy of anxiety tremors and the frequency of restlessness patterns. The energy of anxiety tremors reflects the intensity of the tiny tremors caused by anxiety or other emotions; the frequency of restlessness patterns reflects the degree of unease the individual experiences during the assessment process.

[0044] Furthermore, feature extraction was performed on the RGB camera data to obtain time series of facial motion unit intensity features, time series of head 3D rotation angle features, and time series of body key point 3D coordinate features.

[0045] The extraction process of facial action unit intensity feature time series: Image data acquired by an RGB camera is analyzed using a facial AU (Action Unit) detection network, outputting a set of intensity values ​​from AU01 to AU28 for each frame. AU is the basic unit of facial muscle movement; different AU combinations can express various facial expressions, such as happiness, sadness, and anger. By detecting the intensity of AUs, the intensity and changes of facial expressions can be quantified, thereby understanding an individual's emotional state.

[0046] The process of extracting the time series of head 3D rotation angle features: The 3D rotation angles of the head, including Roll, Pitch, and Yaw, are extracted from RGB camera data using a 3D pose estimation network. These angles can describe the rotation of the head in 3D space and reflect the individual's head movement posture, such as whether the head is tilted or turned, which helps to analyze the individual's attention direction and body posture.

[0047] The process of extracting the 3D coordinate features of body key points over time: 3D pose estimation networks can also extract the 3D coordinate time series of body key points. Body key points include parts such as shoulders, elbows, wrists, hips, knees, and ankles. By obtaining the 3D coordinates of these key points, it is possible to reconstruct an individual's body posture and movement trajectory, and understand the individual's body movements and posture changes during the assessment process, such as whether they are unstable or leaning forward.

[0048] Furthermore, feature extraction was performed on the event camera data to obtain time series of micro-expression category probability features, time series of saccade parameter features, and time series of pupil dynamic features.

[0049] The process of extracting the time series of micro-expression category probability features involves processing event camera data and outputting the time series of micro-expression category probabilities in 100ms windows. Micro-expressions are fleeting facial expressions that can realistically reflect an individual's inner emotions. By detecting the category probabilities of micro-expressions, we can understand the hidden emotional changes of an individual within a short period of time.

[0050] The process of extracting time series features of saccade parameters: Extracting saccade parameters, including frequency and amplitude time series, from event camera data. Saccades are rapid eye movements, and their frequency and amplitude can reflect an individual's level of attention and visual exploration behavior. For example, frequent saccades may indicate a lack of concentration, while larger saccades may indicate that the individual is rapidly scanning their surroundings.

[0051] The process of extracting time series of pupil dynamic features involves extracting time series of dynamic pupil features, such as changes in pupil size. Pupil size is affected by light intensity and individual emotional state; pupils typically dilate when emotionally agitated or stressed. Analyzing pupil dynamic features can provide further insight into an individual's psychological state and physiological responses.

[0052] Finally, the various feature time series extracted from millimeter-wave radar data, RGB camera data, and event camera data were used as the initial modal features corresponding to different modalities. These initial modal features exist in the form of time series, and within a unified analysis window (e.g., 5-10 seconds), each modality has a series of feature vectors arranged in chronological order. These initial modal features provide a rich data foundation for subsequent comprehensive psychological state assessment, reflecting the physiological and behavioral characteristics of the individual being assessed from multiple dimensions, and helping to more accurately assess the individual's psychological state.

[0053] Based on the above embodiments, the step of tokenizing each of the modal initial features and concatenating the tokenized modal initial features to obtain the target token sequence includes: Based on a preset analysis time window, the initial features of each modality are aligned with the time window to obtain a sequence of modal features with the same time window. Based on the modality type, the modality feature sequences are divided into corresponding time segments to obtain the time segment features of each modality feature sequence; The time segment features are tokenized to obtain multiple time segment token sequences corresponding to each tokenized modal initial feature; Each modality type embedding vector and time segment position embedding vector are added to each of the time segment token sequences to obtain each modality token sequence; wherein, the modality type embedding vector is used to distinguish the time segment token sequences of different modalities; the time segment position embedding vector is used to represent the time order information of each time segment token sequence within its respective time window; The global context token is concatenated with each of the modal token sequences to obtain the target token sequence; wherein the global context token is used to aggregate the encoded global information.

[0054] In this invention, a common analysis time window is preset, for example, a pre-defined analysis time window. The preset analysis time window length is 10 seconds. Setting this length requires considering multiple factors. On one hand, it needs to be long enough to capture meaningful psychophysiological dynamics, as changes in psychological states typically require a certain time span to be fully presented. On the other hand, it cannot be too long, otherwise it will significantly increase the computational load and reduce the system's real-time performance. The preset analysis time window length can be determined with the help of the duration characteristics of common psychological states.

[0055] Different modal initial feature sequences have different sampling rates and lengths. For example, the HRV feature sequence of millimeter-wave radar. It could be a HRV vector every 5 seconds; while the facial action unit intensity feature time series It could be an AU vector per frame; time series of micro-expression category probability features. , corresponding to the time step feature dimension, where, It is modal The sequence length, These are its feature dimensions. Alignment is required to enable analysis within a unified timeframe.

[0056] In this invention, for modalities with sampling rates higher than the target token sequence frequency, such as RGB frame-level AU sequences, temporal downsampling or feature aggregation can be performed within a preset analysis time window. For example, a window-level feature vector can be obtained by averaging or maximizing the AU vectors of all frames within the window, or by encoding with a small 1D CNN (one-dimensional convolutional neural network) or LSTM (long short-term memory network), thereby converting high-frequency sampled data into data that matches the target window frequency.

[0057] For modalities with a sampling rate lower than the target token sequence frequency, such as an HRV sequence calculated every 5 seconds, interpolation or padding operations are performed. Interpolation estimates data for other time points within the window based on known data points using mathematical methods (such as linear interpolation or spline interpolation). Padding involves repeatedly filling existing data points into the corresponding positions within the window to ensure sufficient data points within the preset analysis window. Through these operations, a set of modal feature vectors corresponding to each preset analysis window is generated, ensuring that all modal feature sequences share the same time window.

[0058] Furthermore, for each modality (after alignment and normalization) feature sequence That is, the modal feature sequence, where, It is a preset analysis time window Number of patches in the time segment within the partition. This is the feature dimension of the segment. Perform the following operations: Time segmentation (Patchification): dividing modalities m Within the preset analysis time window The feature sequence within is further divided into A series of consecutive time segments (patches) of equal or overlapping length. For example, if an HRV sequence has two 5-second vectors within a 10-second interval, then... =2, that is, it is divided into 2 time segments, each segment corresponding to a 5-second HRV vector; if the AU sequence is frame-level (e.g., 600 frames @ 60fps), it can be divided into The dataset is divided into 10 one-second segments, each containing 60 frames of data. Features within each segment are then aggregated, for example, by averaging or using other aggregation methods, to obtain the features for each time segment. This partitioning breaks down the features of a long sequence into multiple shorter time-segment features for subsequent processing.

[0059] Patch Embedding: The features of each time segment (which may be vectors themselves, or obtained through convolution / pooling of finer-grained features within the segment) are mapped to a fixed dimension through a modality-specific learnable linear projection layer or a small MLP (Multilayer Perceptron). Patch tokens, thus for each modality m Generate a containing A sequence of tokens: That is, the sequence of tokens for multiple time segments corresponding to the initial features of each tokenized modality, where, .

[0060] Furthermore, the obtained time segment token sequences are labeled and modal / positional encoded, as follows: Modality Type Embedding: a sequence of time-slice tokens for each modality. Add a learnable modality-type embedding vector to each. ,in, This is a time segment token sequence corresponding to millimeter-wave radar data. This is a sequence of tokens representing time segments corresponding to RGB camera data. This represents the sequence of tokens for time segments corresponding to event camera data. Data from different modalities possess different features and semantic information. By incorporating modality type embedding vectors, the pre-trained Transformer model can distinguish tokens from different sources. For example, millimeter-wave radar data, RGB camera data, and event camera data have different physical meanings and feature representations. Modality type embedding vectors help the model identify these differences, thereby better handling data from different modalities.

[0061] Temporal Patch Position Embedding: Temporal patch tokens within each modality Add a learnable fragment-level absolute position embedding vector This refers to the temporal segment location embedding vector. Temporal sequence information is crucial for understanding physiological behavioral data; for example, changes in an action or facial expression occur sequentially. By incorporating the temporal segment location embedding vector, each temporal segment token can be preserved within a predefined analysis time window. The temporal sequence information within the data allows the model to perceive its temporal characteristics. For example, the positional encoding of the first time segment is different from that of the second time segment, and the model can use these encodings to understand the temporal order. After adding the corresponding modality type embedding vector and time segment position embedding vector to each time segment token sequence, the respective modality token sequences are obtained.

[0062] Global Context Tag Token ([CLS] Token): A learnable special [CLS] token is added before concatenating all modal token sequences. This token's role is to aggregate global information after Transformer encoding for tasks such as classification or regression. For example, when determining an individual's mental state, the [CLS] token can synthesize information from all modalities, combining the global [CLS] token with the individual modal token sequences (with modality type embeddings and positional encodings) in a predetermined order (e.g., ...). ,in It is a sequence of time segment tokens (with modality type embedding vector and time segment position embedding vector added) concatenated to form a single, long ( The input token sequence (i.e. the target token sequence) is then fed into a large Transformer encoder to give an overall judgment result.

[0063] In this invention, a preset analysis time window is used. It can be set to 10-30 seconds, which is long enough to capture meaningful psychophysiological dynamics, but not too long to cause excessive computation and reduced real-time performance. The duration characteristics of common psychological states can be used to help determine the duration.

[0064] Number of time segments (For each modality), there can be 5-10 segments. The number of segments affects the length of the final input sequence and the smallest temporal granularity that the model can perceive.

[0065] Fragment Token Dimension It can be 768 or 1024, consistent with the hidden layer dimension of subsequent large Transformer models.

[0066] This invention addresses a specific tokenization strategy for heterogeneous multimodal physiological behavior time series. In particular, the combination of "time segment tokenization" with "modality type embedding" and "segment-level position encoding" can effectively convert multi-source data with different sampling rates and feature dimensions into an input format that can be uniformly processed by large Transformer models while preserving modality specificity and temporal information. This lays the foundation for the successful application of large models on complex physiological behavior data.

[0067] Figure 2 The flowchart of the physiological behavior tokenizer provided by the present invention can be referred to. Figure 2 As shown, multimodal feature sequences (HRV, ECG morphology, micro-motion feature sequences; AU intensity, 3D pose, head motion sequences; micro-expressions, eye movements, pupil dynamic sequences) from millimeter-wave radar, RGB cameras, and event cameras are acquired. Then, these multimodal feature sequences are aligned to a time window and normalized. Subsequently, each modality is segmented into time segments and embedded with dedicated segments to generate modality token sequences. Next, a special [CLS] token, modality type embedding, and time segment position encoding are added to finally construct a unified input token sequence (i.e., the target token sequence) for use by large Transformer models.

[0068] Based on the above embodiments, the comprehensive psychological state assessment model is trained through the following steps: Based on the psychological evaluation results of sample fragments corresponding to each modal token sequence in the sample token sequence, the psychological state evaluation label is constructed. Based on the psychological state assessment labels, each modal token sequence in the sample token sequence is labeled to obtain the multimodal psychological state clinical dataset; Based on the multimodal psychological state clinical dataset, the pre-trained Transformer model and the multi-task psychological state prediction head are progressively fine-tuned to obtain the comprehensive psychological state assessment model. The multi-task mental state prediction head includes at least an emotion classification head, an anxiety / depression severity regression head, a cognitive load classification head, and a personality trait regression head. The emotion classification head consists of a fully connected layer and a Softmax activation function, used to output the probability distribution of discrete emotion categories. The loss function of the emotion classification head is cross-entropy loss. The anxiety / depression severity regression head consists of a fully connected layer and a linear activation function, used to predict continuous severity scores of anxiety and / or depression. The loss function of the anxiety / depression severity regression head is mean squared error or Huber loss. The cognitive load classification head consists of a fully connected layer and a Softmax activation function, used to output the probability distribution of cognitive load levels. The loss function of the cognitive load classification head is cross-entropy loss. The personality trait regression head includes five parallel regression subheadings, each consisting of a fully connected layer and a linear activation function, used to output continuous scores for each dimension of the Big Five personality traits.

[0069] In this invention, the sample token sequence may contain data of multiple modalities, such as text, images, and speech, with each modality's token sequence corresponding to a segment of information from the sample. By performing psychological assessments on these segments—for example, analyzing the emotions expressed in text segments or the states of people presented in image segments—psychological assessment results are obtained for each sample segment. Then, these psychological assessment results are used to construct psychological state assessment labels. These labels serve as crucial evidence for subsequent annotation and model training, summarizing the characteristics of the sample across different psychological state dimensions, such as emotion category and degree of anxiety or depression.

[0070] Furthermore, based on the constructed psychological state assessment labels, each modal token sequence in the sample token sequence is labeled. The psychological state assessment labels are accurately assigned to the corresponding modal token sequences, ensuring that each sequence has a clear psychological state identifier. After labeling, a multimodal psychological state clinical dataset is obtained. This dataset contains data from multiple modalities and their corresponding psychological state assessment labels, providing rich data resources for subsequent model training and helping the model learn the relationship between different modalities and psychological states.

[0071] In this invention, a pre-trained Transformer model is used as the basic framework. The pre-trained Transformer model is pre-trained on ultra-large-scale general data (such as text, images, or multimodal data), for example, BERT-Large (Bidirectional Encoder Representations from Transformers): 24 layers, 1024 hidden units, 16 attention heads, approximately 340M parameters; RoBERTa-Large (Robustly Optimized BERT Pretraining Approach): The structure is similar to BERT-Large, but the pretraining strategy is superior.

[0072] Perceiver IO: A general-purpose, modality-independent Transformer architecture capable of handling high-dimensional, multimodal inputs. It maps the input to a fixed-size latent space through a cross-attention mechanism, which is then processed by a Transformer encoder, whose encoder part can serve as the core. These large-scale models have learned powerful and general-purpose sequence modeling and feature representation capabilities, and can be effectively transferred to downstream tasks through fine-tuning.

[0073] In this invention, a pre-trained Transformer model receives a sequence of target tokens and outputs a context-dependent hidden state vector for each input token. Typically, the final hidden state vector corresponding to the first special marker [CLS] is taken. (in, The hidden layer dimension of the Transformer (e.g., 768 or 1024) serves as an aggregated representation of the entire input sequence for subsequent classification or regression tasks. Alternatively, the hidden states of all output tokens can be averaged or attention-pooled.

[0074] Furthermore, in the output of the encoder core of the pre-trained Transformer model... Multiple independent, lightweight mental state prediction task heads are connected in parallel above this, forming a multi-task mental state prediction head. These heads are responsible for different mental state assessment subtasks, including an emotion classification head, an anxiety / depression severity regression head, a cognitive load classification head, and a personality trait regression head. This multi-task learning (MTL) architecture allows the model to learn and predict multiple related mental states simultaneously, and improves overall performance and generalization ability by sharing the underlying Transformer encoder to utilize common information between tasks.

[0075] Next, based on a multimodal clinical dataset of mental states, the pre-trained Transformer model and the multi-task mental state prediction head were progressively fine-tuned. Progressive fine-tuning is a method of gradually adjusting model parameters by iteratively training on the dataset to gradually adapt the model to a specific mental state assessment task. During this process, the model utilizes the labeled information in the dataset to continuously optimize its parameters, thereby improving the prediction accuracy for different mental states. After progressive fine-tuning, a comprehensive mental state assessment model was obtained. This model can simultaneously process data from multiple modalities and assess various mental states, exhibiting strong generalization ability and practicality.

[0076] In this invention, by sharing a powerful pre-trained Transformer encoder and building multiple lightweight mental state prediction task heads on top of it, parameter-efficient multi-task learning is achieved. This not only allows for the simultaneous evaluation of multiple mental states but also improves the performance of each individual task through implicit knowledge transfer between tasks, especially when labeled data is relatively scarce. Furthermore, the parallel design of the mental state prediction task heads makes it easy to add or remove the dimensions of mental states to be evaluated as needed.

[0077] Specifically, the main structure of the multi-task mental state prediction head provided by this invention is as follows: The emotion classification head consists of one or two fully connected layers (e.g., The system consists of a fully connected layer followed by a Softmax activation function. The fully connected layer maps the input feature vector to spaces representing different emotion categories, while the Softmax activation function transforms these output values ​​into a probability distribution, ensuring that the sum of the probabilities of all emotion categories is 1. The emotion classification head is used for output. The probability distribution of discrete emotion categories, for example, can output the probability of emotion categories such as happiness, sadness, anger, fear, surprise, and disgust, or a more detailed valence-arousal quadrant, thereby determining the main emotion expressed by the sample.

[0078] The loss function for the emotion classification head is cross-entropy loss. Cross-entropy loss is commonly used in classification problems to measure the difference between the probability distribution of the model's output and the true label. By minimizing the cross-entropy loss, the model's predictions can be made closer to the true emotion category.

[0079] Anxiety / depression severity regression head consists of one or two fully connected layers (e.g., The fully connected layer is composed of a linear activation function (or Sigmoid / Tanh, to constrain the output range). The fully connected layer maps the input features to a numerical space, and the linear activation function maintains the continuity of the output. Alternatively, activation functions such as Sigmoid / Tanh can be used to constrain the output range to better reflect reality. It should be noted that this invention can also set separate regression heads for anxiety and depression.

[0080] Anxiety / depression severity regression heads are used to predict continuous severity scores for anxiety and / or depression. For example, they can predict the GAD-7 total score (Generalized Anxiety Disorder Scale total score) and / or the PHQ-9 total score (Patient Health Questionnaire Depression Symptom Scale total score), thereby assessing the level of anxiety and depression in a sample.

[0081] The loss function for the anxiety / depression severity regression head can be either mean squared error (MSE) loss or Huber loss. MSE loss calculates the sum of squared errors between predicted and true values; minimizing MSE loss allows the model's predictions to be closer to the true scores. Huber loss, on the other hand, is more robust to outliers; when outliers exist in the data, using Huber loss can reduce their impact on model training.

[0082] The cognitive load classifier head is similar to the emotion classifier head, consisting of fully connected layers (e.g., The fully connected layer is followed by a Softmax activation function. The fully connected layer maps the input features to spaces with different cognitive load levels, and the Softmax activation function transforms the output into a probability distribution.

[0083] The cognitive load classification header is used to output the probability distribution of cognitive load levels. For example, it can output the probability of low, medium, and high cognitive load levels, or the probability distribution of the corresponding rating intervals of scales such as NASA-TLX, thereby assessing the cognitive load of the sample.

[0084] The loss function for the cognitive load classifier is cross-entropy loss. Similar to the sentiment classifier, minimizing the cross-entropy loss makes the model's predictions closer to the true cognitive load level.

[0085] The personality trait regression head consists of five parallel regression subheadings, each similar to the anxiety / depression severity regression head, and is constructed from fully connected layers (e.g., The input features are then composed of a linear activation function. Each regression subheader corresponds to one of the five dimensions of the Big Five personality traits (openness (O), conscientiousness (C), extraversion (E), agreeableness (A), and neuroticism (N), mapping the input features to the numerical space of the corresponding dimension).

[0086] The personality trait regression head is used to output continuous scores for each dimension of the Big Five personality traits. For example, it corresponds to factor scores on the BFI-2 scale, thereby assessing the personality traits of the sample.

[0087] The loss function of the personality trait regression head uses a weighted sum of the MSE losses of five dimensions. That is, the MSE loss of each dimension is calculated separately and then they are summed (the weights can be adjusted according to the importance of different dimensions). By minimizing this summation loss, the model's predictions on each dimension are closer to the true scores.

[0088] Based on the above embodiments, the progressive fine-tuning of the pre-trained Transformer model and the multi-task mental state prediction head based on the multimodal mental state clinical dataset to obtain the comprehensive mental state assessment model includes: The encoder parameters of the pre-trained Transformer model are frozen, and the multi-task mental state prediction head is initially trained based on the multimodal mental state clinical dataset and the first learning rate to obtain the initially trained multi-task mental state prediction head. Unfreeze the top preset number of Transformer layers in the encoder of the pre-trained Transformer model whose encoder parameters are all frozen to obtain the pre-trained Transformer model after initial unfreezing. Based on the multimodal mental state clinical dataset and the second learning rate, the pre-trained Transformer model after initial thawing and the pre-trained multi-task mental state prediction head are jointly trained to obtain the pre-trained Transformer model and the pre-trained multi-task mental state prediction head after initial fine-tuning, respectively. Unfreeze all Transformer layers in the encoder of the pre-trained Transformer model after initial fine-tuning to obtain the fully unfrozen pre-trained Transformer model. Based on the multimodal mental state clinical dataset and the third learning rate, the fully unfrozen pre-trained Transformer model and the preliminarily fine-tuned multi-task mental state prediction head are jointly fine-tuned to obtain the comprehensive mental state assessment model. The first learning rate, the second learning rate, and the third learning rate decrease sequentially.

[0089] In this invention, the progressive fine-tuning strategy employs a phased, top-to-bottom progressive unfreezing and differential learning rate fine-tuning method to fine-tune the model. The specific process is as follows: The first step involves training only the multi-task mental state prediction head: All core parameters of the encoder in the pre-trained Transformer model are frozen, ensuring that the encoder parameters remain unchanged and are not updated during training. Then, the multi-task mental state prediction head is initially trained using a multimodal mental state clinical dataset and a relatively large learning rate (i.e., the initial learning rate).

[0090] This stage aims to enable the multi-task mental state prediction head to quickly learn the mapping from general features extracted from the pre-trained model to specific mental state labels. Because the pre-trained model has already learned rich feature representations on large-scale general data, freezing the encoder parameters can ensure that these general features are not destroyed, while using a relatively large initial learning rate allows the prediction head to quickly adapt to new mental state evaluation tasks and converge to the initial learning state, resulting in the pre-trained multi-task mental state prediction head.

[0091] The second step involves gradually unfreezing and fine-tuning the top layers of the Transformer: After the multi-task mental state prediction head has initially converged, the top predetermined number of Transformer layers in the encoder of the pre-trained Transformer model, whose encoder parameters are all frozen, are unfrozen. In this invention, the predetermined number of layers at the top can be determined according to the actual situation, for example, the last 1 / 4 or 1 / 3 of the core layers of the Transformer encoder. After unfreezing, the parameters of these layers will be updated in subsequent training.

[0092] In this invention, some top-level Transformer layers are unfrozen because these layers are closer to the output and their parameters are more task-specific. By unfreezing these layers and allowing them to participate in training, the model can gradually adapt to the details and requirements of specific psychological state evaluation tasks while maintaining its ability to extract general features from the lower layers, resulting in a pre-trained Transformer model that has been initially unfrozen.

[0093] Furthermore, based on a multimodal mental state clinical dataset and a second learning rate, the pre-trained Transformer model after initial unfreezing and the pre-trained multi-task mental state prediction head are jointly trained. During training, the parameters of the unfrozen top-level Transformer layer and the prediction head are updated according to the loss function. Through joint training, the unfrozen top-level Transformer layer and the prediction head work together better, further optimizing the model's ability to extract and map mental state features. In this invention, a smaller second learning rate (compared to the first learning rate) is used because the model has begun to gradually adapt to the task, and a smaller learning rate can avoid excessively drastic parameter updates, ensuring training stability, and obtaining the pre-trained Transformer model and the pre-trained multi-task mental state prediction head after initial fine-tuning.

[0094] The third step is to fine-tune more or all Transformer layers: Unfreeze all Transformer layers in the encoder of the pre-trained Transformer model after the initial fine-tuning, so that all layers of the encoder will participate in subsequent training and their parameters can be updated according to the training data.

[0095] In this invention, based on performance on the validation set, after the model has achieved certain training results in the previous stages, further unfreezing more or even all layers allows the model to adapt more comprehensively to specific tasks. After being fully unfrozen, the model can fully utilize the information from all layers to learn the mental state evaluation task, further improving the model's performance and generalization ability, resulting in a fully unfrozen pre-trained Transformer model.

[0096] Furthermore, based on a multimodal mental state clinical dataset and a third learning rate, the fully unfrozen pre-trained Transformer model and the initially fine-tuned multi-task mental state prediction head were jointly fine-tuned. During the fine-tuning process, the parameters of the entire model were precisely adjusted according to the loss function. Using a very small third learning rate (decreasing sequentially compared to the first two learning rates) for end-to-end fine-tuning allows the model to further optimize parameters when it is already close to convergence, making the model's prediction results more accurate and stable. Through this fine-tuning, a comprehensive mental state assessment model capable of accurately evaluating multiple mental states is finally obtained.

[0097] Throughout the training process, differential learning rates were employed, meaning the first, second, and third learning rates decreased sequentially, setting different learning rates for layers of different depths. Generally, parameters closer to the input layer (i.e., the more general feature extractors learned in the pre-trained model) used smaller learning rates to avoid corrupting their learned knowledge; parameters closer to the output layer (more task-specific) could use relatively larger learning rates. In the initial training phase of the prediction head, a larger learning rate was needed for rapid learning; during the joint training phase of unfreezing the layers, the learning rate was appropriately reduced to ensure training stability; and in the joint fine-tuning phase after complete unfreezing, a very small learning rate was used for fine-tuning to prevent excessive parameter updates from disrupting the learned features and mapping relationships, thus ensuring the model can efficiently and stably adapt to the specific task and produce accurate results.

[0098] In this invention, the multimodal psychological state clinical dataset includes a healthy control group, clinical / subclinical diagnostic groups for various common mental disorders (such as GAD, SAD, MDD, PTSD), and subjects whose specific psychological states were induced in the laboratory using standardized paradigms (emotion induction, cognitive load tasks, simulated social stress). Diversity of demographic variables such as age, gender, and cultural background is ensured.

[0099] The process of synchronously acquiring multimodal data in the multimodal psychological state clinical dataset can refer to the processing process of millimeter-wave radar data, RGB camera data and event camera data in the above embodiments.

[0100] The multidimensional labeling process can be carried out by a labeling team of no fewer than three psychiatrists and clinical psychologists with extensive clinical experience (e.g., attending physician or above, >5 years of experience), and grouped using gold standard clinical diagnostic interview tools (such as MINI Plus, SCID-5-CV).

[0101] For each data acquisition segment (e.g., corresponding to a preset analysis time window) Using a set of validated, multidimensional psychological assessment scales (such as HAM-A, HAM-D, GAD-7, PHQ-9, PSS, BFI-2, PANAS-X, etc.) and behavioral observation records (such as video-based specific behavior coding), the data are independently scored or categorized by annotation experts.

[0102] For dynamic states such as emotions, continuous labeling (such as using joystick to record two-dimensional valence-arousal) or event-based fragment labeling can be used.

[0103] Furthermore, rigorous annotator training, annotation guideline development, and consistency assessment among annotators are implemented (e.g., calculating Cohen's Kappa or ICC, requiring a good or excellent level, such as Kappa > 0.75). Inconsistent annotations are collectively discussed and ultimately decided.

[0104] Ultimately, a dataset with a sufficient sample size (e.g., hundreds of subjects, each with multiple data fragments, totaling tens of thousands of labeled samples) and extremely high label quality is constructed, namely the multimodal psychological state clinical dataset.

[0105] Based on the above embodiments, the total loss of the multi-task mental state prediction head is obtained by weighted summation of the losses of each mental state prediction task head in the multi-task mental state prediction head, wherein the weight coefficient corresponding to each mental state prediction task head is obtained based on the homoscedastic uncertainty learning of the mental state prediction task head during the training process.

[0106] In this invention, in a multi-task mental state assessment scenario, the multi-task mental state prediction head includes multiple different mental state prediction task heads, each task head being responsible for predicting a specific mental state.

[0107] To comprehensively evaluate the model's overall performance across all these mental state prediction tasks, a total loss function needs to be calculated to guide model training optimization. This total loss... Using a weighted summation method, its mathematical expression is: ; in, This represents the total number of tasks related to predicting mental states. It is the first The loss of each task (such as cross-entropy or MSE). It is the first The weighted coefficients correspond to the losses of each task. By using this weighted summation method, the contribution of different tasks to the total loss can be adjusted more flexibly, thereby guiding the model's learning focus and optimization direction on different tasks.

[0108] Homoscedastic uncertainty refers to the phenomenon where the variance of a model's prediction error remains constant across different input data during training. In multi-task learning, different tasks may have varying learning difficulties and noise levels. For example, certain psychological states may be more difficult to learn on their corresponding prediction tasks due to complex symptoms and significant individual differences, resulting in a larger variance in the model's prediction error on those tasks; while other psychological states are relatively easy to predict, with smaller variances in their prediction errors. This difference in the variance of prediction errors between different tasks is a manifestation of homoscedastic uncertainty.

[0109] Traditional weighting methods often involve assigning equal or fixed weights to all tasks, or setting fixed weights based on human experience. However, this approach fails to consider the inherent differences between tasks. This invention employs a dynamic loss balancing strategy based on uncertainty weighting, which automatically learns the optimal task weights based on the homoscedasticity uncertainty of each task during training. For tasks that are more difficult to learn or have more noise, the model assigns them larger weights, allowing the model to focus more on these tasks during training and dedicate more resources to optimization, thereby improving the model's performance on these challenging tasks.

[0110] As training progresses, the model's learning performance on different tasks will continuously change. Some tasks that were initially difficult to learn may gradually become easier with further training, while some tasks that were initially easy may become difficult due to changes in data distribution or other factors. The method of learning weight coefficients based on homoscedasticity uncertainty can dynamically capture these changes and adjust the weights of each task in real time. This allows the model to continuously optimize effectively based on the current task difficulty and learning progress, improving training efficiency and effectiveness.

[0111] Furthermore, this invention employs standard regularization techniques such as Dropout (in the Transformer layer and prediction head) and weight decay to prevent overfitting. Data augmentation techniques (such as applying small perturbations to the input feature sequence and random masking of time segments) can be considered to improve the model's generalization ability.

[0112] This invention, through high-quality, expert-curated clinical datasets, combined with a fine-tuning process involving progressive thawing and differential learning rates, and an intelligent multi-task loss balancing mechanism, maximizes the efficient and stable transfer and specialization of the powerful general representation capabilities of large pre-trained models to the multi-dimensional psychological state assessment tasks of this invention. This significantly differs from existing methods that involve simple, coarse end-to-end fine-tuning of large models, ensuring better performance, stability, and practical value. Furthermore, the deep involvement of psychiatrists (such as providing prior guiding weights as a reference) provides a core guarantee for the successful implementation of this strategy, making the model's learning process more aligned with actual clinical needs.

[0113] Based on the above embodiments, after inputting the target token sequence into the trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed output by the comprehensive psychological state assessment model, the method further includes: Based on the psychological state assessment results, real-time monitoring data of the psychological state indicators of the individual to be assessed, a multi-dimensional psychological state assessment report, and historical change curves of the psychological state indicators are obtained. Based on the real-time monitoring data of the psychological state indicators, the multi-dimensional assessment report of the psychological state, and the historical change curves of the psychological state indicators, a comprehensive psychological state profile of the individual to be assessed is constructed.

[0114] In this invention, based on the results of psychological state assessment, real-time monitoring data of psychological state indicators for the individual to be assessed is constructed either in real time or according to a set period (e.g., daily, weekly). These indicators are key quantitative information for assessing an individual's psychological state, such as stress index, emotional positivity, and anxiety / depression risk level. Real-time monitoring can promptly capture subtle changes in an individual's psychological state, providing timely data support for subsequent analysis and intervention; periodic acquisition helps to observe long-term trends and cyclical change patterns in psychological state.

[0115] Simultaneously, specific scores are provided for each dimension of the psychological state assessment results, and these scores are divided into different levels, such as mild, moderate, and severe anxiety, thus forming a multi-dimensional psychological state assessment report. This detailed scoring and grading can more accurately describe the degree of an individual's performance in different psychological states, providing professionals (such as psychiatrists) with more detailed diagnostic references.

[0116] In this invention, the multidimensional psychological state assessment report not only provides the current assessment results but also compares the individual's assessment results with a normal baseline or the individual's own historical data. The normal baseline can be the average psychological state level of the general population. By comparing with the normal baseline, it can be determined whether the individual's psychological state deviates from the normal range; comparing with the individual's historical data can observe changes in the individual's psychological state, such as whether there has been improvement or deterioration, thereby providing a basis for developing personalized intervention measures.

[0117] Furthermore, based on the psychological state assessment results, dynamic change curves of key psychological state indicators are displayed over a period of time (such as days, weeks, or even months). These curves can intuitively present the evolution trend and fluctuation pattern of an individual's psychological state, helping users and professionals understand the patterns of psychological state changes. For example, by observing the change curve of the stress index, it is possible to discover when an individual experiences greater stress and how stress gradually accumulates or eases over time.

[0118] In this invention, users can select different time ranges to view historical change curves, enabling them to analyze changes in psychological states at different time scales. Short-term time ranges (such as several days) can be used to observe fluctuations in daily psychological states, while long-term time ranges (such as several months) help to identify long-term trends and potential problems in psychological states.

[0119] In this invention, real-time monitoring data of psychological state indicators can be used to construct an overview dashboard, which displays the current values ​​or trends of core psychological state indicators in an intuitive way, either in real time or periodically. For example, the dashboard can display the current values ​​of indicators such as stress index and emotional positivity, as well as their trends over a period of time, in the form of charts (such as bar charts or line charts) or numbers. This allows users to quickly understand their current psychological state and promptly identify any abnormalities. For example, different colors can be used to represent different levels of psychological state (such as green for normal, yellow for mild abnormality, and red for severe abnormality), making it easy for users to quickly determine whether their psychological state is within a healthy range.

[0120] Simultaneously, the specific scores, ratings, and comparisons with baseline data from the multi-dimensional psychological state assessment report are integrated into a comprehensive psychological state profile. Through this detailed report, professional users (such as psychiatrists) can gain a deep understanding of an individual's specific situation across each psychological state dimension, including the degree of performance, differences from normal levels, and individual trends, providing a comprehensive basis for further diagnosis and treatment.

[0121] In this invention, to enhance users' and clinicians' trust and understanding of the assessment results in the comprehensive psychological state profile, it is possible to explore integrating several interpretability technologies. While these technologies are not directly steps in constructing the profile, they can provide reasonable explanations for the assessment results within the profile, further enriching the connotation of the comprehensive psychological state profile. Specifically, this includes the following aspects: Attention Visualization: For Transformer-based models, visualize the weight distribution of their internal self-attention and cross-modal attention (if an explicit cross-modal interaction layer exists). This allows us to understand which input tokens the model prioritizes when predicting mental states, and which time segments or modalities these tokens correspond to. For example, finding that the model prioritizes an individual's heart rate variability data and facial micro-expression features over specific time periods when predicting anxiety states helps explain the basis of the model's evaluation results.

[0122] Feature importance analysis: Employing model-agnostic explanatory methods such as LIME (Local Interpretable Model-agnostic Explanations) or SHAP (SHapley Additive exPlanations), this analysis identifies which primitive physiological / behavioral features (or combinations thereof) contribute most to a specific psychological state assessment result. For example, feature importance analysis revealed that sleep quality, frequency of social activities, and the amount of negative vocabulary used in an emotional diary are the three most important features when assessing an individual's depressive state. This can help users and professionals understand which factors have a significant impact on an individual's psychological state, thus enabling targeted interventions.

[0123] Rule matching and case interpretation based on a psychiatric knowledge base (PKB): The model's output feature patterns and assessment results are compared with typical cases or clinical discrimination rules stored in the psychiatric knowledge base. For example, if the model judges "moderate anxiety," typical physiological behavior patterns supporting this judgment can be retrieved from the psychiatric knowledge base (such as "a persistently high LF / HF ratio in HRV, accompanied by frequent occurrence of a specific micro-expression pattern B, consistent with the description of early manifestations of GAD in the psychiatric knowledge base") as an auxiliary interpretation of the assessment results. This knowledge-based interpretation can help professionals verify the model's assessment results and make comprehensive judgments in conjunction with their own clinical experience, while also making it easier for users to understand the clinical significance of the assessment results.

[0124] Figure 3 This is a schematic diagram of the overall process of the psychological state assessment method based on multimodal data and model fine-tuning provided by the present invention, which can be referred to. Figure 3 As shown, the multimodal sensing front end collects data, then performs dedicated initial feature extraction on each modality. Further, a physiological behavior tokenizer converts these extracted features into a unified token sequence, which is then input into a large-scale pre-trained Transformer encoder. This encoder is then evaluated by a multi-task mental state prediction head (including an emotion classification head, an anxiety / depression severity regression head, a cognitive load classification head, and a personality trait regression head) to obtain a comprehensive mental state profile. The large-scale pre-trained Transformer encoder is fine-tuned using a multimodal mental state clinical dataset, and the entire process is guided by a psychiatric knowledge base (PKB), ultimately outputting a comprehensive mental state profile.

[0125] The following describes the psychological state assessment system based on multimodal data and model fine-tuning provided by the present invention. The psychological state assessment system based on multimodal data and model fine-tuning described below can be referred to in correspondence with the psychological state assessment method based on multimodal data and model fine-tuning described above.

[0126] Figure 4 This is a schematic diagram of the structure of the psychological state assessment system based on multimodal data and model fine-tuning provided by the present invention, as shown below. Figure 4 As shown, this invention provides a psychological state assessment system based on multimodal data and model fine-tuning, including a multimodal data feature extraction module 401, a physiological behavior tokenizer 402, and a comprehensive psychological state assessment module 403. The multimodal data feature extraction module 401 is used to extract features from the target multimodal sensor data of the individual to be assessed, obtaining initial modal features corresponding to different modalities. The physiological behavior tokenizer 402 is used to tokenize each of the initial modal features and concatenate the tokenized initial modal features to obtain a target token sequence. The comprehensive psychological state assessment module 403 is used to... The target token sequence is input into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed, output by the comprehensive psychological state assessment model. The comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset. Each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks. Each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment tag.

[0127] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0128] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communications interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call logical instructions in the memory 503 to execute a psychological state assessment method based on multimodal data and model fine-tuning. This method includes: extracting features from the target multimodal sensor data of the individual to be assessed to obtain initial modal features corresponding to different modalities; tokenizing each of the initial modal features and concatenating the tokenized initial modal features to obtain a target token sequence; inputting the target token sequence into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed output by the comprehensive psychological state assessment model; wherein the comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset; each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks; and each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment label.

[0129] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which, when executed by a computer, enable the computer to execute the psychological state assessment method based on multimodal data and model fine-tuning provided by the above methods. This method comprises: extracting features from the target multimodal sensor data of the individual to be assessed to obtain initial modal features corresponding to different modalities; tokenizing each of the initial modal features and concatenating the tokenized initial modal features to obtain a target token sequence; inputting the target token sequence into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed output by the comprehensive psychological state assessment model; wherein, the comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset; each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks; and each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment label.

[0131] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the psychological state assessment method based on multimodal data and model fine-tuning provided in the above embodiments. The method includes: extracting features from the target multimodal sensor data of the individual to be assessed to obtain initial modal features corresponding to different modalities; tokenizing each of the initial modal features and concatenating the tokenized initial modal features to obtain a target token sequence; inputting the target token sequence into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed output by the comprehensive psychological state assessment model; wherein, the comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset; each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks; each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment label.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assessing psychological states based on multimodal data and model fine-tuning, characterized in that, include: Feature extraction is performed on the target multimodal sensing data of the individual to be evaluated to obtain the initial modal features corresponding to different modalities; Each of the modal initial features is tokenized, and the tokenized modal initial features are concatenated to obtain the target token sequence. The target token sequence is input into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed, output by the comprehensive psychological state assessment model. The comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset. Each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks. Each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment tag.

2. The psychological state assessment method based on multimodal data and model fine-tuning according to claim 1, characterized in that, The feature extraction of the target multimodal sensing data of the individual to be evaluated yields initial modal features corresponding to different modalities, including: Based on a multimodal sensing front end, millimeter-wave radar data, RGB camera data, and event camera data corresponding to the individual to be evaluated are collected; Feature extraction was performed on the millimeter-wave radar data to obtain event sequences of heart rate variability parameters, time sequences of key morphological features of electrocardiograms, and time sequences of micromotor pattern features. Feature extraction is performed on the RGB camera data to obtain time series of facial motion unit intensity features, time series of head three-dimensional rotation angle features, and time series of body key point three-dimensional coordinate features. Feature extraction is performed on the event camera data to obtain time series of micro-expression category probability features, time series of saccade parameter features, and time series of pupil dynamic features; Based on the heart rate variability parameter event sequence, the electrocardiogram key morphological feature time sequence, the micro-motion pattern feature time sequence, the facial motion unit intensity feature time sequence, the head three-dimensional rotation angle feature time sequence, the body key point three-dimensional coordinate feature time sequence, the micro-expression category probability feature time sequence, the saccade parameter feature time sequence, and the pupil dynamic feature time sequence, the modal initial features corresponding to different modalities are obtained.

3. The psychological state assessment method based on multimodal data and model fine-tuning according to claim 1, characterized in that, The step of tokenizing each of the modal initial features and concatenating the tokenized modal initial features to obtain the target token sequence includes: Based on a preset analysis time window, the initial features of each modality are aligned with the time window to obtain a sequence of modal features with the same time window. Based on the modality type, the modality feature sequences are divided into corresponding time segments to obtain the time segment features of each modality feature sequence; The time segment features are tokenized to obtain multiple time segment token sequences corresponding to each tokenized modal initial feature; Each modality type embedding vector and time segment position embedding vector are added to each of the time segment token sequences to obtain each modality token sequence; wherein, the modality type embedding vector is used to distinguish the time segment token sequences of different modalities; the time segment position embedding vector is used to represent the time order information of each time segment token sequence within its respective time window; The global context token is concatenated with each of the modal token sequences to obtain the target token sequence; wherein the global context token is used to aggregate the encoded global information.

4. The psychological state assessment method based on multimodal data and model fine-tuning according to any one of claims 1 to 3, characterized in that, The comprehensive psychological state assessment model is trained through the following steps: Based on the psychological evaluation results of sample fragments corresponding to each modal token sequence in the sample token sequence, the psychological state evaluation label is constructed. Based on the psychological state assessment labels, each modal token sequence in the sample token sequence is labeled to obtain the multimodal psychological state clinical dataset; Based on the multimodal psychological state clinical dataset, the pre-trained Transformer model and the multi-task psychological state prediction head are progressively fine-tuned to obtain the comprehensive psychological state assessment model. The multi-task mental state prediction head includes at least an emotion classification head, an anxiety / depression severity regression head, a cognitive load classification head, and a personality trait regression head. The emotion classification head consists of a fully connected layer and a Softmax activation function, used to output the probability distribution of discrete emotion categories. The loss function of the emotion classification head is cross-entropy loss. The anxiety / depression severity regression head consists of a fully connected layer and a linear activation function, used to predict continuous severity scores of anxiety and / or depression. The loss function of the anxiety / depression severity regression head is mean squared error or Huber loss. The cognitive load classification head consists of a fully connected layer and a Softmax activation function, used to output the probability distribution of cognitive load levels. The loss function of the cognitive load classification head is cross-entropy loss. The personality trait regression head includes five parallel regression subheadings, each consisting of a fully connected layer and a linear activation function, used to output continuous scores for each dimension of the Big Five personality traits.

5. The psychological state assessment method based on multimodal data and model fine-tuning according to claim 4, characterized in that, The comprehensive psychological state assessment model is obtained by progressively fine-tuning the pre-trained Transformer model and the multi-task psychological state prediction head based on the multimodal psychological state clinical dataset, including: The encoder parameters of the pre-trained Transformer model are frozen, and the multi-task mental state prediction head is initially trained based on the multimodal mental state clinical dataset and the first learning rate to obtain the initially trained multi-task mental state prediction head. Unfreeze the top preset number of Transformer layers in the encoder of the pre-trained Transformer model whose encoder parameters are all frozen to obtain the pre-trained Transformer model after initial unfreezing. Based on the multimodal mental state clinical dataset and the second learning rate, the pre-trained Transformer model after initial thawing and the pre-trained multi-task mental state prediction head are jointly trained to obtain the pre-trained Transformer model and the pre-trained multi-task mental state prediction head after initial fine-tuning, respectively. Unfreeze all Transformer layers in the encoder of the pre-trained Transformer model after initial fine-tuning to obtain a fully unfrozen pre-trained Transformer model. Based on the multimodal mental state clinical dataset and the third learning rate, the fully unfrozen pre-trained Transformer model and the preliminarily fine-tuned multi-task mental state prediction head are jointly fine-tuned to obtain the comprehensive mental state assessment model. The first learning rate, the second learning rate, and the third learning rate decrease sequentially.

6. The psychological state assessment method based on multimodal data and model fine-tuning according to claim 5, characterized in that, The total loss of the multi-task mental state prediction head is obtained by weighted summation of the losses of each mental state prediction task head in the multi-task mental state prediction head, wherein the weight coefficient corresponding to each mental state prediction task head is obtained based on the homoscedastic uncertainty learning of the mental state prediction task head during the training process.

7. The psychological state assessment method based on multimodal data and model fine-tuning according to claim 1, characterized in that, After inputting the target token sequence into the trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed output by the comprehensive psychological state assessment model, the method further includes: Based on the psychological state assessment results, real-time monitoring data of the psychological state indicators of the individual to be assessed, a multi-dimensional psychological state assessment report, and historical change curves of the psychological state indicators are obtained. Based on the real-time monitoring data of the psychological state indicators, the multi-dimensional assessment report of the psychological state, and the historical change curves of the psychological state indicators, a comprehensive psychological state profile of the individual to be assessed is constructed.

8. A psychological state assessment system based on multimodal data and model fine-tuning, characterized in that, include: The multimodal data feature extraction module is used to extract features from the target multimodal sensing data of the individual to be evaluated, and obtain the initial modal features corresponding to different modalities. A physiological behavior tokenizer is used to tokenize each of the modal initial features and concatenate the tokenized modal initial features to obtain a target token sequence. The comprehensive psychological state assessment module is used to input the target token sequence into a trained comprehensive psychological state assessment model to obtain the psychological state assessment result of the individual to be assessed, output by the comprehensive psychological state assessment model. The comprehensive psychological state assessment model is obtained by fine-tuning a pre-trained Transformer model with a multi-task psychological state prediction head based on a multimodal psychological state clinical dataset. Each psychological state prediction task head in the multi-task psychological state prediction head is constructed according to different psychological state prediction tasks. Each sample data in the multimodal psychological state clinical dataset is labeled with a corresponding psychological state assessment label.

Images