A Method and System for Assessing the Mental State of Adolescents Based on Multimodal Sensing Signals

Abstract

This invention discloses a method and system for assessing the psychological state of adolescents based on multimodal perception signals. The method includes: acquiring facial expression signals, speech signals, facial temperature signals, and heart rate signals of the subject; preprocessing each modality signal and inputting it into a built-in feature recognition model to obtain recognition results; binarizing and encoding the facial expression and speech signals to categorize them into an overt emotion group, and the heart rate and facial temperature signals into an implicit physiological group; weighting and summing the results according to preset modality weights to obtain a group composite value and comparing it with a preset group judgment threshold to obtain intermediate group results; and using preset two-group cross-validation rules to logically combine the two intermediate results to generate a final auxiliary assessment result and rule path identifier. The method and system of this application achieve full transparency and traceability in the judgment process, effectively identify implicit state characteristics, and possess feedback-based closed-loop iterative optimization capabilities.

Description

Technical Field

[0001] This invention relates to the field of smart healthcare technology, and in particular to a method and system for assessing the psychological state of adolescents based on multimodal sensing signals. Background Technology

[0002] With increasing societal attention to the mental health of adolescents, how to conduct supplementary assessments of their mental state using objective physiological signals in non-clinical settings such as schools and homes has become a pressing technical challenge. Adolescents' physiological and psychological development is still immature, and their emotional expression and physiological response patterns differ significantly from adults. On the one hand, some adolescents tend to suppress or conceal their outward emotional expressions when experiencing fluctuations in their mental state, making assessment methods based on single overt behavioral signals difficult to capture their true state. On the other hand, developmental changes in the autonomic nervous system during puberty result in different baseline ranges, response amplitudes, and recovery speeds of physiological signals compared to adults, leading to significant biases when directly applying signal analysis parameters and judgment criteria used for adults.

[0003] Some existing multimodal signal analysis solutions attempt to simultaneously collect overt behavioral signals such as facial expressions and speech, as well as implicit physiological signals such as heart rate and skin temperature. However, these solutions still face the following technical shortcomings in practical applications: First, various signals are usually analyzed independently and then simply superimposed, without establishing a cross-validation mechanism between overt and implicit signals. When subjects intentionally or unintentionally conceal their outward expressions, the system cannot effectively identify them through implicit physiological signals. Second, the signal processing and judgment process is mostly a black box structure, lacking traceable intermediate parameters and judgment logic from the original signal to the final output. When the output results do not match the actual situation, it is difficult to locate the source of the deviation and cannot provide a basis for continuous improvement of the system. Third, there is a lack of signal collection standards, feature recognition parameters, and judgment rule systems specifically for adolescents. The system performs reasonably well when targeting adults, but its adaptability is insufficient when directly transferred to adolescent scenarios. Fourth, existing technical solutions are mostly deployed only once and lack a closed-loop mechanism for iterative optimization of internal parameters based on actual usage feedback, making it impossible to continuously improve the long-term accuracy of the system.

[0004] Therefore, there is a need in this field for a technical solution that can standardize the acquisition and processing of multi-source physiological signals, establish a cross-validation mechanism between explicit and implicit signals, make the judgment process transparent and traceable throughout, and have closed-loop iterative optimization capabilities, so as to meet the actual needs of the auxiliary assessment of the psychological state of adolescents. Summary of the Invention

[0005] To address the technical problems existing in the background art, this invention proposes a method and system for assessing the psychological state of adolescents based on multimodal sensing signals.

[0006] The proposed method for assessing the mental state of adolescents based on multimodal sensory signals includes the following steps: S1. Acquire the subject's multimodal perception signals, which include facial expression signals, speech signals, facial temperature signals, and heart rate signals; S2. Preprocess the multimodal sensing signal to obtain multimodal sensing data, which includes standardized facial input data, standardized voice input data, standardized temperature input data, and standardized heart rate sequence. S3. Input the multimodal sensing data into the built-in feature recognition model respectively to obtain the recognition results of each modality; S4. Compare the results of each modality recognition with the preset modality coding threshold to obtain the corresponding modality coding value. The modality coding value includes facial expression coding value, voice coding value, facial temperature coding value and heart rate coding value. S5. Assign facial expression coding values ​​and voice coding values ​​to the overt emotion group, and heart rate coding values ​​and facial temperature coding values ​​to the implicit physiological group. According to the preset modal weights, the modal coding values ​​in the overt emotion group and the modal coding values ​​in the implicit physiological group are weighted and summed to obtain the comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group. The comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group are compared with the preset group judgment thresholds to obtain the intermediate results of the overt emotion group and the implicit physiological group, respectively. S6. Using a preset dual-group cross-validation rule, the intermediate results of the explicit emotion group and the implicit physiological group are logically combined to generate the final auxiliary evaluation result and rule path identifier.

[0007] Preferably, the built-in feature recognition model includes a facial expression feature recognition subnetwork, a voice feature recognition subnetwork, a facial temperature feature recognition subnetwork, and a heart rate feature recognition subnetwork; step S3 specifically includes: Standardized facial input data is input into the facial expression feature recognition subnetwork to obtain facial expression recognition results; Standardized speech input data is input into the speech feature recognition subnetwork to obtain speech recognition results; Standardized temperature input data is input into the facial temperature feature recognition subnetwork to obtain facial temperature recognition results; The standardized heart rate sequence is input into the heart rate feature recognition subnetwork to obtain the heart rate recognition result.

[0008] Preferably, the facial expression feature recognition subnetwork is an expression region attention residual convolutional subnetwork, including a full-face feature branch, a key region feature branch, a region attention fusion layer, and a temporal smoothing layer; Among them, the full-face feature branch is composed of the first convolutional layer, two residual convolutional blocks and a global pooling layer connected in sequence, which is used to extract the full-face global expression features of the standardized facial input data. The key region feature branch is used to input local images of the eyebrow and eye area, nose wing area and mouth corner area in the standardized facial input data into the local convolutional block with shared parameters to extract the local micro-expression features of each key region. The regional attention fusion layer is used to calculate the correlation between each local micro-expression feature and the global facial expression feature and generate attention weights. Based on the attention weights, the local micro-expression features are weighted and concatenated to obtain the fused expression features. The temporal smoothing layer is used to perform one-dimensional convolutional smoothing on the fused facial expression features of consecutive frames, and outputs the facial expression recognition results.

[0009] Preferably, the speech feature recognition subnetwork is a multi-scale time-frequency convolutional subnetwork, including a Mel spectrum input layer, a short-term emotion branch, a prosodic variation branch, a frequency band attention layer, and a temporal attention pooling layer; Among them, the Mel spectrogram input layer is used to receive standardized speech input data and output Mel spectrograms; Short-term emotion branch is used to extract local energy change features in Mel spectrograms using 3×3 convolution kernels; The prosodic variation branch is used to extract continuous variation features of speech rate and pitch in the Mel spectrogram using one-dimensional convolution that expands along the time direction. The frequency band attention layer is used to assign different weights to the high-frequency, low-frequency, and energy-concentrated frequency bands of the Mel spectrogram, thus obtaining the frequency band weighted features; The temporal attention pooling layer is used to weight and converge the effective speech frames in the frequency band weighted features to output the speech recognition result.

[0010] Preferably, the facial temperature feature recognition subnetwork is a dynamic heatmap convolutional subnetwork, including a temperature region node construction layer, a dynamic adjacency matrix generation layer, a spatial graph convolutional layer, and a temporal convolutional layer; Among them, the temperature region node construction layer is used to construct temperature map structure data by using the temperature sequences of the forehead, nose, left cheek, right cheek and eye area in the standardized temperature input data as node features; A dynamic adjacency matrix generation layer is used to generate edge weights based on the spatial adjacency relationship of facial regions and the correlation between regional temperature changes, thus obtaining a dynamic adjacency matrix. Spatial graph convolutional layers are used to extract temperature conduction relationships between regions based on node features and dynamic adjacency matrices. The temporal convolutional layer is used to extract the temperature change trend in the temperature conduction relationship features between regions and output the facial temperature recognition result.

[0011] Preferably, the heart rate feature recognition subnetwork is a gated residual one-dimensional temporal convolutional subnetwork, including an RR interval input layer, a multi-scale dilated convolutional layer, a gated residual connection layer, a rhythm attention layer, and an output layer; The RR interval input layer is used to receive the standardized heart rate sequence. Multi-scale dilated convolutional layers are used to extract short-term and medium-term circadian rhythm fluctuation features from normalized heart rate sequences using parallel one-dimensional convolutions with dilation rates of 1, 2, and 4. The gated residual connection layer is used to gate and preserve the convolutional features according to the artifact ratio of the standardized heart rate sequence, so as to obtain the denoised rhythm features. The rhythm attention layer is used to assign weights to abnormal fluctuation segments in the denoised rhythm features to obtain weighted rhythm features. The output layer is used to output heart rate recognition results based on weighted rhythm features.

[0012] Preferably, the preset modal coding thresholds include preset facial expression modal coding thresholds, preset speech modal coding thresholds, preset facial temperature modal coding thresholds, and preset heart rate modal coding thresholds; step S4 specifically includes: When the facial expression recognition result is greater than or equal to the preset facial expression modality coding threshold, the facial expression coding value is 1; when the facial expression recognition result is less than the preset facial expression modality coding threshold, the facial expression coding value is 0. When the speech recognition result is greater than or equal to the preset speech modality coding threshold, the speech coding value is 1; when the speech recognition result is less than the preset speech modality coding threshold, the speech coding value is 0. When the facial temperature recognition result is greater than or equal to the preset facial temperature modal coding threshold, the facial temperature coding value is 1; when the facial temperature recognition result is less than the preset facial temperature modal coding threshold, the facial temperature coding value is 0. When the heart rate recognition result is greater than or equal to the preset heart rate modality coding threshold, the heart rate coding value is 1; when the heart rate recognition result is less than the preset heart rate modality coding threshold, the heart rate coding value is 0.

[0013] Preferably, the step of weighting and summing the modal coding values ​​in the explicit emotion group and the implicit physiological group according to preset modal weights to obtain the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group is as follows: S_e = ω_f×b_f + ω_v×b_v; Where S_e is the comprehensive value of the overt emotion group; b_f is the facial expression coding value; b_v is the speech coding value; ω_f is the facial expression preset modal weight; ω_v is the speech preset modal weight, and ω_f+ω_v=1; S_i = ω_h×b_h + ω_t×b_t; Where S_i is the implicit physiological group composite value; b_h is the heart rate encoded value; b_t is the facial temperature encoded value; ω_h is the heart rate preset modal weight; ω_t is the facial temperature preset modal weight, and ω_h+ω_t=1.

[0014] Preferably, the preset group determination threshold includes a preset explicit emotion group determination threshold and a preset implicit physiological group determination threshold; the step of comparing the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group with the preset group determination thresholds to obtain intermediate results of the explicit emotion group and the implicit physiological group, respectively, specifically includes: When the overall value of the overt emotion group is greater than or equal to the preset threshold for overt emotion group judgment, the intermediate result of the overt emotion group is positive; when the overall value of the overt emotion group is less than the preset threshold for overt emotion group judgment, the intermediate result of the overt emotion group is negative. When the comprehensive value of the implicit physiological group is greater than or equal to the preset implicit physiological group judgment threshold, the intermediate result of the implicit physiological group is positive; when the comprehensive value of the implicit physiological group is less than the preset implicit physiological group judgment threshold, the intermediate result of the implicit physiological group is negative.

[0015] Preferably, step S6 specifically includes: If the intermediate result of the explicit emotion group is positive and the intermediate result of the implicit physiology group is positive, then a key focus auxiliary assessment result is generated, and the rule path is marked as both groups being positive. If the intermediate result of the explicit emotion group is negative and the intermediate result of the implicit physiology group is positive, then a key focus auxiliary assessment result is generated, and the rule path is identified as explicit negative and implicit positive. If the intermediate result of the explicit emotion group is positive and the intermediate result of the implicit physiological group is negative, then when both the facial expression coding value and the voice coding value are 1, a key concern auxiliary assessment result is generated. The rule path is identified as explicit positive and implicit negative, and both facial expression and voice are positive. Otherwise, a regular concern auxiliary assessment result is generated. If the intermediate result of both the explicit emotion group and the implicit physiology group is negative, then a routine attention-based auxiliary assessment result is generated, and the rule path is marked as both groups being negative.

[0016] Preferably, step S2 specifically includes: Facial expression signals are processed by face region detection, key region localization, invalid frame removal, pose correction and scale unification to obtain standardized facial input data. The speech signal is processed by removing silence segments, suppressing environmental noise, normalizing amplitude, framing and windowing, and time-frequency conversion to obtain standardized speech input data. The effective region is located, outlier is removed, temperature field is registered, and regional temperature sequence is extracted from the facial temperature signal to obtain standardized temperature input data. The heart rate signal is subjected to baseline drift suppression, power line interference suppression, artifact removal, peak detection, and rhythm sequence processing to obtain a standardized heart rate sequence.

[0017] Preferably, it further includes: The final auxiliary assessment results, intermediate results of the explicit emotion group, intermediate results of the implicit physiological group, rule path identifiers, and external clinical review data are input into a pre-set large language model with background prompts, and the model outputs adjustment suggestions for the pre-set modality weights, pre-set group judgment thresholds, or pre-set two-group cross-validation rules.

[0018] The proposed system for assessing the mental state of adolescents based on multimodal sensory signals includes: The signal acquisition and evaluation module is used to receive the multimodal perception signals of the subject, including facial expression signals, speech signals, facial temperature signals, and heart rate signals. The multimodal perception signals are preprocessed to obtain multimodal perception data. The multimodal perception data is then input into a built-in feature recognition model to obtain the recognition results for each modality. Each modality recognition result is compared with a preset modality coding threshold to obtain the corresponding modality coding value. The modality coding values ​​include facial expression coding values, speech coding values, facial temperature coding values, and heart rate coding values. Facial expression coding values ​​and speech coding values ​​are categorized into the overt emotion group, and heart rate coding values ​​and facial temperature coding values ​​are categorized into the implicit physiological group. Based on preset modality weights, the modality coding values ​​in the overt emotion group and the implicit physiological group are weighted and summed to obtain the comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group. The comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group are compared with preset group judgment thresholds to obtain intermediate results for the overt emotion group and the implicit physiological group, and structured data is output. The integrated evaluation rule module receives structured data and uses preset dual-group cross-validation rules to logically combine the intermediate results of the explicit emotion group and the implicit physiological group to generate the final auxiliary evaluation result and rule path identifier. The feedback optimization agent module receives the final auxiliary assessment results, the intermediate results of the explicit emotion group, the intermediate results of the implicit physiological group, and the rule path identifiers. It also acquires external clinical review data and uses a pre-set large language model with background prompts to perform attribution analysis on the differences between the final auxiliary assessment results and the clinical review data. It outputs adjustment suggestions for the pre-set modality weights, pre-set group judgment thresholds, or pre-set two-group cross-validation rules.

[0019] This invention presents a method and system for assessing the psychological state of adolescents based on multimodal sensory signals. By simultaneously acquiring four modalities of signals—facial expression, voice, facial temperature, and heart rate—and structurally grouping them into explicit emotional and implicit physiological groups, a cross-validation mechanism between explicit behavioral and implicit physiological signals is constructed. This mechanism can capture implicit characteristics through physiological indicators that are not easily controlled, even when external emotional expression is masked or suppressed, thus improving the technical identification capability of concealed states. From raw signal acquisition, preprocessing, feature recognition, binarization encoding, group weighting, intra-group judgment, to bi-group cross-validation, each step generates recordable intermediate parameters and judgment criteria, achieving full transparency and traceability of the judgment process and overcoming the "black box" problem. The model's lack of interpretability is addressed by incorporating parameter settings and judgment logic specifically tailored to adolescents within the integrated evaluation rules. This includes cross-validation rules and intra-group weight constraints for situations where explicit and implicit signals are separated, enabling the evaluation process to adapt to the physiological and psychological development characteristics of adolescents and avoiding the adaptation bias caused by directly applying adult models. Furthermore, by introducing a large language model with pre-set background prompts to form a feedback optimization closed loop, the system can perform differential attribution analysis on the externally input clinical review data and the system's output auxiliary evaluation results, and output adjustment suggestions for specific parameters of weights, thresholds, or cross-validation rules. This allows the system to continuously iterate and optimize based on actual usage feedback, overcoming the technical problem of performance degradation over time in traditional one-time deployment solutions. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the workflow of the adolescent mental state assessment method based on multimodal sensing signals proposed in this invention. Figure 2 This is a schematic diagram of the system architecture of the adolescent psychological state assessment system based on multimodal sensing signals proposed in this invention. Detailed Implementation

[0021] Reference Figure 1 The present invention proposes a method for assessing the mental state of adolescents based on multimodal perception signals, comprising the following steps: S1. Acquire the subject's multimodal perception signals, which include facial expression signals, speech signals, facial temperature signals, and heart rate signals.

[0022] Specifically, two sets of dedicated acquisition devices were used to simultaneously acquire four modal signals from the subjects. The HF10 non-contact physiological and psychological intelligent detector was responsible for simultaneously acquiring facial expression signals, voice signals, and facial temperature signals from the subjects in their natural state; the aLink-R211B single-lead miniature electrocardiogram recorder was responsible for acquiring the subjects' heart rate signals. The four types of signals were linked together using a unified acquisition identifier and acquisition time index to ensure complete alignment of multimodal data for the same subject within the same acquisition round. The acquisition environment was a standard campus psychological counseling room, with subjects in a natural sitting posture, and the acquisition time was approximately three minutes.

[0023] S2. Preprocess the multimodal sensing signals to obtain multimodal sensing data, which includes standardized facial input data, standardized voice input data, standardized temperature input data, and standardized heart rate sequence.

[0024] In this embodiment, step S2 specifically includes: Facial expression signals are processed by face region detection, key region localization, invalid frame removal, pose correction and scale unification to obtain standardized facial input data. The speech signal is processed by removing silence segments, suppressing environmental noise, normalizing amplitude, framing and windowing, and time-frequency conversion to obtain standardized speech input data. The effective region is located, outlier is removed, temperature field is registered, and regional temperature sequence is extracted from the facial temperature signal to obtain standardized temperature input data. The heart rate signal is subjected to baseline drift suppression, power line interference suppression, artifact removal, peak detection, and rhythm sequence processing to obtain a standardized heart rate sequence.

[0025] Specifically, the following preprocessing operations are performed on the raw signals of each modality to obtain standardized multimodal sensing data: For facial expression signals, face region detection is performed, and key region localization algorithms are used to extract key regions such as eyebrows, eyes, nose wings, and corners of the mouth. Invalid frames such as closed eyes and large-scale shaking are removed. After pose correction and scale unification, standardized facial input data is obtained. Preferably, brightness normalization and pixel value standardization are further performed to reduce the impact of ambient light changes on subsequent recognition.

[0026] For the speech signal, silence segment removal is performed to eliminate blank segments at the beginning and end of the acquisition. Environmental noise is suppressed by Wiener filtering, amplitude normalization is performed to eliminate volume differences, Hamming window is used for frame division and windowing, and the signal is converted into a Mel spectrogram by short-time Fourier transform to obtain standardized speech input data.

[0027] For facial temperature signals, effective region localization is performed to eliminate invalid regions such as those obscured by hair and clothing. Abnormal temperature points are eliminated using the three-standard-deviation method. Affine transformation is used to complete temperature field registration. Regional temperature sequences are extracted according to areas such as the forehead, nose, and cheeks to obtain standardized temperature input data.

[0028] For the heart rate signal, 50Hz power frequency interference suppression and 0.5Hz high-pass filtering were performed to eliminate baseline drift. Independent component analysis was used to remove body motion artifacts. The Pan-Tompkins algorithm was used to detect the R peak, extract the RR interval sequence, and perform linear interpolation correction on abnormal heartbeats to obtain a standardized heart rate sequence.

[0029] S3. Input the multimodal sensing data into the built-in feature recognition model to obtain the recognition results of each modality.

[0030] In this embodiment, the built-in feature recognition model includes a facial expression feature recognition subnetwork, a voice feature recognition subnetwork, a facial temperature feature recognition subnetwork, and a heart rate feature recognition subnetwork; step S3 specifically includes: Standardized facial input data is input into the facial expression feature recognition subnetwork to obtain facial expression recognition results; Standardized speech input data is input into the speech feature recognition subnetwork to obtain speech recognition results; Standardized temperature input data is input into the facial temperature feature recognition subnetwork to obtain facial temperature recognition results; The standardized heart rate sequence is input into the heart rate feature recognition subnetwork to obtain the heart rate recognition result.

[0031] Specifically, the facial expression feature recognition subnetwork is an expression region attention residual convolutional subnetwork, which includes a full-face feature branch, a key region feature branch, a region attention fusion layer, and a temporal smoothing layer; Among them, the full-face feature branch is composed of the first convolutional layer, two residual convolutional blocks and a global pooling layer connected in sequence, which is used to extract the full-face global expression features of the standardized facial input data. The key region feature branch is used to input local images of the eyebrow and eye area, nose wing area and mouth corner area in the standardized facial input data into the local convolutional block with shared parameters to extract the local micro-expression features of each key region. The regional attention fusion layer is used to calculate the correlation between each local micro-expression feature and the global facial expression feature and generate attention weights. Based on the attention weights, the local micro-expression features are weighted and concatenated to obtain the fused expression features. The temporal smoothing layer is used to perform one-dimensional convolutional smoothing on the fused facial expression features of consecutive frames, and outputs the facial expression recognition results.

[0032] Specifically, the speech feature recognition subnetwork is a multi-scale time-frequency convolutional subnetwork, including a Mel spectrum input layer, a short-term emotion branch, a prosodic variation branch, a frequency band attention layer, and a temporal attention pooling layer; Among them, the Mel spectrogram input layer is used to receive standardized speech input data and output Mel spectrograms; Short-term emotion branch is used to extract local energy change features in Mel spectrograms using 3×3 convolution kernels; The prosodic variation branch is used to extract continuous variation features of speech rate and pitch in the Mel spectrogram using one-dimensional convolution that expands along the time direction. The frequency band attention layer is used to assign different weights to the high-frequency, low-frequency, and energy-concentrated frequency bands of the Mel spectrogram, thus obtaining the frequency band weighted features; The temporal attention pooling layer is used to weight and converge the effective speech frames in the frequency band weighted features to output the speech recognition result.

[0033] Specifically, the facial temperature feature recognition subnetwork is a dynamic heatmap graph convolutional subnetwork, which includes a temperature region node construction layer, a dynamic adjacency matrix generation layer, a spatial graph convolutional layer, and a temporal convolutional layer; Among them, the temperature region node construction layer is used to construct temperature map structure data by using the temperature sequences of the forehead, nose, left cheek, right cheek and eye area in the standardized temperature input data as node features; A dynamic adjacency matrix generation layer is used to generate edge weights based on the spatial adjacency relationship of facial regions and the correlation between regional temperature changes, thus obtaining a dynamic adjacency matrix. Spatial graph convolutional layers are used to extract temperature conduction relationships between regions based on node features and dynamic adjacency matrices. The temporal convolutional layer is used to extract the temperature change trend in the temperature conduction relationship features between regions and output the facial temperature recognition result.

[0034] Specifically, the heart rate feature recognition subnetwork is a gated residual one-dimensional temporal convolutional subnetwork, including an RR interval input layer, a multi-scale dilated convolutional layer, a gated residual connection layer, a rhythm attention layer, and an output layer; The RR interval input layer is used to receive standardized heart rate sequences. Multi-scale dilated convolutional layers are used to extract short-term and medium-term circadian rhythm fluctuation features from normalized heart rate sequences using parallel one-dimensional convolutions with dilation rates of 1, 2, and 4. The gated residual connection layer is used to gate and preserve the convolutional features according to the artifact ratio of the standardized heart rate sequence, so as to obtain the denoised rhythm features. The rhythm attention layer is used to assign weights to abnormal fluctuation segments in the denoised rhythm features to obtain weighted rhythm features. The output layer is used to output heart rate recognition results based on weighted rhythm features.

[0035] In this embodiment, the built-in feature recognition model is trained using a supervised learning approach, with the four sub-networks trained independently but sharing a unified loss function framework. First, multimodal training samples with external clinical review labels are acquired. These labels are obtained by psychiatrists who annotate each subject according to the clinical diagnostic gold standard. All samples are then divided into training, validation, and test sets based on subject identification, ensuring that samples from the same subject do not simultaneously appear in different sets to evaluate the model's generalization ability to subjects not previously seen.

[0036] During the training phase, the facial expression feature recognition subnetwork (expression region attention residual convolution subnetwork) was trained with standardized facial input data, the speech feature recognition subnetwork (multi-scale time-frequency convolution subnetwork) was trained with standardized speech input data, the facial temperature feature recognition subnetwork (dynamic heatmap convolution subnetwork) was trained with standardized temperature input data, and the heart rate feature recognition subnetwork (gated residual one-dimensional time convolution subnetwork) was trained with standardized heart rate sequences. The forward propagation paths of each subnetwork are consistent with the hierarchical structure described above. The facial expression subnetwork extracts features through the full-face feature branch and the key region feature branch, then is weighted and concatenated by the region attention fusion layer and output through the temporal smoothing layer. The speech subnetwork extracts features through the short-term emotion branch and the prosodic variation branch, then is weighted by the frequency band attention layer and output through the temporal attention pooling layer. The facial temperature feature recognition subnetwork constructs a graph structure through the temperature region node construction layer and the dynamic adjacency matrix generation layer, then outputs through the spatial graph convolutional layer and the temporal convolutional layer. The heart rate subnetwork extracts features through the multi-scale dilated convolutional layer, then is denoised by the gated residual connection layer and output through the rhythm attention layer.

[0037] The training objective employs a weighted sum of three types of losses: Modality classification loss, calculated as the binary cross-entropy loss between the output recognition results of each sub-network and the external clinical review labels, constrains the independent recognition ability of each modality; Group consistency loss calculates the cosine similarity between facial expression recognition results and speech recognition results within the explicit emotion group, and the cosine similarity between heart rate recognition results and facial temperature recognition results within the implicit physiological group, constraining the feature consistency of modalities within the same group; Confidence calibration loss is the expected calibration error between the output confidence of each sub-network and the true accuracy in the final auxiliary evaluation result, constraining the reliability of the confidence. The weights of the three types of losses are adjusted based on the group matching effect on the validation set.

[0038] After training, the encoding threshold and group decision threshold for each modality are determined by scanning on the validation set. The selection criterion is the highest consistency rate between the two-group cross-validation results on the validation set and the external clinical review labels. The four trained sub-networks and their parameters are then embedded into a built-in feature recognition model. During deployment, each modality outputs its recognition result along with its modality confidence score, which is used by the subsequent binarization encoding and fusion evaluation rule module.

[0039] S4. Compare the results of each modality recognition with the preset modality coding threshold to obtain the corresponding modality coding value. The modality coding value includes facial expression coding value, speech coding value, facial temperature coding value and heart rate coding value.

[0040] In this embodiment, the preset modal coding thresholds include preset facial expression modal coding thresholds, preset speech modal coding thresholds, preset facial temperature modal coding thresholds, and preset heart rate modal coding thresholds; step S4 specifically includes: When the facial expression recognition result is greater than or equal to the preset facial expression modality coding threshold, the facial expression coding value is 1; when the facial expression recognition result is less than the preset facial expression modality coding threshold, the facial expression coding value is 0. When the speech recognition result is greater than or equal to the preset speech modality coding threshold, the speech coding value is 1; when the speech recognition result is less than the preset speech modality coding threshold, the speech coding value is 0. When the facial temperature recognition result is greater than or equal to the preset facial temperature modal coding threshold, the facial temperature coding value is 1; when the facial temperature recognition result is less than the preset facial temperature modal coding threshold, the facial temperature coding value is 0. When the heart rate recognition result is greater than or equal to the preset heart rate modality coding threshold, the heart rate coding value is 1; when the heart rate recognition result is less than the preset heart rate modality coding threshold, the heart rate coding value is 0.

[0041] S5. Assign facial expression coding values ​​and voice coding values ​​to the explicit emotion group, and heart rate coding values ​​and facial temperature coding values ​​to the implicit physiological group. Based on the preset modal weights, sum the modal coding values ​​in the explicit emotion group and the modal coding values ​​in the implicit physiological group to obtain the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group. Compare the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group with the preset group judgment thresholds to obtain the intermediate results of the explicit emotion group and the implicit physiological group, respectively.

[0042] In this embodiment, the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group are obtained by weighting and summing the modal coding values ​​in the explicit emotion group and the implicit physiological group according to the preset modal weights, respectively. Specifically: S_e = ω_f×b_f + ω_v×b_v; Where S_e is the comprehensive value of the overt emotion group; b_f is the facial expression coding value; b_v is the speech coding value; ω_f is the facial expression preset modal weight; ω_v is the speech preset modal weight, and ω_f+ω_v=1; S_i = ω_h×b_h + ω_t×b_t; Where S_i is the implicit physiological group composite value; b_h is the heart rate encoded value; b_t is the facial temperature encoded value; ω_h is the heart rate preset modal weight; ω_t is the facial temperature preset modal weight, and ω_h+ω_t=1.

[0043] In this embodiment, the preset group judgment threshold includes a preset explicit emotion group judgment threshold and a preset implicit physiological group judgment threshold; the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group are compared with the preset group judgment thresholds to obtain the intermediate results of the explicit emotion group and the implicit physiological group, respectively, specifically including: When the overall value of the overt emotion group is greater than or equal to the preset threshold for overt emotion group judgment, the intermediate result of the overt emotion group is positive; when the overall value of the overt emotion group is less than the preset threshold for overt emotion group judgment, the intermediate result of the overt emotion group is negative. When the comprehensive value of the implicit physiological group is greater than or equal to the preset implicit physiological group judgment threshold, the intermediate result of the implicit physiological group is positive; when the comprehensive value of the implicit physiological group is less than the preset implicit physiological group judgment threshold, the intermediate result of the implicit physiological group is negative.

[0044] S6. Using a preset dual-group cross-validation rule, the intermediate results of the explicit emotion group and the implicit physiological group are logically combined to generate the final auxiliary evaluation result and rule path identifier.

[0045] In this embodiment, step S6 specifically includes: If the intermediate result of the explicit emotion group is positive and the intermediate result of the implicit physiology group is positive, then a key focus auxiliary assessment result is generated, and the rule path is marked as both groups being positive. If the intermediate result of the explicit emotion group is negative and the intermediate result of the implicit physiology group is positive, then a key focus auxiliary assessment result is generated, and the rule path is identified as explicit negative and implicit positive. If the intermediate result of the explicit emotion group is positive and the intermediate result of the implicit physiological group is negative, then when both the facial expression coding value and the voice coding value are 1, a key concern auxiliary assessment result is generated. The rule path is identified as explicit positive and implicit negative, and both facial expression and voice are positive. Otherwise, a regular concern auxiliary assessment result is generated. If the intermediate result of both the explicit emotion group and the implicit physiology group is negative, then a routine attention-based auxiliary assessment result is generated, and the rule path is marked as both groups being negative.

[0046] Specifically, in this embodiment, the preset facial expression modality coding threshold is 0.5, the preset voice modality coding threshold is 0.5, the preset facial temperature modality coding threshold is 0.5, and the preset heart rate modality coding threshold is 0.5.

[0047] Suppose that the modal recognition results of a certain subject are: facial expression recognition result = 0.72, speech recognition result = 0.38, facial temperature recognition result = 0.61, and heart rate recognition result = 0.85. The four modal coding values ​​are: facial expression coding value = 1, speech coding value = 0, facial temperature coding value = 1, and heart rate coding value = 1.

[0048] Specifically, in this embodiment, ω_f is 0.5, ω_v is 0.5, ω_h is 0.6, ω_t is 0.4, the preset threshold for the explicit emotion group is 0.5, the preset threshold for the implicit physiological group is 0.5, and the calculated group comprehensive value is: S_e=0.5, Si=1; the intermediate result of the explicit emotion group is positive, and the intermediate result of the implicit physiological group is positive.

[0049] Based on the results of this embodiment, the intermediate results of both the overt emotion group and the implicit physiological group were positive. According to the first rule of the preset two-group cross-validation, a key-focus auxiliary assessment result was generated, with the corresponding rule path identifier being "both groups are positive". This result indicates that both the subjects' overt emotional expression and implicit physiological response showed abnormal characteristics, with the two dimensions mutually corroborating each other, suggesting further clinical psychological interviews.

[0050] Taking another subject as an example, the triggering conditions of other rule paths are explained: If the intermediate result of the overt emotion group is negative and the intermediate result of the implicit physiological group is positive, a key focus auxiliary assessment result is generated, and the rule path is marked as "overt negative and implicit positive", indicating that the subject has a tendency to conceal emotions; if the intermediate result of the overt emotion group is positive and the intermediate result of the implicit physiological group is negative, and the facial expression coding value and the voice coding value are both 1, then a key focus auxiliary assessment result is generated, and the rule path is marked as "overt positive and implicit negative and both facial expression and voice are positive"; if facial expression and voice are not both 1, then a regular focus auxiliary assessment result is generated; if the intermediate results of both groups are negative, a regular focus auxiliary assessment result is generated, and the rule path is marked as "both groups are negative".

[0051] In this embodiment, it also includes: S7. Input the final auxiliary assessment results, the intermediate results of the explicit emotion group, the intermediate results of the implicit physiological group, the rule path identifiers, and the external clinical review data into the pre-set large language model with background prompts, and output adjustment suggestions for the pre-set modality weights, pre-set group judgment thresholds, or pre-set two-group cross-validation rules.

[0052] In this embodiment, the background cues configured in the large language model include: a description of the characteristics of adolescents' emotional expression masking, the correspondence between different modal signals and depression-related physiological and psychological mechanisms, and a complete definition of the fusion judgment rules. In this step, the large language model with preset background cues can be a general-purpose large language model or a dedicated large language model fine-tuned based on psychological assessment scenario data. Examples include GPT series models, Wenxin Yiyan / ERNIE Bot, Tongyi Qianwen / Qwen, Xunfei Xinghuo, ChatGLM, LLaMA, etc., or a locally deployed model with equivalent natural language understanding, reasoning, and generation capabilities. The large language model first compares the auxiliary assessment results with the clinical review results to determine the type of difference; then, it combines the four-modal coding values, rule path identifiers, and statistical information of samples with the same type of difference in historical batches to perform core attribution analysis of the difference; finally, it outputs qualitative and quantitative adjustment suggestions for preset modality weights, preset group judgment thresholds, or preset two-group cross-validation rules.

[0053] For example, after accumulating multiple batches of data, if the large language model finds that the initial screening results under the "explicitly negative and implicitly positive" rule path have a match rate of less than 80% with the clinical review results, and the core bias is manifested as overjudgment driven by a positive heart rate monomodality, then the large language model may suggest: moderately increasing the heart rate encoding threshold, or reducing the heart rate weight of the implicit physiological group to below 0.55, in order to optimize the judgment specificity. System maintainers can then iteratively update the corresponding parameters in the fusion judgment rules after evaluating these suggestions based on actual clinical practice.

[0054] Throughout the entire assessment process described above, from raw signal acquisition, preprocessing, feature recognition, binarization encoding, group weighting, within-group determination, two-group cross-validation, result output, to the generation of parameter adjustment suggestions, each step generates recordable intermediate data, forming a complete decision-making traceability chain. The subjects ultimately receive an auxiliary assessment conclusion and its corresponding rule basis, rather than uninterpreted binary labels, thus providing a transparent, traceable, and objective reference for subsequent clinical decisions.

[0055] Reference Figure 2 The present invention proposes a multimodal sensory signal-based adolescent psychological state assessment system, comprising: The signal acquisition and evaluation module is used to receive the multimodal perception signals of the subjects, including facial expression signals, speech signals, facial temperature signals, and heart rate signals. The multimodal perception signals are preprocessed to obtain multimodal perception data. The multimodal perception data is input into the built-in feature recognition model to obtain the recognition results of each modality. The recognition results of each modality are compared with the preset modality coding thresholds to obtain the corresponding modality coding values. The modality coding values ​​include facial expression coding values, speech coding values, facial temperature coding values, and heart rate coding values. The facial expression coding values ​​and speech coding values ​​are classified into the explicit emotion group, and the heart rate coding values ​​and facial temperature coding values ​​are classified into the implicit physiological group. The modality coding values ​​in the explicit emotion group and the implicit physiological group are weighted and summed according to the preset modality weights to obtain the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group. The comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group are compared with the preset group judgment thresholds to obtain the intermediate results of the explicit emotion group and the implicit physiological group, and the structured data is output. The integrated evaluation rule module receives structured data and uses preset dual-group cross-validation rules to logically combine the intermediate results of the explicit emotion group and the implicit physiological group to generate the final auxiliary evaluation result and rule path identifier. The feedback optimization agent module receives the final auxiliary assessment results, the intermediate results of the explicit emotion group, the intermediate results of the implicit physiological group, and the rule path identifiers. It also acquires external clinical review data and uses a pre-set large language model with background prompts to perform attribution analysis on the differences between the final auxiliary assessment results and the clinical review data. It outputs adjustment suggestions for the pre-set modality weights, pre-set group judgment thresholds, or pre-set two-group cross-validation rules.

[0056] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for assessing the mental state of adolescents based on multimodal sensory signals, characterized in that, Includes the following steps: S1. Acquire the subject's multimodal perception signals, which include facial expression signals, speech signals, facial temperature signals, and heart rate signals; S2. Preprocess the multimodal sensing signal to obtain multimodal sensing data, which includes standardized facial input data, standardized voice input data, standardized temperature input data, and standardized heart rate sequence. S3. Input the multimodal sensing data into the built-in feature recognition model respectively to obtain the recognition results of each modality; S4. Compare the results of each modality recognition with the preset modality coding threshold to obtain the corresponding modality coding value. The modality coding value includes facial expression coding value, voice coding value, facial temperature coding value and heart rate coding value. S5. Assign facial expression coding values ​​and voice coding values ​​to the overt emotion group, and heart rate coding values ​​and facial temperature coding values ​​to the implicit physiological group. According to the preset modal weights, the modal coding values ​​in the overt emotion group and the modal coding values ​​in the implicit physiological group are weighted and summed to obtain the comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group. The comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group are compared with the preset group judgment thresholds to obtain the intermediate results of the overt emotion group and the implicit physiological group, respectively. S6. Using a preset dual-group cross-validation rule, the intermediate results of the explicit emotion group and the implicit physiological group are logically combined to generate the final auxiliary evaluation result and rule path identifier.

2. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 1, characterized in that, The built-in feature recognition model includes a facial expression feature recognition subnetwork, a voice feature recognition subnetwork, a facial temperature feature recognition subnetwork, and a heart rate feature recognition subnetwork. Step S3 specifically includes: Standardized facial input data is input into the facial expression feature recognition subnetwork to obtain facial expression recognition results; Standardized speech input data is input into the speech feature recognition subnetwork to obtain speech recognition results; Standardized temperature input data is input into the facial temperature feature recognition subnetwork to obtain facial temperature recognition results; The standardized heart rate sequence is input into the heart rate feature recognition subnetwork to obtain the heart rate recognition result.

3. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 2, characterized in that, The facial expression feature recognition subnetwork is an expression region attention residual convolutional subnetwork, including a full-face feature branch, a key region feature branch, a region attention fusion layer, and a temporal smoothing layer; Among them, the full-face feature branch is composed of the first convolutional layer, two residual convolutional blocks and a global pooling layer connected in sequence, which is used to extract the full-face global expression features of the standardized facial input data. The key region feature branch is used to input local images of the eyebrow and eye area, nose wing area and mouth corner area in the standardized facial input data into the local convolutional block with shared parameters to extract the local micro-expression features of each key region. The regional attention fusion layer is used to calculate the correlation between each local micro-expression feature and the global facial expression feature and generate attention weights. Based on the attention weights, the local micro-expression features are weighted and concatenated to obtain the fused expression features. The temporal smoothing layer is used to perform one-dimensional convolutional smoothing on the fused facial expression features of consecutive frames, and outputs the facial expression recognition results.

4. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 2, characterized in that, The speech feature recognition subnetwork is a multi-scale time-frequency convolutional subnetwork, including a Mel spectrum input layer, a short-term emotion branch, a prosodic variation branch, a frequency band attention layer, and a temporal attention pooling layer; Among them, the Mel spectrogram input layer is used to receive standardized speech input data and output Mel spectrograms; Short-term emotion branch is used to extract local energy change features in Mel spectrograms using 3×3 convolution kernels; The prosodic variation branch is used to extract continuous variation features of speech rate and pitch in the Mel spectrogram using one-dimensional convolution that expands along the time direction. The frequency band attention layer is used to assign different weights to the high-frequency, low-frequency, and energy-concentrated frequency bands of the Mel spectrogram, thus obtaining the frequency band weighted features; The temporal attention pooling layer is used to weight and converge the effective speech frames in the frequency band weighted features to output the speech recognition result.

5. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 2, characterized in that, The facial temperature feature recognition subnetwork is a dynamic heatmap convolutional subnetwork, including a temperature region node construction layer, a dynamic adjacency matrix generation layer, a spatial graph convolutional layer, and a temporal convolutional layer; Among them, the temperature region node construction layer is used to construct temperature map structure data by using the temperature sequences of the forehead, nose, left cheek, right cheek and eye area in the standardized temperature input data as node features; A dynamic adjacency matrix generation layer is used to generate edge weights based on the spatial adjacency relationship of facial regions and the correlation between regional temperature changes, thus obtaining a dynamic adjacency matrix. Spatial graph convolutional layers are used to extract temperature conduction relationships between regions based on node features and dynamic adjacency matrices. The temporal convolutional layer is used to extract the temperature change trend in the temperature conduction relationship features between regions and output the facial temperature recognition result.

6. The method for assessing the psychological state of adolescents based on multimodal sensing signals according to claim 2, characterized in that, The heart rate feature recognition subnetwork is a gated residual one-dimensional temporal convolutional subnetwork, including an RR interval input layer, a multi-scale dilated convolutional layer, a gated residual connection layer, a rhythm attention layer, and an output layer. The RR interval input layer is used to receive the standardized heart rate sequence. Multi-scale dilated convolutional layers are used to extract short-term and medium-term circadian rhythm fluctuation features from normalized heart rate sequences using parallel one-dimensional convolutions with dilation rates of 1, 2, and 4. The gated residual connection layer is used to gate and preserve the convolutional features according to the artifact ratio of the standardized heart rate sequence, so as to obtain the denoised rhythm features. The rhythm attention layer is used to assign weights to abnormal fluctuation segments in the denoised rhythm features to obtain weighted rhythm features. The output layer is used to output heart rate recognition results based on weighted rhythm features.

7. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 2, characterized in that, The preset modal coding thresholds include preset facial expression modal coding thresholds, preset voice modal coding thresholds, preset facial temperature modal coding thresholds, and preset heart rate modal coding thresholds; Step S4 specifically includes: When the facial expression recognition result is greater than or equal to the preset facial expression modality coding threshold, the facial expression coding value is 1; when the facial expression recognition result is less than the preset facial expression modality coding threshold, the facial expression coding value is 0. When the speech recognition result is greater than or equal to the preset speech modality coding threshold, the speech coding value is 1; when the speech recognition result is less than the preset speech modality coding threshold, the speech coding value is 0. When the facial temperature recognition result is greater than or equal to the preset facial temperature modal coding threshold, the facial temperature coding value is 1; when the facial temperature recognition result is less than the preset facial temperature modal coding threshold, the facial temperature coding value is 0. When the heart rate recognition result is greater than or equal to the preset heart rate modality coding threshold, the heart rate coding value is 1; when the heart rate recognition result is less than the preset heart rate modality coding threshold, the heart rate coding value is 0.

8. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 1, characterized in that, The method involves weighting and summing the modal coding values ​​in the explicit emotion group and the implicit physiological group according to preset modal weights to obtain the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group, respectively. S_e = ω_f×b_f + ω_v×b_v; Where S_e is the comprehensive value of the overt emotion group; b_f is the facial expression encoding value; b_v is the speech encoding value; ω_f is the facial expression preset modal weight; ω_v is the speech preset modal weight, and ω_f+ω_v=1; S_i = ω_h×b_h + ω_t×b_t; Where S_i is the implicit physiological group composite value; b_h is the heart rate encoded value; b_t is the facial temperature encoded value; ω_h is the heart rate preset modal weight; ω_t is the facial temperature preset modal weight, and ω_h+ω_t=1.

9. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 1, characterized in that, The preset group judgment thresholds include preset explicit emotion group judgment thresholds and preset implicit physiological group judgment thresholds. The step of comparing the comprehensive value of the explicit emotion group and the comprehensive value of the implicit physiological group with preset group judgment thresholds to obtain intermediate results for the explicit emotion group and the implicit physiological group, respectively, specifically includes: When the overall value of the overt emotion group is greater than or equal to the preset threshold for overt emotion group judgment, the intermediate result of the overt emotion group is positive; when the overall value of the overt emotion group is less than the preset threshold for overt emotion group judgment, the intermediate result of the overt emotion group is negative. When the comprehensive value of the implicit physiological group is greater than or equal to the preset implicit physiological group judgment threshold, the intermediate result of the implicit physiological group is positive; when the comprehensive value of the implicit physiological group is less than the preset implicit physiological group judgment threshold, the intermediate result of the implicit physiological group is negative.

10. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 9, characterized in that, Step S6 specifically includes: If the intermediate result of the explicit emotion group is positive and the intermediate result of the implicit physiology group is positive, then a key focus auxiliary assessment result is generated, and the rule path is marked as both groups being positive. If the intermediate result of the explicit emotion group is negative and the intermediate result of the implicit physiology group is positive, then a key focus auxiliary assessment result is generated, and the rule path is identified as explicit negative and implicit positive. If the intermediate result of the explicit emotion group is positive and the intermediate result of the implicit physiological group is negative, then when both the facial expression coding value and the voice coding value are 1, a key concern auxiliary assessment result is generated. The rule path is identified as explicit positive and implicit negative, and both facial expression and voice are positive. Otherwise, a regular concern auxiliary assessment result is generated. If the intermediate result of the explicit emotion group is negative and the intermediate result of the implicit physiology group is negative, then a routine attention-based auxiliary assessment result is generated, and the rule path is marked as both groups being negative.

11. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 1, characterized in that, Step S2 specifically includes: Facial expression signals are processed by face region detection, key region localization, invalid frame removal, pose correction and scale unification to obtain standardized facial input data. The speech signal is processed by removing silence segments, suppressing environmental noise, normalizing amplitude, framing and windowing, and time-frequency conversion to obtain standardized speech input data. The effective region is located, outlier is removed, temperature field is registered, and regional temperature sequence is extracted from the facial temperature signal to obtain standardized temperature input data. The heart rate signal is subjected to baseline drift suppression, power line interference suppression, artifact removal, peak detection, and rhythm sequence processing to obtain a standardized heart rate sequence.

12. The method for assessing the psychological state of adolescents based on multimodal perception signals according to claim 1, characterized in that, Also includes: The final auxiliary assessment results, intermediate results of the explicit emotion group, intermediate results of the implicit physiological group, rule path identifiers, and external clinical review data are input into a pre-set large language model with background prompts, and the model outputs adjustment suggestions for the pre-set modality weights, pre-set group judgment thresholds, or pre-set two-group cross-validation rules.

13. A system for assessing the psychological state of adolescents based on multimodal sensing signals, characterized in that: include: The signal acquisition and evaluation module is used to receive the multimodal perception signals of the subject, including facial expression signals, speech signals, facial temperature signals, and heart rate signals. The multimodal perception signals are preprocessed to obtain multimodal perception data. The multimodal perception data is then input into a built-in feature recognition model to obtain the recognition results for each modality. Each modality recognition result is compared with a preset modality coding threshold to obtain the corresponding modality coding value. The modality coding values ​​include facial expression coding values, speech coding values, facial temperature coding values, and heart rate coding values. Facial expression coding values ​​and speech coding values ​​are categorized into the overt emotion group, and heart rate coding values ​​and facial temperature coding values ​​are categorized into the implicit physiological group. Based on preset modality weights, the modality coding values ​​in the overt emotion group and the implicit physiological group are weighted and summed to obtain the comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group. The comprehensive value of the overt emotion group and the comprehensive value of the implicit physiological group are compared with preset group judgment thresholds to obtain intermediate results for the overt emotion group and the implicit physiological group, and structured data is output. The integrated evaluation rule module receives structured data and uses preset dual-group cross-validation rules to logically combine the intermediate results of the explicit emotion group and the implicit physiological group to generate the final auxiliary evaluation result and rule path identifier. The feedback optimization agent module receives the final auxiliary assessment results, the intermediate results of the explicit emotion group, the intermediate results of the implicit physiological group, and the rule path identifiers. It also acquires external clinical review data and uses a pre-set large language model with background prompts to perform attribution analysis on the differences between the final auxiliary assessment results and the clinical review data. It outputs adjustment suggestions for the pre-set modality weights, pre-set group judgment thresholds, or pre-set two-group cross-validation rules.

Images