Intelligent dynamic assessment method for psychological states of teenagers based on multi-modal data

Abstract

The invention relates to the technical field of intelligent assessment of psychological states, and discloses an intelligent dynamic assessment method for psychological states of teenagers based on multi-modal data. The method comprises the steps that multi-modal psychological data of a target assessment object is acquired, the multi-modal psychological data comprises voice emotion information and expression behavior relations, and dynamic features are extracted from the multi-modal psychological data; performing feature fusion processing on the psychological data by adopting a trained machine learning model to generate a multi-modal feature set containing a mapping relation between modal type identifiers and time sequence codes; based on the multi-modal feature set, performing time sequence correlation analysis on the dynamic features through a state analysis network to obtain a fusion result containing emotion dimension features and cognitive mode features; and inputting a fusion result into an evaluation model for dynamic state deduction processing, and outputting psychological state evaluation elements. According to the invention, comprehensive and dynamic assessment of the psychological state can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent psychological state assessment, and particularly to a method for intelligent dynamic assessment of psychological state of adolescents based on multi-modal data. BACKGROUND

[0002] Psychological state assessment has urgent application needs in many fields of modern society, such as clinical psychological treatment, campus psychological health management, and psychological monitoring of special occupational groups. For a long time, traditional psychological state assessment methods mainly rely on subjective observation of professional personnel and various psychological scales filled in by the assessment object. This approach relies too much on the personal experience and subjective judgment of the assessor, and different assessors may have significant differences in judging the same assessment object, resulting in poor stability of the assessment results. At the same time, when filling in the scale, the assessment object may conceal the true situation due to social desirability effect or self-cognitive bias, further affecting the reliability of the assessment results.

[0003] With the development of digital technology, psychological state assessment methods based on single-modal data have emerged, such as inferring psychological state only by analyzing voice data or facial expression data. However, psychological state is a complex system formed by the interaction of emotion, cognition, and behavior, and single-modal data can only reflect one side of the psychological state, and cannot outline its complete picture. For example, voice data can reflect the emotional excitement of the assessment object, but it is difficult to reflect the cognitive pattern in the depths of the heart; facial expression data can show immediate emotional response, but it cannot capture the underlying psychological activity implied in language.

[0004] In addition, most existing assessment methods are static, i.e., one-time assessment based on data collected at a certain time, ignoring the dynamic nature of psychological state. The psychological state of a person will change constantly with time, environmental changes, and experienced events, and static assessment can only get fragmented information at a certain time, and cannot reflect its change trajectory and development trend. In terms of multi-modal data processing, existing technologies often simply stack features from different sources without establishing an internal relationship between modalities, resulting in the inability to fully utilize the collaborative information contained in multi-modal data, making it difficult to achieve the desired accuracy of the assessment results.

[0005] In actual application scenarios, these problems make it difficult for existing assessment methods to meet the needs of comprehensive, real-time, and accurate assessment of psychological state. For example, in the daily monitoring of patients with depression, it is necessary to capture the subtle changes in their psychological state in a timely manner in order to adjust the treatment plan, but existing assessment methods often cannot do this; in the psychological state monitoring of high-altitude workers, it is necessary to grasp their emotional and cognitive state in real time, and the lag and one-sidedness of existing methods may pose safety hazards. SUMMARY

[0006] The present application aims to provide a method for intelligent dynamic assessment of the psychological state of adolescents based on multi-modal data to solve the problems raised in the background art.

[0007] To achieve the above-mentioned purpose, the present application provides a method for intelligent dynamic assessment of the psychological state of adolescents based on multi-modal data, which comprises:

[0008] S1, acquiring multi-modal psychological data of a target evaluation object and extracting dynamic features in the multi-modal psychological data; wherein the multi-modal psychological data contains voice emotional information and expression behavior relationship;

[0009] S2, using a trained machine learning model to perform feature fusion processing on the multi-modal psychological data to generate a multi-modal feature set; wherein the multi-modal feature set contains a mapping relationship between modal type identification and time sequence coding;

[0010] S3, based on the multi-modal feature set, using a state analysis network to perform time sequence correlation analysis on the dynamic features to generate a feature fusion result containing emotional dimension features and cognitive mode features;

[0011] S4, inputting the feature fusion result into an evaluation model for dynamic state deduction processing to output a psychological state evaluation element; wherein the psychological state evaluation element contains a state label and a trend prediction parameter generated based on a feature evolution sequence.

[0012] Optionally, the S2 using a trained machine learning model to perform feature fusion processing on the multi-modal psychological data to generate a multi-modal feature set comprises:

[0013] S21, performing modal domain processing on the multi-modal psychological data to generate a plurality of data domains and corresponding initial feature parameters;

[0014] S22, extracting a time sequence feature sequence of each data domain and performing attention weighting processing on the time sequence feature sequence to generate a weighted multi-domain feature vector;

[0015] S23, performing similarity calculation on the multi-domain feature vector and a preset modal correlation threshold to screen out candidate features that meet the confidence condition;

[0016] S24, performing redundancy elimination processing on the candidate features to generate a multi-modal feature set containing modal type identification, and each multi-modal feature contains a normalized time sequence coding relative to the multi-modal psychological data;

[0017] Wherein, based on the multi-modal feature set, a state analysis network is used to perform time sequence correlation analysis on the dynamic features, generating a feature fusion result containing emotional dimension features and cognitive mode features, including:

[0018] According to the normalized time sequence encoding in the multi-modal feature set, time window interception processing is performed on the multi-modal psychological data, generating a plurality of feature time segments;

[0019] Standardization and feature enhancement processing are performed on each feature time segment, generating optimized state analysis input data;

[0020] The sequence feature extraction is performed on the state analysis input data by using the recurrent memory subnetwork in the state analysis network, generating an emotional dimension feature vector;

[0021] The feature mode matching is performed on the feature time segment by using the mode classification model in parallel, generating a cognitive mode probability distribution;

[0022] The emotional dimension feature vector and the cognitive mode probability distribution are spliced to generate a multi-modal feature fusion result.

[0023] Optionally, the S4 inputs the feature fusion result into an evaluation model to perform dynamic state deduction processing, and outputs a psychological state evaluation element, including:

[0024] S41, analyze the context association relationship of the current evaluation scene, and generate a state priority weight associated with the multi-modal feature fusion result;

[0025] S42, generate a state transition matrix based on the state priority weight, and perform traversal sorting on the modality type identifier in the multi-modal feature set;

[0026] S43, select the optimal evolution sequence from the state transition matrix by using a dynamic programming strategy, and generate a basic evaluation set containing emotional labels, cognitive states and trend prediction;

[0027] S44, logically integrate the basic evaluation set and the trend prediction parameter to generate a state description segment conforming to the target evaluation framework.

[0028] Optionally, the method further includes:

[0029] Real-time capture state response data during evaluation execution, generating a feedback log containing feature offset and state change information;

[0030] Extract the abnormal state features in the feedback log, and perform similarity matching on the abnormal state features and the historical evaluation case library to generate an adaptive adjustment instruction;

[0031] updating a modality correlation threshold parameter of the machine learning model based on the adaptive adjustment instruction;

[0032] injecting the updated modality correlation threshold parameter into the evaluation model to recalculate state priority weights in a state transition matrix.

[0033] Optionally, the abnormal state features in the feedback log are extracted, and the abnormal state features are matched with a historical evaluation case library for similarity to generate an adaptive adjustment instruction, including:

[0034] dividing the abnormal state features into time windows to generate a plurality of state segments and corresponding multi-modal data sequences;

[0035] performing root cause analysis on each state segment using a trained abnormal classification model to generate a classification label containing feature recognition bias, time series drift, and response delay;

[0036] retrieving a solution template matching the classification label from the historical evaluation case library to generate a candidate adjustment strategy set;

[0037] Based on the matching degree of the multi-modal data sequence and the candidate adjustment strategy set, the strategy with the highest confidence is selected to generate an adaptive adjustment instruction.

[0038] Optionally, the trained abnormal classification model is used to perform root cause analysis on each state segment to generate a classification label containing feature recognition bias, time series drift, and response delay, including:

[0039] performing time series segmentation processing on the state segment to generate a multi-modal data sequence slice containing a start trigger time and an end time;

[0040] performing key frame sampling processing on the multi-modal data sequence slice to generate a state change key frame set and corresponding timestamp index;

[0041] extracting the feature offset between adjacent key frames in the state change key frame set to generate a feature evolution trajectory vector and a time interval sequence;

[0042] performing trend calculation processing on the feature evolution trajectory vector to generate an abnormal evolution mode feature vector;

[0043] inputting the abnormal evolution mode feature vector into a spatio-temporal convolution subnet of the trained abnormal classification model to extract local features to generate a spatial abnormal activation map;

[0044] performing sliding window mean filtering processing on the time interval sequence to generate a smoothed response delay time sequence;

[0045] The response delay time sequence is input into a gated recurrent subnet of the trained anomaly classification model to perform period pattern matching, to generate a time-dimension anomaly score; the spatial anomaly activation map is subjected to channel maximum pooling processing to generate a spatial-dimension anomaly score; the spatial-dimension anomaly score and the time-dimension anomaly score are subjected to feature cross fusion processing to generate a spatio-temporal joint anomaly probability distribution;

[0046] Peak detection results of deviation probability, time series drift probability and response delay probability are extracted from the spatio-temporal joint anomaly probability distribution; a preset probability threshold is subjected to dynamic threshold comparison processing according to the peak detection results to generate a classification label candidate set containing probability ranking;

[0047] The classification label candidate set is subjected to timestamp index alignment verification processing to generate a target classification label consistent with an abnormal state phase in the multi-modal data sequence slice; the target classification label is subjected to anomaly type reverse verification processing with the feature evolution trajectory vector to generate a final classification label set containing confidence weight.

[0048] Optionally, the method further comprises:

[0049] A virtual feature disturbance parameter is injected before evaluation model execution, and the virtual feature disturbance parameter is used to simulate a random offset scenario of multi-modal data features;

[0050] A fault tolerance processing result of the evaluation model on the disturbed features is monitored to generate a stability evaluation index;

[0051] When the stability evaluation index is lower than a preset threshold, an online learning mode of the trained machine learning model is triggered;

[0052] A gradient backpropagation update is performed on a time series feature parameter of the trained machine learning model based on feature difference data before and after disturbance;

[0053] The monitoring of the fault tolerance processing result of the evaluation model on the disturbed features to generate the stability evaluation index comprises:

[0054] A number of evaluation failures and an abnormal capture success rate caused by feature errors in an evaluation execution process are counted;

[0055] A ratio of the number of evaluation failures to a total number of evaluation steps is calculated to generate a first evaluation coefficient;

[0056] An event proportion in the abnormal capture success rate that matches a preset fault tolerance rule is extracted to generate a second evaluation coefficient;

[0057] The first evaluation coefficient and the second evaluation coefficient are subjected to weighted summation to generate a comprehensive stability evaluation index.

[0058] Optionally, the gradient back propagation update of the time sequence feature parameters of the trained machine learning model based on the feature difference data before and after the disturbance includes:

[0059] Timestamp alignment processing is performed on the feature difference data before and after the disturbance to generate a matching pair sequence containing the feature set before the disturbance and the feature set after the disturbance;

[0060] The feature offset in the matching pair sequence is extracted to generate a horizontal feature vector and a vertical feature vector of each multi-modal data;

[0061] Distance calculation is performed on the horizontal feature vector and the vertical feature vector to generate a feature difference vector set of each multi-modal data;

[0062] A feature regression loss function is constructed according to the feature difference vector set, and the feature regression loss function contains the mean square error of the predicted feature of the trained machine learning model and the actual feature after the disturbance;

[0063] A loss value tensor required for gradient back propagation is generated by performing a differentiable transformation on the feature regression loss function;

[0064] The time sequence feature parameters of the trained machine learning model are traversed, the partial derivative of the loss value tensor to each time sequence feature is calculated, and a time sequence feature gradient matrix is generated;

[0065] Momentum smoothing processing is performed on the time sequence feature gradient matrix to generate a parameter update direction vector after attenuation;

[0066] Based on the parameter update direction vector and a preset learning rate parameter, the time sequence feature parameters of the trained machine learning model are iteratively adjusted in increments;

[0067] The change rate of the feature regression loss function is monitored in real time during the incremental adjustment process to generate a loss convergence state identifier;

[0068] When the loss convergence state identifier reaches a stable threshold, the gradient back propagation update is terminated and the time sequence feature parameters are frozen;

[0069] The updated time sequence feature parameters are injected into the forward propagation path of the trained machine learning model, and the modal domain processing and feature fusion processing are re-executed to verify the feature extraction precision improvement result.

[0070] Optionally, the method further includes:

[0071] A cross-scene evaluation adapter is constructed, and the index semantic difference of different evaluation frameworks is parsed through the cross-scene evaluation adapter;

[0072] convert the mental state evaluation element into underlying evaluation instructions supported by the target scene;

[0073] preserve the temporal dependency relationship and exception handling context of the state label during the conversion process;

[0074] inject evaluation optimization parameters matched by the target scene to generate evaluation result description code that meets cross-scene execution conditions.

[0075] Optionally, the cross-scene evaluation adapter is constructed, and the semantic differences of indicators of different evaluation frameworks are resolved through the cross-scene evaluation adapter, including:

[0076] An evaluation framework semantic rule library is established to store the indicator mapping table and parameter passing path of each scene;

[0077] The mental state evaluation element is parsed by an abstract semantic tree to generate intermediate representation layer data;

[0078] Based on the intermediate representation layer data, the indicator mapping table is queried to generate a scene-compatible indicator replacement scheme;

[0079] The parameter passing path with conflicts is processed by dependency injection to generate an unambiguous indicator conversion result;

[0080] The parameter passing path with conflicts is processed by dependency injection to generate an unambiguous indicator conversion result, including:

[0081] Identify conflict nodes with type ambiguity or scope overlap in the parameter passing path to generate a conflict path identifier set;

[0082] Extract context metadata of each node in the conflict path identifier set, which includes parameter type declaration, lifetime marker, and call stack fingerprint;

[0083] Generate an interface proxy class name corresponding to the conflict node according to the context metadata, which inherits the native interface of the target scene and injects type coercion constraints;

[0084] Resolve the dependency relationship chain of the interface proxy class name in the target scene to generate a dynamic binding strategy containing parameter scope isolation boundaries;

[0085] Map the conflict nodes in the parameter passing path to scene-compatible interface method instances based on the dynamic binding strategy to generate a dynamic binding queue of parameter instances;

[0086] Traverse the interface method instance in the dynamic binding queue, verify the compatibility state of the interface method instance and the target scene index semantics, and generate a parameter binding validity state code;

[0087] According to the parameter binding validity state code, the interface method instance without type conflict is screened out, and a scene-independent intermediate index sequence is generated;

[0088] The intermediate index sequence is filled according to the semantic rules of the target scene, and an unambiguous index conversion result is generated.

[0089] Compared with the prior art, the beneficial effects of the present application are:

[0090] The juvenile psychological state intelligent dynamic evaluation method based on multi-modal data can capture the psychological activity signs of the evaluation object from different aspects by collecting multi-modal psychological data containing voice emotional information and expression behavior relationship. The voice emotional information contains various emotion-related clues such as intonation, speed and tone, and the expression behavior relationship transmits the instant psychological reaction through facial muscle movement and eye movement, which complement each other and together constitute a rich information source reflecting the psychological state, avoiding the evaluation limitations caused by single information of single modal data.

[0091] In the feature extraction and fusion stage, dynamic features are extracted and processed by using a trained machine learning model to generate a multi-modal feature set containing modal type identification and time sequence coding mapping relationship. This processing method breaks through the limitation of simple splicing in traditional feature fusion. By establishing the mapping between modal type and time sequence coding, the features of different modalities can be placed in a unified time sequence framework, so that the correlation between modal features can be fully explored, and the fused features can better reflect the overall characteristics of the psychological state.

[0092] The state analysis network performs time sequence correlation analysis on the dynamic features to generate a fusion result containing emotional dimension features and cognitive mode features. This process deeply considers the continuity and correlation of psychological state in the time dimension. Through time sequence correlation analysis, the internal relationship between psychological features at different time points can be found, and the interaction process of emotional changes and cognitive mode evolution can be clearly presented, so that the understanding of psychological state no longer stays at the isolated feature level, but extends to the grasp of its dynamic development law.

[0093] The feature fusion result is input into an evaluation model for dynamic state deduction, and psychological state evaluation elements are output. This dynamic deduction mechanism can adapt to the real-time change characteristics of the psychological state. By continuously receiving new multi-modal data, updating the feature information and performing deduction, the evaluation elements reflecting the current psychological state can be generated in real time, so that the evaluation process is synchronized with the change of the psychological state, and subtle changes that may be ignored by static evaluation can be captured in time.

[0094] The method forms a complete process from multi-modal data acquisition, dynamic feature extraction, multi-modal feature fusion, time sequence correlation analysis to dynamic state deduction, and each link cooperates with each other to make the evaluation of the psychological state more comprehensive, in-depth and in line with the nature of the dynamic change. This evaluation method can adapt to the needs of psychological state evaluation in different scenarios, whether it is long-term tracking of patient psychological state in clinical diagnosis or real-time grasp of student learning psychology in the field of education, it can provide more valuable evaluation information. BRIEF DESCRIPTION OF DRAWINGS

[0095] Figure 1 The working principle diagram of the intelligent dynamic evaluation method of the psychological state of adolescents based on multi-modal data is provided.

[0096] Figure 2 The flowchart for feature fusion processing and time sequence correlation analysis is provided.

[0097] Figure 3 The flowchart for dynamic state deduction processing is provided.

[0098] Figure 4 The flowchart for real-time feedback and dynamic update is provided.

[0099] Figure 5 The flowchart for abnormal state feature processing and adjustment instruction generation is provided. DETAILED DESCRIPTION

[0100] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0101] As shown in Figure 1 The present application provides an intelligent dynamic evaluation method of the psychological state of adolescents based on multi-modal data, which will be described in detail below in combination with specific steps:

[0102] S1, acquire multi-modal psychological data of a target evaluation object, and extract dynamic features in the multi-modal psychological data; wherein the multi-modal psychological data contains speech emotion information and expression behavior relationship.

[0103] The dynamic features refer to feature information extracted from the multi-modal psychological data of the target evaluation object and changing over time. In combination with the speech emotion information and expression behavior relationship contained in the multi-modal psychological data, specifically, the dynamic features in the speech emotion information include formant trajectory, short-time energy spectrum, and other changing features over time; the dynamic features in the expression behavior relationship include facial action unit intensity, eye movement vector, muscle micro-expression change rate, and other evolution features in the time dimension. These features can reflect the change trajectory of the psychological state of the evaluation object at different time points, providing a basis for subsequent dynamic evaluation.

[0104] In a feasible implementation manner, as shown in Figure 2 The flowchart shown in FIG. 1 is a feature fusion processing and time sequence correlation analysis process. The collected speech emotion information and expression behavior relationship data are processed in a modal domain. The multi-modal psychological data is decomposed into independent speech data domain and expression data domain through a pre-defined modal segmentation algorithm. Each data domain generates corresponding initial feature parameters; wherein the formant trajectory, short-time energy spectrum, and facial action unit intensity, eye movement vector, and muscle micro-expression change rate are extracted for the speech domain and expression domain respectively. The time sequence features of each data domain are subjected to attention weighting processing, and a multi-head attention mechanism is used to calculate feature importance scores. A learnable weight matrix is constructed, which generates correlation scores through the dot product operation of feature vectors and context vectors, and is normalized by a softmax function. The features with correlation scores exceeding a dynamic threshold are assigned exponentially increasing weight coefficients, and the low correlation features are subjected to weight attenuation. The weighted multi-domain feature vector is output, which retains the spatio-temporal structure information of the original data. Specifically, when the time sequence features of each data domain are subjected to attention weighting processing, a multi-head attention mechanism is first used to calculate feature importance scores, which can capture the correlation between features from multiple dimensions. A learnable weight matrix is constructed to perform dot product operation on the feature vectors and context vectors of each data domain, so as to measure the correlation between the features and the context. Then, the operation result is normalized by a softmax function to obtain standardized correlation scores. According to the dynamic threshold, the correlation scores are judged, the high correlation features exceeding the threshold are assigned exponentially increasing weight coefficients to highlight their importance, and the low correlation features below the threshold are subjected to weight attenuation to reduce their influence on subsequent processing. The final output of the weighted multi-domain feature vector will completely retain the structure relationship of the original data in time and space, ensuring the time sequence integrity and spatial correlation of the features.

[0105] S2, performing feature fusion processing on the multi-modal psychological data by using the trained machine learning model to generate a multi-modal feature set; wherein the multi-modal feature set comprises a mapping relationship between a modal type identifier and a time sequence encoding.

[0106] wherein the multi-domain feature vector is matched with a preset modal correlation threshold. The cosine similarity of the weighted feature vector in the Hilbert space is calculated, and a similarity threshold dynamic adjustment mechanism is set. When the feature stability in the continuous multiple time windows is higher than the critical value, the threshold strictness is automatically increased. The candidate feature set that meets the confidence condition is screened out, and the candidate feature set contains the intersection of cross-modal features. Redundancy elimination processing is performed on the candidate feature set, and the statistical dependence between features is analyzed by mutual information entropy calculation. A feature mutual information matrix is constructed, and when the values of the non-diagonal elements in the matrix exceed the preset upper limit of the entropy value, feature dimensionality reduction operation is performed. The principal component analysis is used to retain the orthogonalized feature subset with mutual information entropy lower than the critical value. The multi-modal feature set containing the modal type identifier is output, wherein each feature is attached with a normalized time sequence encoding. The time sequence encoding is generated based on the ratio of the timestamp sequence to the feature sampling period, and the absolute time coordinate is mapped to the standard time axis of [0, 1], for example, each 0.1 second feature point corresponds to an increase of 0.02 in the time sequence encoding value.

[0107] In an available implementation, a fixed time length sliding window is used to intercept psychological data, and the window length is dynamically configured according to the evaluation scene. The default setting for the clinical diagnosis scene is a 5-second window, and the setting for the behavior observation scene is a 2-second window. Standardization processing is performed on the intercepted feature time segments, the mean and standard deviation of each modal feature are calculated, and Z-score transformation is applied to convert the feature values to a distribution space with zero mean and unit variance. Random noise conforming to the Gaussian distribution is injected into the standardized feature segments, and the noise intensity is set to 5%-10% of the feature standard deviation to enhance the robustness of the model to input disturbances. The optimized state analysis input data is output, which retains the relative time relationship of each modal feature.

[0108] Optionally, the specific implementation process of S2 includes S21-S24:

[0109] S21, performing modal domain processing on the multi-modal psychological data to generate a plurality of data domains and corresponding initial feature parameters;

[0110] S22, extracting a time sequence feature sequence of each data domain and performing attention weighting processing on the time sequence feature sequence to generate a weighted multi-domain feature vector;

[0111] S23, performing similarity calculation on the multi-domain feature vector and the preset modal correlation threshold to screen out candidate features that meet the confidence condition;

[0112] S24, performing redundancy elimination processing on the candidate features to generate a multi-modal feature set containing modal type identifiers, each multi-modal feature containing normalized time-series encoding relative to the multi-modal psychological data;

[0113] wherein, based on the multi-modal feature set, a state analysis network is used to perform time-series correlation analysis on the dynamic features to generate a feature fusion result containing emotional dimension features and cognitive mode features, including:

[0114] According to the normalized time-series encoding in the multi-modal feature set, time window interception processing is performed on the multi-modal psychological data to generate a plurality of feature time segments;

[0115] Standardization and feature enhancement processing is performed on each feature time segment to generate optimized state analysis input data;

[0116] The sequence feature extraction is performed on the state analysis input data by using the recurrent memory subnetwork in the state analysis network to generate an emotional dimension feature vector;

[0117] The feature mode matching is performed on the feature time segment in parallel by using the mode classification model to generate a cognitive mode probability distribution;

[0118] The emotional dimension feature vector and the cognitive mode probability distribution are spliced to generate a multi-modal feature fusion result.

[0119] The specific content of the sequence feature extraction performed on the state analysis input data by using the recurrent memory subnetwork in the state analysis network to generate an emotional dimension feature vector is as follows.

[0120] When the sequence feature extraction is performed by using the recurrent memory subnetwork in the state analysis network, the subnetwork uses a three-layer gated recurrent unit architecture to process the state analysis input data through the coordinated operation of the reset gate and the update gate. The reset gate is responsible for controlling the forgetting proportion of historical state information, and the update gate adjusts the degree of new feature information integrated into the current state. The optimized state analysis input data is input into the recurrent memory subnetwork in time steps, and the network processes the time-series features moment by moment to learn the long-term dependency patterns in the data. After traversing all the features of the time steps, the hidden layer state of the last moment is taken as the emotional dimension feature vector output. The dimension of the vector is consistent with the hidden layer dimension of the gated recurrent unit in the recurrent memory subnetwork, and can fully reflect the emotional change characteristics of the evaluation object in the time dimension.

[0121] S3, based on the multi-modal feature set, a state analysis network is used to perform time-series correlation analysis on the dynamic features to generate a feature fusion result containing emotional dimension features and cognitive mode features.

[0122] In one possible implementation, the state analysis network is used to analyze the time sequence correlation of the dynamic features. The specific implementation process includes:

[0123] The recurrent subnetwork of the state analysis network processes the time sequence data. The recurrent subnetwork uses a three-layer gated recurrent unit architecture, and the hidden layer dimension is set to 256 neurons. The unit state updating process includes the cooperation of the reset gate and the update gate, and specifically includes: the reset gate controls the forgetting proportion of the historical state, and the update gate adjusts the fusion degree of the new feature information. The feature fragments of each time step are input into the network in turn, and the network learns the long-term dependence mode through time reverse propagation. The hidden layer state at the final moment is output as an emotional dimension feature vector, and the vector dimension is consistent with the hidden layer dimension of the recurrent unit. At the same time, a pattern classification model is used to process the feature time fragments in parallel. The classification model uses a combination structure of a convolutional neural network and a fully connected layer, and the input layer receives multi-channel data of the feature fragments. The pattern features of the local receptive field are extracted through three convolutional layers, and the convolution kernel size is configured as 3*3*3. The feature dimension is compressed through the maximum pooling operation in the pooling layer, and finally the cognitive mode probability distribution vector is generated by the softmax classification layer, and the vector dimension corresponds to the preset number of cognitive mode classifications. The cognitive mode probability distribution vector is the result generated by the pattern classification model after matching the feature mode of the feature time fragment, and the vector dimension corresponds to the preset number of cognitive mode classifications. Each element in the vector represents the probability that the psychological characteristics of the evaluation object conform to a certain preset cognitive mode, and through these probability values, the distribution tendency of the evaluation object at the cognitive level can be intuitively presented, providing a quantitative basis for the cognitive level for subsequent fusion of emotional dimension features and generation of comprehensive evaluation results.

[0124] The specific implementation process of fusing the emotional dimension feature vector and the cognitive mode probability distribution vector includes:

[0125] The tensor splicing operation is performed along the feature dimension; wherein the specific process of aligning the dimensions before splicing includes: when the emotional feature vector dimension is higher than the cognitive mode probability distribution vector, the cognitive mode probability distribution vector is linearly projected to expand the dimension; when the cognitive distribution dimension is higher, the emotional feature vector is upgraded through a fully connected layer. The fusion result matrix formed by splicing has a row dimension of the total number of feature fragments and a column dimension of the sum of the dimensions of the two vectors. The result matrix retains the time sequence coding index of the original feature fragments, forming a multi-modal feature fusion result containing the emotional state intensity gradient and the cognitive mode distribution.

[0126] S4, input the feature fusion result into the evaluation model for dynamic state deduction processing, and output a psychological state evaluation element; wherein the psychological state evaluation element includes a state label and a trend prediction parameter generated based on the feature evolution sequence.

[0127] For example,Figure 3 Fig. 1 shows a flowchart of the dynamic state inference process; as Figure 4 Fig. 2 shows a flowchart of real-time feedback and dynamic update.

[0128] Optionally, the implementation process of S4 includes S41-S44:

[0129] S41, analyze the context association of the current evaluation scene, and generate a state priority weight associated with the multi-modal feature fusion result;

[0130] S42, generate a state transition matrix based on the state priority weight, and perform traversal sorting on the modal type identifier in the multi-modal feature set;

[0131] In a feasible implementation, a domain knowledge graph of the current evaluation scene is loaded; wherein the domain knowledge graph of the current evaluation scene stores the causal rules between emotional states, cognitive mode constraint conditions and historical state transition records in a triple structure. The centrality index of each emotional state node is calculated by traversing the knowledge graph nodes through the graph neural network, and the state priority weight matrix is generated by multiplying the node in-degree value and the feature confidence. Based on the state priority weight matrix, a state transition matrix is constructed, wherein the dimension of the state transition matrix is determined by the number of emotional categories in the current multi-modal feature set. The element definition rule of the matrix is: if there is a legal path for the transition from emotional state A to B in the knowledge graph, the element value is equal to the product of the priority weight and the time decay coefficient; otherwise, it is zero. At the same time, topological sorting is performed on the modal type identifier in the multi-modal feature set, the dependence relationship between the modes is analyzed by using Kahn algorithm, and an ordered feature processing queue is formed. Specifically, the domain knowledge graph of the current evaluation scene is loaded, which stores the causal rules between emotional states, cognitive mode constraint conditions and historical state transition records in the form of triples. The centrality index of each emotional state node in the knowledge graph is calculated by traversing the nodes through the graph neural network, and the state priority weight matrix is generated by multiplying the node in-degree value and the feature confidence. Based on the weight matrix, a state transition matrix is constructed, and the dimension of the matrix is determined by the number of emotional categories in the multi-modal feature set. If there is a legal transition path between emotional states in the knowledge graph, the corresponding element value of the matrix is the product of the priority weight and the time decay coefficient, otherwise it is zero. The dependence relationship between the modal type identifiers in the multi-modal feature set is analyzed by using Kahn algorithm, and an ordered feature processing queue is formed by topological sorting, ensuring that the feature processing order conforms to the dependence logic between the modes.

[0132] S43, select the optimal evolution sequence from the state transition matrix by using dynamic programming strategy, and generate a basic evaluation set containing emotional labels, cognitive states and trend prediction;

[0133] In a feasible implementation, the embodiment of the present application adopts a dynamic programming strategy to search for an optimal state evolution path, and specifically comprises the following steps:

[0134] (1) defining a state evolution cost function, which is composed of three parts: the emotion label probability of the current state, the feature fusion result, the transition cost from the previous state to the current state in the state transition matrix, and the logical conflict value between the cognitive mode feature and the current state;

[0135] (2) performing an initialization stage, and assigning an initial cost to each emotional state; wherein the cost is equal to the negative logarithm of the initial probability;

[0136] (3) performing a recursive stage, and performing recursion by using the Viterbi algorithm, specifically comprising the following steps: traversing the feature processing queue of each time step, calculating the minimum cumulative cost of all possible states at the current time and recording the backtracking path; and selecting the path with the minimum cumulative cost at the last time step as the optimal sequence output. Wherein the sequence includes an emotion label sequence sorted by time, a cognitive state classification code corresponding to each label (taking the classification corresponding to the maximum probability), and a trend prediction parameter (generated by extrapolating the cost of the next three time steps). The results obtained through the above steps form a basic evaluation set.

[0137] S44, logically integrating the basic evaluation set and the trend prediction parameter to generate a state description segment that meets the target evaluation framework.

[0138] Wherein, the description segment syntax rules are predefined in the framework configuration file, for example, the clinical scene template includes three mandatory fields of “subject emotional state”, “cognitive mode performance” and “short-term evolution trend”. The emotion label sequence is mapped into a composite structure of time phrase + state description (such as “presenting anxiety characteristics for the first 2 seconds, and turning into calm for the next 3 seconds”) according to the time axis. The cognitive state code is converted into a natural language description by looking up the predefined cognitive mode dictionary. The trend prediction parameter generates a trend quantization value by linear interpolation, such as “the anxiety value will decrease by 30% within 5 seconds”. The integrated description segment retains the timestamp markers of the original sequence to form the final psychological state evaluation element.

[0139] Optionally, the method further comprises:

[0140] capturing state response data in real time during the evaluation execution, and generating a feedback log containing feature offset and state change information;

[0141] extracting abnormal state features from the feedback log, and performing similarity matching between the abnormal state features and a historical evaluation case library to generate adaptive adjustment instructions;

[0142] In a feasible implementation process, a real-time feedback mechanism is started synchronously during the evaluation execution process. The state response data stream is captured by an asynchronous thread, which contains the feature offset of the intermediate layer of the evaluation model, i.e., the spatial distance between the real-time feature and the benchmark feature, and the state change event (emotion category switching or cognitive mode mutation). The data stream is formatted to generate a structured feedback log, wherein the log entry contains a timestamp, an offset vector, and an event type code.

[0143] Optionally, the abnormal state feature in the feedback log is extracted, and the abnormal state feature is matched with a historical evaluation case library to generate an adaptive adjustment instruction, including:

[0144] The abnormal state feature is divided into a time window to generate a plurality of state segments and corresponding multi-modal data sequences;

[0145] In a feasible implementation process, the process of dynamically dividing the execution time window of the abnormal feature extraction includes: setting an initial window length based on the typical abnormal duration in the historical evaluation case library, and defaulting to 1.2 seconds; when the abnormal density in the window exceeds a threshold value, triggering dynamic expansion (maximum expansion to 3 seconds). Wherein, each window generates an independent state segment and a corresponding multi-modal data sequence fingerprint (MD5 digest value of sequence features).

[0146] An abnormal classification model trained is used to perform root cause analysis on each state segment to generate a classification label containing feature recognition bias, time sequence drift, and response delay;

[0147] The trained abnormal classification model processes the state segment, specifically including: inputting the state segment into a three-dimensional convolution layer to extract spatial features, capturing time sequence patterns through a long short-term memory network, and finally outputting a multi-dimensional classification label containing feature recognition bias, time sequence drift, and response delay.

[0148] The abnormal classification model is a model for root cause analysis of abnormal state segments, which can identify three types of abnormalities: feature recognition bias, time sequence drift, and response delay. Its structure combines a space-time convolution subnet and a gated recurrent subnet. The space-time convolution subnet is responsible for extracting spatial abnormal information in the abnormal state feature to generate a spatial abnormal activation map; the gated recurrent subnet is used to capture abnormal patterns in the time dimension to generate a time dimension abnormal score. The output of the two can be obtained after feature cross fusion. The space-time joint abnormal probability distribution, and then the precise classification of the root cause of the abnormality. The establishment of this model needs to collect a labeled data set containing various types of abnormal state features first, and then use these data sets to train the model. Through iterative adjustment of the feature extraction parameters of the space-time convolution subnet and the time sequence learning parameters of the gated recurrent subnet, the model's ability to recognize and classify abnormal features is optimized to meet the accuracy requirements of root cause analysis.

[0149] wherein, as Figure 5 Fig. 3 shows a flowchart of the abnormal state feature processing and adjustment instruction generation.

[0150] Optionally, the trained abnormal classification model is used to perform root cause analysis on each state segment to generate classification labels containing feature recognition bias, time sequence drift, and response delay, including:

[0151] The state segment is subjected to time sequence segmentation processing to generate a multi-modal data sequence slice containing a start trigger time and an end time;

[0152] Wherein, the start trigger time and the end time are determined by setting a dynamic time boundary detection mechanism to detect the second derivative zero crossing point of the multi-modal data stream, generating a sequence slice with a variable time length but meeting the minimum information constraint. Each slice contains a speech spectrum feature matrix and an expression geometry feature tensor under a continuous timestamp index.

[0153] The multi-modal data sequence slice is subjected to key frame sampling processing to generate a state change key frame set and a corresponding timestamp index;

[0154] Wherein, the multi-modal data sequence slice is subjected to key frame sampling processing, wherein the sampling algorithm is dynamically adjusted based on the feature evolution rate, and the gradient change rate is used as the core criterion. The feature difference measurement formula (1) of adjacent time frames is defined as follows:

[0155]

[0156] Wherein, Γ(t) represents the feature change intensity at time point t, M is the total number of modalities, is the feature vector of the mth modality at time t, and τ is the time reference unit (default is 0.1 seconds). When Γ(t) exceeds the adaptive threshold (which is determined by multiplying the average intensity of the first ten time windows by a coefficient of 1.3), the time point is marked as a key frame. The generated state change key frame set carries a timestamp index offset;

[0157] The feature offset between adjacent key frames in the state change key frame set is extracted to generate a feature evolution trajectory vector and a time interval sequence;

[0158] The feature offset between adjacent key frames in the key frame set is extracted; the multi-dimensional feature difference between the two key frames t k and t k+1 is calculated, and a feature evolution trajectory vector D k is constructed, which is represented by the following formula (2):

[0159]

[0160] Simultaneously record the key frame time interval sequence

[0161] Trend calculation is performed on the feature evolution trajectory vector to generate an abnormal evolution mode feature vector;

[0162] A linear regression model of the feature trajectory is fitted in the sliding time window, a slope parameter is extracted to constitute a mode feature, and the mode feature is represented by the following formula (3):

[0163] S = [β1, β2, …, β M ](3)

[0164] Wherein, β m m represents the evolution trend coefficient of the mth mode feature, and a positive value represents an enhancement trend and a negative value represents a decay trend. S represents a mode feature vector reflecting the evolution direction and intensity of an abnormal state.

[0165] The abnormal evolution mode feature vector is input into a spatio-temporal convolution subnetwork of the trained anomaly classification model to extract local features and generate a spatial anomaly activation map;

[0166] Wherein, the spatio-temporal convolution subnetwork receives the abnormal evolution mode feature vector S and expands the receptive field through three layers of dilated convolution operations: the first layer of convolution kernel size is 3xM (time dimension x mode dimension), and the dilated coefficient is 1; the second layer of kernel size is 5xM, and the dilated coefficient is 2; the third layer of kernel size is 7xM, and the dilated coefficient is 4. After each layer of convolution, a ReLU activation function and batch normalization processing are applied, and finally a spatial anomaly activation map Wherein T ′ is the length after time dimension compression.

[0167] The time interval sequence is subjected to sliding window mean filtering to generate a smoothed response delay time sequence;

[0168] In an available implementation, the response delay time sequence ΔT is subjected to sliding window mean filtering using a time dimension processing path. Wherein, the window width is set to 20% of the sequence length, and the smoothed data is obtained by using a Hanning window to weight the smoothed data.

[0169] The response delay time sequence is input into a gated recurrent subnetwork of the trained anomaly classification model to perform periodic pattern matching, generate a time dimension anomaly score; the spatial anomaly activation map is subjected to channel maximum pooling to generate a spatial dimension anomaly score; the spatial dimension anomaly score and the time dimension anomaly score are subjected to feature cross fusion processing to generate a spatio-temporal joint anomaly probability distribution;

[0170] Wherein, the smoothed sequence is input into the gating cycle subnetwork, which adopts a double-layer bidirectional GRU architecture, and the hidden layer dimension is set to 64 units. The forward and backward output states are spliced and passed through a time attention layer to calculate the importance weight of each time step, and generate a time-dimension anomaly score vector Wherein, T ″ is the number of time steps of the GRN output.

[0171] The peak detection results of the deviation probability, the time series drift probability and the response delay probability are extracted from the spatio-temporal joint anomaly probability distribution; and the preset probability threshold is dynamically compared according to the peak detection results to generate a classification label candidate set containing probability ranking;

[0172] Wherein, a feature fusion module is adopted to receive the double-path output, i.e. the time-dimension processing path + the spatial-dimension anomaly output; a cross-channel maximum pooling operation is performed on the spatial anomaly activation map A to reduce the dimension and generate a spatial-dimension anomaly score vector The vector is combined with the time-dimension anomaly score vector E t to be input into a feature cross layer for calculation, and the specific calculation process includes:

[0173] The vector outer product is calculated to generate a T ′ ×T ″ interaction matrix.

[0174] A learnable position encoding is added to reserve the spatio-temporal correlation;

[0175] The two-layer fully connected network is compressed to a three-dimensional probability space;

[0176] The spatio-temporal joint anomaly probability distribution is output , and the three dimensions thereof correspond to the probability estimates of the feature identification deviation p d , the time series drift p s and the response delay p r , respectively.

[0177] Wherein, peak detection and classification decision are performed in the probability space, and a dynamic threshold generation rule is defined, wherein different anomaly probabilities θ are output, which are represented by the following formula (4):

[0178] θ = μ + ασ (4)

[0179] Wherein, μ is the mean value of the probability of historical normal samples, σ is the standard deviation, and α is a sensitivity control factor (default 1.5). When p d > θ d , p s > θ s or p r > θ rTime trigger candidate label generation. Three types of anomaly probability are sorted from high to low, and a sorted index classification label candidate set is generated.

[0180] Timestamp index alignment verification processing is performed on the classification label candidate set to generate a target classification label consistent with the abnormal state phase in the multi-modal data sequence slice; the target classification label is subjected to abnormal type reverse verification processing with the feature evolution trajectory vector to generate a final classification label set containing confidence weights.

[0181] Among them, the embodiment of the present application adopts a double verification mechanism to complete the timestamp index alignment verification processing on the classification label candidate set, specifically including:

[0182] Align the timestamp index: align the timestamp t k Map back to the original data slice, check whether there is a bottom feature jump (such as a voice fundamental frequency mutation exceeding 50Hz or an expression motion unit intensity jump exceeding 40%) supporting the abnormal event within 1 second before and after this moment.

[0183] Reverse verification of abnormal type: reverse map the abnormal type (such as response delay) specified by the classification label to the feature evolution trajectory vector, and verify whether there is an expected pattern (such as the delay type should be in the form of exponential decay) in D k The verified label is attached with a confidence weight, which is represented by the following formula (5):

[0184]

[0185] The final generated form is the final classification label set {(type, w, t k )} where t kThe precise moment of marking the abnormality occurs; w is the confidence weight, and Pi is the type value of the i-th label. Among them, the feature matching degree is a quantitative index for measuring the matching degree of the current abnormal state feature evolution trajectory vector (constructed by the feature offset amount of the adjacent key frame speech emotion information, the formant trajectory, and the feature offset amount of the facial action unit intensity and eye movement vector of the expression behavior relationship) and the corresponding classification label (such as feature recognition deviation, time sequence drift) feature mode in the preset typical abnormal feature mode library. It is obtained by calculating the cosine similarity of the current speech feature and the typical mode, the Euclidean distance normalized value of the expression feature and the typical mode, and the weighted sum according to the weight. The value range is [0, 1]; the time consistency is the overlap ratio of the abnormal starting-terminating time interval labeled by the abnormal classification label and the actual feature jump (such as speech fundamental frequency mutation, expression key point displacement) time interval in the multi-modal data sequence, and the average value of the absolute value of the Pearson correlation coefficient of the abnormal probability change trend corresponding to the label and the multi-modal feature evolution trajectory trend. The value range is [0, 1], which is used to ensure that the label is consistent with the actual abnormal feature occurrence phase in the time dimension, and both are used as the verification basis of the validity of the label. The abnormal probability (p_i) corresponding to the label cooperates to participate in the confidence weight calculation.

[0186] Retrieve the solution template matched with the classification label from the historical evaluation case library to generate a candidate adjustment strategy set;

[0187] Based on the matching degree of the multi-modal data sequence and the candidate adjustment strategy set, the strategy with the highest confidence is selected to generate an adaptive adjustment instruction.

[0188] In a feasible implementation, the inverted index strategy is used to retrieve solutions from the historical evaluation case library. The specific process includes: constructing an inverted index table of abnormal type labels and case items, and retrieving matched candidate solution templates. Similarity filtering is performed on the candidate set, and the Jaccard similarity coefficient of the current multi-modal data sequence and the case library sequence fingerprint is calculated. The coefficient is calculated by matching the key event position difference in the data stream. Among them, the candidate solution with a similarity coefficient greater than 0.85 is selected to join the adjustment strategy set. Finally, the strategy is selected through a weighted voting mechanism. Each candidate strategy is assigned a basic weight according to its historical success record, and an additional weight is added to the current scene strategy matching degree. The strategy with the highest total weight is selected to generate an adaptive adjustment instruction.

[0189] Based on the adaptive adjustment instruction, the modal correlation threshold parameter of the machine learning model is dynamically updated;

[0190] In a feasible implementation, the incremental gradient adjustment method is used to update the modal correlation threshold. Specifically, it includes:

[0191] Analyzing the parameter update direction (increase / decrease) and amplitude value (discrete gradient level) in the adaptive adjustment instruction;

[0192] The dynamic modification of the modal correlation threshold parameter during the machine learning model running period includes: inserting a threshold correction operator in the similarity calculation layer, which linearly transforms the original threshold according to the gradient level, such as gradient level 1 corresponding to threshold ±0.05.

[0193] The updated modal correlation threshold parameter takes effect immediately, triggering the recalculation of the state transition matrix, clearing the existing matrix cache, and regenerating the state priority weight and transition matrix element value using the new threshold. Among them, the evaluation model automatically adopts the updated logic parameter to continue executing the analysis task in the next processing period.

[0194] The updated modal correlation threshold parameter is injected into the evaluation model to recalculate the state priority weight in the state transition matrix.

[0195] Optionally, the method further comprises:

[0196] Injecting a virtual feature disturbance parameter before the evaluation model executes, which is used to simulate the random offset scenario of multi-modal data features;

[0197] Among them, for the speech emotion information stream, the amplitude modulation disturbance scheme includes: selecting 10% of the speech frames at a fixed sampling interval, and applying ±15% random fluctuation to the amplitude spectrum. For the expression behavior data stream, the spatial coordinate offset disturbance is implemented: randomly selecting 20% of the facial key points, and generating a displacement disturbance of [-0.2, 0.2] centimeters in three-dimensional space. Through the system configuration interface, the disturbance parameter configuration file is dynamically loaded, and the configuration file defines the disturbance type and action range in JSON structure. The key parameters are shown in Table 1:

[0198] Table 1

[0199]

[0200] After executing the disturbance, the evaluation model fault tolerance monitoring process is started. A real-time diagnostic log system is established to record abnormal events in the evaluation process: when the emotion label mutates in adjacent time windows (such as 0-2 seconds label "calm", 2-4 seconds mutate to "angry"), it is recorded as an evaluation failure event caused by feature error; When the feature offset exceeds the safety threshold but is successfully corrected by the model (such as speech amplitude mutation is identified as device noise), it is recorded as an abnormal capture success event. In the 15-minute monitoring period, a total of 42 evaluation steps were recorded, and the diagnostic log output example data record is as follows:

[0201] 3:12: facial expression coordinate offset 0.18 cm triggers alarm, corrected emotion label remains stable (abnormality capture success);

[0202] 5:47: voice fundamental frequency mutation causes "calm" → "anxious" label jump (evaluation failure);

[0203] 11:03: three modalities feature drift simultaneously, model automatically switches to alternative analysis path (abnormality capture success).

[0204] Monitor the fault-tolerant processing results of the evaluation model on the disturbed features, and generate a stability evaluation index;

[0205] The process of generating the stability index based on the diagnostic log includes: extracting the total evaluation steps 42 times within a 15-minute monitoring period, and counting the evaluation failure events 7 times. Calculate the first evaluation coefficient: 7 / 42 ≈ 0.167. From the 28 abnormality capture events, identify 19 events that meet the fault-tolerant rules (such as feature drift amplitude within the fault-tolerant interval, multi-modal conflict within the threshold range), generate the second evaluation coefficient: 19 / 28 ≈ 0.679. Weighted calculation using 0.6:0.4 weight distribution: 0.6×0.167+0.4×0.679≈0.365, get the comprehensive stability evaluation index 0.365 (range [0,1], the lower the value, the more stable).

[0206] When the stability evaluation index is lower than the preset threshold, trigger the online learning mode of the trained machine learning model;

[0207] When the stability index is lower than the preset threshold 0.4, trigger the online learning mode of the machine learning model. Select the feature data of the sixth monitoring time window (8:30-9:45) for parameter update. This period contains three effective disturbance events:

[0208] Voice feature disturbance: the fundamental frequency value from 198 Hz → 215 Hz → 226 Hz in frames 52-55;

[0209] Facial expression feature disturbance: left eyebrow key point coordinates (3.12, 5.78) → (3.32, 5.62) → (3.15, 5.83);

[0210] Cross-modal joint disturbance: voice energy spectrum decreases by 3 dB while mouth corner coordinate offset is 0.12 cm.

[0211] Wherein, a multi-dimensional space distance measurement method is used to construct a difference matrix. Among them, the voice fundamental frequency feature generates a horizontal vector (198, 215, 226) and a vertical vector (198, 217.6, 219.2), the latter is the baseline value before disturbance. Calculate the corrected Euclidean distance between them:

[0212] wherein the expression feature calculates the coordinate displacement vector Δ = [(3.32-3.15), (5.62-5.83)] = (0.17, -0.21), the modulus length The difference matrix is formed by aggregating all feature difference vectors in the time window, with the matrix dimension being 8 (feature number) x 15 (time step).

[0213] The time series feature parameters of the machine learning model completed training are updated by gradient backpropagation based on the feature difference data before and after the disturbance;

[0214] wherein the momentum acceleration mechanism is used for gradient backpropagation update. The initial learning rate is set to 0.002, and the momentum factor is 0.89. The time series feature parameter update direction is calculated according to the weighted history gradient. The specific process includes: the current iteration gradient vector takes the column mean of the difference matrix, and the momentum vector is the exponential weighted average of the previous five iteration gradients. The specific execution of the sixth iteration, the voice feature parameter update value is:

[0215] New gradient = current gradient (6.97,...) x 0.11 + momentum vector (7.81,...) x 0.89;

[0216] Parameter update value = -learning rate 0.002 x new gradient;

[0217] wherein after completing each full parameter update, the fast inference process is executed on the validation set, including: comparing the output features of the feature extraction layer with the benchmark features for cosine similarity, and recording the similarity average change trajectory. The validation results after the sixth iteration are recorded as:

[0218] 1st update: similarity 0.812→2nd: 0.827→3rd: 0.834

[0219] 4th: 0.841→5th: 0.843→6th: 0.845

[0220] wherein when the similarity improvement amplitude of three consecutive iterations is <0.004 (the 4th-6th improvement +0.002, +0.002, +0.002), the loss convergence state is determined to reach the stable condition. The current time series feature parameters are frozen, the model parameter snapshot file (containing 32,119 floating point parameter values) is generated, and the current update period is terminated.

[0221] The updated machine learning model is immediately put into online verification. The modal sub-domain processing of the sixth time window is re-executed: the standard deviation of the fundamental frequency sequence after speech feature sub-domain processing decreases from 12.4 Hz before updating to 9.7 Hz; the false positive rate of coordinate offset detection after expression feature sub-domain processing decreases from 22% to 15%. In the feature fusion stage, the cross-modal feature alignment accuracy increases from 78.3% to 83.6%, proving that the feature extraction accuracy has been enhanced. The entire update cycle takes 137 seconds, during which the evaluation service remains uninterrupted.

[0222] The monitoring and evaluation model evaluates the fault tolerance processing results of the disturbed features to generate a stability evaluation index, including:

[0223] The number of evaluation failures caused by feature errors during statistical evaluation and the abnormal capture success rate are counted;

[0224] The ratio of the number of evaluation failures to the total number of evaluation steps is calculated to generate a first evaluation coefficient;

[0225] The proportion of events matching the preset fault tolerance rules in the abnormal capture success rate is extracted to generate a second evaluation coefficient;

[0226] The first evaluation coefficient and the second evaluation coefficient are weighted and summed to generate a comprehensive stability evaluation index.

[0227] Optionally, the time series feature parameters of the trained machine learning model are updated by gradient backpropagation based on the feature difference data before and after the disturbance, including:

[0228] The timestamp alignment processing is performed on the feature difference data before and after the disturbance to generate a matching pair sequence containing the feature set before the disturbance and the feature set after the disturbance;

[0229] The feature offset in the matching pair sequence is extracted to generate a horizontal feature vector and a vertical feature vector for each multi-modal data;

[0230] The distance between the horizontal feature vector and the vertical feature vector is calculated to generate a feature difference vector set for each multi-modal data;

[0231] A feature regression loss function is constructed according to the feature difference vector set, and the feature regression loss function contains the mean square error between the predicted feature of the trained machine learning model and the actual feature after the disturbance;

[0232] The feature regression loss function is subjected to a differentiable transformation to generate a loss value tensor required for gradient backpropagation;

[0233] The time series feature parameters of the trained machine learning model are traversed, and the partial derivative of the loss value tensor to each time series feature is calculated to generate a time series feature gradient matrix;

[0234] The momentum smoothing processing is performed on the gradient matrix of the time sequence feature to generate a parameter update direction vector after attenuation;

[0235] Based on the parameter update direction vector and the preset learning rate parameter, the time sequence feature parameters of the trained machine learning model are iteratively adjusted in increments;

[0236] In the incremental adjustment process, the change rate of the feature regression loss function is monitored in real time to generate a loss convergence state identifier;

[0237] When the loss convergence state identifier reaches a stable threshold, the gradient backpropagation update is terminated and the time sequence feature parameters are frozen;

[0238] The updated time sequence feature parameters are injected into the forward propagation path of the trained machine learning model, and the modal domain processing and feature fusion processing are re-executed to verify the feature extraction accuracy improvement result.

[0239] Optionally, the method further comprises:

[0240] A cross-scenario evaluation adapter is constructed, and the index semantic differences of different evaluation frameworks are parsed through the cross-scenario evaluation adapter;

[0241] Optionally, the cross-scenario evaluation adapter is constructed, and the index semantic differences of different evaluation frameworks are parsed through the cross-scenario evaluation adapter, comprising:

[0242] An evaluation framework semantic rule library is established to store the index mapping table and parameter transmission path of each scene;

[0243] The evaluation framework semantic rule library is established using a triple storage structure to organize data, and the entry directory of the rule library includes the source scene index name, the target scene corresponding index name, the conversion constraint condition, and the historical conversion success rate parameter.

[0244] The psychological state evaluation elements are subjected to abstract semantic tree parsing to generate intermediate representation layer data;

[0245] In a feasible implementation, the input elements are structured as hierarchical objects in JSON format, and the top-level field includes a state label sequence, a trend prediction parameter, and an abnormal context stack. The parser disassembles the objects layer by layer to decompose a composite state description (such as "anxiety with attention distraction") into atomic emotional labels ("anxiety") and cognitive labels ("attention distraction"); and the time series prediction value is disassembled into three atomic parameters of the base value, the change slope, and the confidence interval. The parsing process preserves the nesting relationship of the original data to generate intermediate representation layer data in the form of a multi-way tree, with leaf nodes storing atomic parameters and non-leaf nodes storing logical operation relationships.

[0246] The intermediate representation layer data traverses the query index mapping table to generate a scene-compatible index replacement scheme.

[0247] In a feasible implementation, the process of traversing the query index mapping table based on the intermediate representation layer data includes: starting from the leaf node and backtracking upwards, and retrieving a matching mapping entry in the semantic rule library. A double matching strategy is used for retrieval: first, the parameter name and data type are accurately matched, and if there is no result, a fuzzy matching algorithm is used to calculate the semantic similarity of the parameter description text. The node that matches successfully is replaced by the target scene index, for example, "class participation" in the education scene is mapped to "social interaction willingness value" in the clinical scene. For the case where there are multiple candidate mappings, a priority sorting mechanism is used to preferentially select the mapping scheme with a high historical conversion success rate, and the compatibility of the parameter value range is also checked.

[0248] The parameter transmission path with conflicts is subjected to dependency injection processing to generate an unambiguous index conversion result.

[0249] The parameter transmission path conflict resolution is performed after mapping. Nodes with type ambiguity in the parameter dependency graph are detected, for example, when two scenes define the "response delay" parameter as millisecond units and grade scores, respectively. The context metadata of the conflict nodes includes: data type declaration (float32 vs int16), life cycle label (session level vs task level), and call stack fingerprint (such as {func: analyze(), line: 203}). An interface proxy class name is generated based on the metadata, and the naming rule is [source scene]_[target scene]Proxy[serial number]. The proxy class inherits the original interface of the target scene and injects type coercion logic: float32 values are discretized into int16 grades according to a preset scale, and session-level parameters are converted to task-level parameters through time slicing.

[0250] The parameter transmission path with conflicts is subjected to dependency injection processing to generate an unambiguous index conversion result, including:

[0251] Identify conflict nodes with type ambiguity or scope overlap in the parameter transmission path, and generate a conflict path identifier set;

[0252] Extract the context metadata of each node in the conflict path identifier set, which includes parameter type declaration, life cycle label, and call stack fingerprint;

[0253] Generate an interface proxy class name corresponding to the conflict node according to the context metadata, and the interface proxy class name inherits the original interface of the target scene and injects type coercion constraints;

[0254] The interface proxy class name is parsed in the dependency chain of the target scene, and a dynamic binding strategy containing parameter scope isolation boundary is generated;

[0255] Based on the dynamic binding strategy, the conflict nodes in the parameter transmission path are mapped to scene-compatible interface method instances, and a dynamic binding queue of parameter instances is generated;

[0256] In a feasible implementation, the dependency chain of the proxy class is parsed using the dynamic binding strategy. The runtime environment of the target scene is scanned to build a dependency graph containing a complete call chain. A scope isolation layer is inserted into the graph: independent memory pages are allocated for source scene parameters, and address space offsets are calculated based on scene IDs; the dependency binding executes a lazy loading mechanism, which triggers parameter conversion only when the target scene actually calls the interface method. A breadth-first search algorithm is used to build the binding queue to ensure that parent and child nodes are initialized in the order of dependency.

[0257] The interface method instances in the dynamic binding queue are traversed to verify the compatibility status of the interface method instances with the target scene indicator semantics, and a parameter binding validity status code is generated;

[0258] The binding validity verification is dynamically performed during queue execution. A parameter syntax tree snapshot is generated at compile time, and the actual call tree is compared with the predefined template at runtime. The verification indicators include: parameter type matching state, scope boundary consistency, and life cycle constraint satisfaction. The parameter binding validity status code is output - a three-bit code representing the type / scope / lifetime verification result. For example, status code 0b110 indicates that the type and scope verification is passed but the lifetime is abnormal. Only when the status code is 0b111 is the binding considered successful.

[0259] According to the parameter binding validity status code, the interface method instances without type conflict are selected to generate scene-independent intermediate indicator sequences;

[0260] The intermediate indicator sequences are filled according to the semantic rules of the target scene to generate unambiguous indicator conversion results.

[0261] In an implementable embodiment, the template filling operation is performed based on target indicator conversion. The intermediate indicator sequence validated by effectiveness verification is reorganized according to the template specification of the target scenario; wherein the clinical scenario adopts the XML template of HL7 standard, and the education scenario adopts the JSONSchema of IMS global learning tools. The filling process injects scenario-specific evaluation optimization parameters, and the clinical scenario adds a sensitive patient marker field, and the education scenario supplements learning behavior analysis markers. The timestamp chain is embedded in the template header in the form of auxiliary fields, recording the key moments of state evolution. The exception context stack is stored in the template extension area after compression encoding, retaining the complete exception handling path fingerprint. The final output evaluation result description code is a binary encoded stream, including a protocol header (scenario identifier + version number), a payload (template content), and a checksum (CRC32 code). The entire conversion process is completed transparently at the output layer of the evaluation model, with an average adapter processing delay controlled within 20 milliseconds.

[0262] Converting the mental state assessment elements into the underlying assessment instructions supported by the target scenario;

[0263] Preserving the time sequence dependency relationship of state labels and exception handling context in the conversion process;

[0264] Injecting the evaluation optimization parameters matched by the target scenario, and generating the evaluation result description code that meets the cross-scenario execution conditions.

[0265] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device.

[0266] Although the embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent dynamic assessment of adolescent psychological state based on multimodal data, characterized in that, include: S1. Obtain multimodal psychological data of the target assessment object and extract dynamic features from the multimodal psychological data; wherein, the multimodal psychological data includes voice emotion information and facial expression behavior relationship; S2. The trained machine learning model is used to perform feature fusion processing on the multimodal psychological data to generate a multimodal feature set; wherein, the multimodal feature set includes a mapping relationship between modality type identifiers and temporal codes; S3. Based on the multimodal feature set, a state analysis network is used to perform temporal correlation analysis on the dynamic features to generate a feature fusion result that includes emotional dimension features and cognitive pattern features; S4. Input the feature fusion result into the evaluation model for dynamic state deduction processing, and output psychological state evaluation elements; wherein, the psychological state evaluation elements include state labels and trend prediction parameters generated based on feature evolution sequences.

2. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 1, characterized in that, S2 employs a trained machine learning model to perform feature fusion processing on the multimodal psychological data, generating a multimodal feature set, including: S21. Perform modality segmentation processing on the multimodal psychological data to generate multiple data domains and corresponding initial feature parameters; S22. Extract the temporal feature sequence of each data domain, and perform attention weighting processing on the temporal feature sequence to generate a weighted multi-domain feature vector; S23. Calculate the similarity between the multi-domain feature vector and the preset modality association threshold, and filter out candidate features that meet the confidence conditions. S24. Redundancy elimination processing is performed on the candidate features to generate a multimodal feature set containing modality type identifiers. Each multimodal feature contains a normalized temporal code relative to the multimodal psychological data. Specifically, based on the multimodal feature set, a state analysis network is used to perform temporal correlation analysis on the dynamic features, generating a feature fusion result that includes emotional dimension features and cognitive pattern features, including: Based on the normalized temporal coding in the multimodal feature set, the multimodal psychological data is subjected to time window truncation processing to generate multiple feature time segments; Each feature time segment is standardized and feature-enhanced to generate optimized state analysis input data. The recurrent memory subnet in the state analysis network is used to extract sequence features from the state analysis input data to generate an emotion dimension feature vector. In parallel, a pattern classification model is used to perform feature pattern matching on the feature time segments to generate a cognitive pattern probability distribution. The emotion dimension feature vector and the cognitive pattern probability distribution are concatenated to generate a multimodal feature fusion result.

3. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 1, characterized in that, The S4 step involves inputting the feature fusion result into the evaluation model for dynamic state deduction processing, and outputting psychological state evaluation elements, including: S41. Analyze the contextual relationships of the current evaluation scenario and generate state priority weights associated with the multimodal feature fusion results; S42. Generate a state transition matrix based on the state priority weights, and traverse and sort the modality type identifiers in the multimodal feature set; S43. Using a dynamic programming strategy, select the optimal evolution sequence from the state transition matrix to generate a basic evaluation set containing emotion labels, cognitive states, and trend predictions; S44. Logically integrate the basic evaluation set and trend prediction parameters to generate a state description segment that conforms to the target evaluation framework.

4. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 3, characterized in that, The method further includes: During the evaluation process, status response data is captured in real time to generate feedback logs containing feature offsets and status change information. Extract the abnormal state features from the feedback log, and perform similarity matching between the abnormal state features and the historical evaluation case library to generate adaptive adjustment instructions; The modality association threshold parameters of the machine learning model are dynamically updated based on the adaptive adjustment instructions. The updated modal association threshold parameters are injected into the evaluation model, and the state priority weights in the state transition matrix are recalculated.

5. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 4, characterized in that, The step of extracting abnormal state features from the feedback log and performing similarity matching between the abnormal state features and the historical evaluation case library to generate adaptive adjustment instructions includes: The abnormal state features are divided into time windows to generate multiple state segments and their corresponding multimodal data sequences; The trained anomaly classification model is used to perform root cause analysis on each state segment, generating classification labels that include feature recognition bias, temporal drift, and response delay. Retrieve solution templates that match the classification labels from the historical evaluation case library to generate a set of candidate adjustment strategies; Based on the matching degree ranking between the multimodal data sequence and the candidate adjustment strategy set, the strategy with the highest confidence is selected to generate an adaptive adjustment instruction.

6. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 5, characterized in that, The trained anomaly classification model is used to perform root cause analysis on each state segment, generating classification labels that include feature recognition bias, temporal drift, and response delay, including: The state segments are processed by time series segmentation to generate multimodal data sequence slices containing start and end times; The multimodal data sequence slices are subjected to keyframe sampling processing to generate a set of state change keyframes and corresponding timestamp indices; Extract the feature offset between adjacent keyframes in the set of keyframes for state changes, and generate feature evolution trajectory vectors and time interval sequences; The feature evolution trajectory vector is processed by trend calculation to generate an abnormal evolution pattern feature vector; The feature vector of the abnormal evolution pattern is input into the spatiotemporal convolutional subnet of the trained abnormal classification model for local feature extraction, generating a spatial abnormal activation map. The time interval sequence is subjected to sliding window mean filtering to generate a smoothed response delay time sequence; The response delay time series is input into the gated recurrent subnet of the trained anomaly classification model for periodic pattern matching to generate a time dimension anomaly score; the spatial anomaly activation map is subjected to channel max pooling to generate a spatial dimension anomaly score; the spatial dimension anomaly score and the time dimension anomaly score are subjected to feature cross-fusion processing to generate a spatiotemporal joint anomaly probability distribution. Peak detection results of feature recognition deviation probability, temporal drift probability, and response delay probability are extracted from the spatiotemporal joint anomaly probability distribution; dynamic threshold comparison processing is performed on the preset probability threshold based on the peak detection results to generate a candidate set of classification labels containing probability ranking; The candidate classification labels are subjected to timestamp index alignment verification to generate target classification labels that are consistent with the phase of the abnormal state in the multimodal data sequence slice; the target classification labels and the feature evolution trajectory vector are subjected to anomaly type reverse verification to generate a final classification label set containing confidence weights.

7. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 1, characterized in that, The method further includes: Virtual feature perturbation parameters are injected before the evaluation model is executed. These virtual feature perturbation parameters are used to simulate the random offset scenario of multimodal data features. Monitor the fault-tolerant processing results of the evaluation model on the perturbed features and generate stability evaluation indicators; When the stability evaluation index is lower than a preset threshold, the online learning mode of the completed machine learning model is triggered. The temporal feature parameters of the trained machine learning model are updated by gradient backpropagation based on the feature difference data before and after the perturbation. The evaluation model monitors its fault-tolerant processing results for the perturbed features and generates stability evaluation indices, including: The number of evaluation failures and the anomaly capture success rate caused by feature errors during the statistical evaluation process are recorded. Calculate the ratio of the number of evaluation failures to the total number of evaluation steps to generate a first evaluation coefficient; Extract the proportion of events that match the preset fault tolerance rules from the anomaly capture success rate, and generate a second evaluation coefficient; The first evaluation coefficient and the second evaluation coefficient are weighted and summed to generate a comprehensive stability evaluation index.

8. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 7, characterized in that, The step of updating the temporal feature parameters of the trained machine learning model through gradient backpropagation based on the feature difference data before and after the perturbation includes: The feature difference data before and after the perturbation are time-stamp aligned to generate a matching pair sequence containing the feature set before the perturbation and the feature set after the perturbation; Extract the feature offsets from the matching pair sequence to generate the horizontal and vertical feature vectors for each multimodal data. The distance between the horizontal feature vector and the vertical feature vector is calculated to generate a set of feature difference vectors for each multimodal data. A feature regression loss function is constructed based on the set of feature difference vectors. The feature regression loss function includes the mean square error between the features predicted by the trained machine learning model and the actual features after perturbation. The feature regression loss function is subjected to a differentiable transformation to generate the loss value tensor required for gradient backpropagation; Iterate through the temporal feature parameters of the trained machine learning model, calculate the partial derivative of the loss tensor with respect to each temporal feature, and generate the temporal feature gradient matrix. The temporal feature gradient matrix is ​​subjected to momentum smoothing to generate a decayed parameter update direction vector; Based on the updated direction vector and the preset learning rate parameter, the temporal feature parameters of the trained machine learning model are iteratively and incrementally adjusted. During the incremental adjustment process, the rate of change of the feature regression loss function is monitored in real time to generate a loss convergence status indicator. When the loss convergence state indicator reaches a stable threshold, the gradient backpropagation update is terminated and the temporal feature parameters are frozen. The updated temporal feature parameters are injected into the forward propagation path of the trained machine learning model, and the modality segmentation and feature fusion processes are re-executed to verify the improved feature extraction accuracy.

9. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 1, characterized in that, The method further includes: Construct a cross-scenario evaluation adapter, and use the cross-scenario evaluation adapter to parse the semantic differences of metrics in different evaluation frameworks; The psychological state assessment elements are converted into underlying assessment instructions supported by the target scenario; The temporal dependencies and exception handling context of the state labels are preserved during the transformation process; Inject the evaluation optimization parameters that match the target scenario to generate an evaluation result descriptor that meets the cross-scenario execution conditions.

10. The intelligent dynamic assessment method for adolescent psychological state based on multimodal data as described in claim 9, characterized in that, The construction of a cross-scenario evaluation adapter, and the parsing of semantic differences in metrics across different evaluation frameworks using the cross-scenario evaluation adapter, includes: Establish a semantic rule base for the evaluation framework, and store the indicator mapping tables and parameter transmission paths for each scenario; Abstract semantic tree parsing is performed on the psychological state assessment elements to generate intermediate representation layer data; Based on the intermediate representation layer data, the indicator mapping table is traversed and queried to generate a scenario-compatible indicator replacement scheme; Dependency injection is performed on conflicting parameter passing paths to generate unambiguous index transformation results; The process of performing dependency injection on conflicting parameter passing paths to generate unambiguous index transformation results includes: Identify conflicting nodes in the parameter passing path that have type ambiguity or overlapping scope, and generate a set of conflicting path identifiers; Extract the context metadata of each node in the set of conflict path identifiers. The context metadata includes parameter type declarations, lifecycle markers, and call stack fingerprints. Based on the context metadata, an interface proxy class name corresponding to the conflict node is generated. The interface proxy class name inherits the native interface of the target scenario and injects type casting constraints. Parse the dependency chain of the interface proxy class name in the target scenario and generate a dynamic binding strategy that includes parameter scope isolation boundaries; Based on the dynamic binding strategy, conflicting nodes in the parameter passing path are mapped to scenario-compatible interface method instances, generating a dynamic binding queue of parameter instances. Traverse the interface method instances in the dynamic binding queue, verify the compatibility status of the interface method instances with the target scenario indicator semantics, and generate parameter binding validity status codes; Based on the parameter binding validity status code, filter out interface method instances without type conflicts and generate a scenario-independent intermediate indicator sequence; The intermediate indicator sequence is filled into the indicator template according to the semantic rules of the target scenario to generate an unambiguous indicator conversion result.

Images