A Real-Time Anxiety State Assessment Method Based on Multimodal Fusion

Abstract

This invention discloses a real-time anxiety state assessment method based on multimodal fusion, comprising the following steps: S1 Collecting multi-turn dialogues and labeling the initial anxiety emotion, action details, and termination anxiety emotion to form triples; S2 Filling the triples according to a template to generate inference input; S3 Matching keywords with a detail vocabulary and extracting the left and right five words to obtain subsequences; S4 Performing word vector, position, and dependency fusion on the subsequences to obtain detail vectors; S5 Feeding the input and detail vectors into a detail-gated HyperCLOVA, activating expert paths according to weights to complete inference and recording the paths; S6 Summarizing the paths into a matrix and calculating the path consistency score; S7 Comparing the inference text with the semantics of the termination anxiety emotion, fusing it with the consistency score, and outputting the assessment result. This invention improves the accuracy and path stability of anxiety emotion inference and enhances the quality of cross-domain dialogue services.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for real-time assessment of anxiety state based on multimodal fusion. Background Technology

[0002] In recent years, clinical and mental health scenarios have required the real-time identification and quantification of anxiety states during calls / conversations to support remote consultations, online counseling, educational interventions, and customer service risk control.

[0003] However, existing methods mostly rely on scale-based offline assessments or single-modal anxiety classification, making it difficult to capture short-term fluctuations and fine-grained anxiety symptoms in multi-round interactions. Furthermore, they suffer from high assessment latency, weak interpretability, and insufficient adaptation to individual differences. While some generative inference methods based on large models can output anxiety-related text, they generally suffer from: coarse-constructed inference inputs lacking structured modeling of anxiety triggers and behavioral details; the model itself is a black box, lacking measurement of expert path selection and stability; and the evaluation metrics are singular, failing to jointly characterize the generated semantics and the stability of the inference path. Summary of the Invention

[0004] One objective of this invention is to propose a real-time assessment method for anxiety states based on multimodal fusion. This invention integrates detail recognition, gating activation, and path consistency analysis mechanisms to construct standardized input text and guide the model to perform anxiety emotion evolution reasoning. It has the advantages of rich assessment dimensions, interpretable reasoning process, and strong adaptability.

[0005] A real-time anxiety state assessment method based on multimodal fusion according to an embodiment of the present invention includes the following steps:

[0006] S1. Collect text data containing several rounds of dialogue and construct a set of anxiety emotion evolution triplets;

[0007] S2. Convert the anxiety emotion evolution triplet into natural language prompt text to generate standardized reasoning task input text;

[0008] S3. Perform detail word recognition operation on each task input text, use the behavior detail word list to match keywords, combine the matched keywords and five words above and below to form a subsequence, and generate a set of detail description subsequences.

[0009] S4. Encode the detailed description subsequences with word vectors, and fuse positional embeddings and syntactic dependency relations to generate detailed description vectors;

[0010] S5. Input the task text and detail description vector into the improved HyperCLOVA model containing a detail-aware gating mechanism. In each Transformer layer, gating weights are generated based on the detail description vector to control the activation path of the expert subnetwork, perform anxiety emotion evolution reasoning operations, record the activated expert path index, and output the anxiety emotion reasoning text set.

[0011] S6. Record the expert path index corresponding to each round of reasoning, calculate the path consistency score of the input text of the reasoning task, and generate a path record matrix.

[0012] S7. Perform semantic comparison between the anxiety inference text set and the anxiety termination state in the anxiety evolution triplet, generate an evaluation vector by combining the path consistency score, and output the evaluation result of anxiety evolution reasoning ability and anxiety level label.

[0013] Optionally, the construction of the anxiety evolution triple set in step S1 includes: labeling the anxiety start state, descriptive behavior phrases and anxiety termination state in the text data of each dialogue, and jointly constructing the anxiety evolution triple set.

[0014] Optionally, the process of converting the anxiety evolution triplet into natural language prompt text in step S2 includes: constructing a rule set based on a preset task prompt template; filling the anxiety start state text, descriptive behavior phrase text, and anxiety end state text in each triplet into the corresponding template placeholder positions; generating prompt text that matches the task type; and constructing standardized inference task input text. The task prompt template and each template text in the rule set contain an anxiety start state placeholder, a descriptive behavior phrase placeholder, and an anxiety end state placeholder.

[0015] Optionally, the detail word identification and keyword matching in step S3 includes the following steps:

[0016] Construct a behavioral detail vocabulary, and divide the vocabulary into sets of entries according to action, gaze, breathing, tone and posture. Establish a synonym mapping rule for each entry, map different word forms with the same or similar meanings to a unified standard entry number, and establish a word form normalization rule to restore different parts of speech, tenses or voice changes to standard word forms.

[0017] The task input text is processed by sentence segmentation and word segmentation, preserving the original word order and punctuation position information. Each word segment is labeled with part-of-speech tags based on natural language processing tools, and the subject-predicate, verb-object, and modifier relationships between word segments are determined based on dependency parsing.

[0018] In the word segmentation results, the behavior detail vocabulary is called to perform matching, and the longest priority matching strategy is adopted: when two or more words have matching conflicts in the same position, the word with the longest length is selected first; if the lengths are the same, the word with the highest frequency in the vocabulary is selected; if the length and frequency are the same, the word with the earliest position in the sentence is selected first.

[0019] The initial matching results are semantically corrected: negative words before the matching words are detected, and if the matching words are completely within the scope of the negative words, the matching is removed; degree adverbs before the matching words are detected, and intensity labels are attached to the matching words, with the intensity coefficient recorded in the labels;

[0020] Centered on the word order position of each retained matching word in the sentence, five candidate subsequences are extracted to the left and right of each of the five adjacent non-punctuation words in the same sentence. When encountering the beginning or end of the sentence, the subsequences are supplemented with the actual extractable words, excluding punctuation marks.

[0021] The candidate subsequences are subjected to overlap merging and deduplication processing to ensure that only one continuous subsequence is retained within the same range. The final detailed description subsequence set is generated based on word order continuity and minimum length constraints. A one-to-one index relationship between each subsequence in the set and the corresponding matching word is established.

[0022] Optionally, step S4 specifically includes:

[0023] For each word in the detailed description subsequence, perform an index lookup operation, call the word vector table that matches the language model used to generate the natural language prompt text, retrieve the fixed-length word vector corresponding to the word, and combine them into a word vector matrix according to the order of the words in the detailed description subsequence;

[0024] Assign an integer number starting from zero to each word in the detailed description subsequence, call the preset position encoding function, calculate the corresponding position embedding vector according to the number, perform a dimension-wise addition between the position embedding vector and the current word vector in the same dimension, replace the original word vector, and generate a representation matrix with fused position embeddings;

[0025] The analysis results obtained by performing dependency parsing on the original sentence are called, and the connection information between each word and the dependency headword is found in the detailed description subsequence. The relative index position of the current word and the dependency headword and the corresponding dependency label are extracted.

[0026] Based on the dependency tag lookup, the preset tag weights are obtained. The vector of the center word is multiplied with the tag weights in a dimension-wise manner, and then added with the vector of the current word in a dimension-wise manner. The weighted superposition result is used to replace the current word vector, thus completing the structural information fusion processing based on dependency relationship.

[0027] One-dimensional average pooling and max pooling operations are performed on the word vectors after structural information fusion processing along the word order of the detail description subsequence, respectively. The two pooling results are then concatenated according to the dimension to construct the detail description vector.

[0028] Optionally, the improved HyperCLOVA model including the detail-aware gating mechanism in step S5 has the following structure and operation process:

[0029] In each layer of the Transformer structure, several expert subnetworks are set up. All expert subnetworks do not share parameters and remain independent.

[0030] The detailed description vector is input into the gate weight generation unit, and a linear transformation and activation function operation are performed to generate a gate control vector equal to the number of expert subnetworks.

[0031] Perform dot product calculation on the gating control vector and the weight vector corresponding to the expert subnetwork, and output the nonnormalized response score for each expert.

[0032] Normalize all expert response scores to form an expert selection probability distribution;

[0033] In each layer, the expert subnetwork with the highest response value is selected according to the expert selection probability distribution. The output of the selected expert subnetwork is weighted and summed according to the corresponding weight in the probability distribution, and then replaced with the original feedforward network output of this layer.

[0034] During the model inference process, the number sequence and corresponding weight value of each activated expert subnetwork are recorded to form an expert path index.

[0035] Optionally, the generation of the path record matrix in step S6 specifically includes:

[0036] In each round of anxiety evolution inference, the activated expert subnetwork numbers and corresponding gating weights of each layer in the improved HyperCLOVA model are recorded. Expert path vectors are constructed in hierarchical order. The expert path vectors corresponding to all inference task input texts are arranged in order to form a path record matrix. A path consistency score is calculated for each path vector. Specifically, in the anxiety evolution triplet type set with the same source as the inference task input text, the mean similarity between the expert path vector and other path vectors is calculated based on normalized cosine similarity. The obtained mean similarity is used as the path consistency score and appended to the path record of the corresponding text to form a path record matrix containing the expert path number sequence and the path consistency score.

[0037] Optionally, the process of generating the evaluation vector in step S7 includes: for each text in the anxiety inference text set, calling the anxiety termination state text in the corresponding anxiety evolution triplet, inputting it into the semantic encoding model consistent with the model's pre-training corpus, obtaining the anxiety inference text vector and the anxiety termination state vector, and calculating the semantic similarity score between the anxiety inference text vector and the anxiety termination state vector; weighting and fusing the semantic similarity score with the path consistency score generated by the corresponding task input text according to a fixed ratio to generate an evaluation value reflecting semantic accuracy and inference path stability; using the evaluation value as the main component of the evaluation vector, combining it with the task input text identifier, the original semantic similarity score, the path consistency score, and the fusion ratio parameter to form the evaluation vector.

[0038] Optionally, the output of the anxiety evolution reasoning ability assessment results includes a partitioning strategy based on the fusion assessment score, generating structured result sets respectively, specifically including:

[0039] When the semantic similarity score is higher than the first threshold and the path consistency score is higher than the second threshold, it is marked as a high consistency and high matching class, and the comprehensive score label is directly output as "inference accurate-stable".

[0040] When the semantic similarity score is higher than the first threshold and the path consistency score is lower than the second threshold, it is marked as semantically reliable but path unstable, and the output comprehensive score label is "reasoning reliable - path scattered", with an additional path perturbation warning label;

[0041] When the semantic similarity score is lower than the first threshold and the path consistency score is higher than the second threshold, it is marked as a path-stable but semantically deviated class, and the output comprehensive score label is "reasoning deviation - path stability" with additional semantic drift hints;

[0042] When both the semantic similarity score and the path consistency score are below the corresponding threshold, they are marked as inference failures, and the comprehensive score label is "inference failure - needs reconstruction", which is recorded as a model fine-tuning re-examination sample.

[0043] All sample evaluation results are summarized to generate a structured evaluation report that includes inference type distribution, path activation mode statistics, and an index of inference failure samples.

[0044] Optionally, the semantic similarity score and the path consistency score are weighted and fused according to a weighted fusion rule to obtain a fusion score;

[0045] Anxiety level labels are output based on the range of the fusion score. These labels are used to identify the current anxiety state of the evaluated individual. The output rules for the anxiety level labels are as follows:

[0046] When the fusion score is less than or equal to the first preset threshold, a low anxiety label is output.

[0047] When the fusion score is greater than the first preset threshold and less than or equal to the second preset threshold, a moderate anxiety label is output.

[0048] When the fusion score is greater than the second preset threshold, a high anxiety label is output.

[0049] The beneficial effects of this invention are:

[0050] (1) Enhance the comprehensiveness of evaluation dimensions: This invention constructs an anxiety emotion evolution triplet and generates standardized reasoning input text based on task templates. It extracts contextual behavioral clues through detail word recognition and encoding mechanisms, completes the semantics of the large language model input, enhances the model's ability to understand micro-behavioral changes in emotion reasoning tasks, and realizes multi-dimensional semantic level evaluation expression.

[0051] (2) Enhance the interpretability of the model reasoning path: By constructing a detail-aware gating mechanism to control the activation path of the expert sub-network, the path index and response weight of each round of reasoning are recorded, and the path consistency score is further calculated based on normalized cosine similarity, so as to introduce a structured visual analysis basis for the reasoning process and make up for the deficiency of invisible path in the evaluation of traditional black box models.

[0052] (3) Achieve refined classification and strategic output of evaluation results: integrate semantic similarity score and path consistency score to generate evaluation vector, set inference quality labels and optimization suggestions based on multiple threshold intervals, distinguish between high consistency-high matching, reliable-unstable, bias-stable and inference failure, and achieve fine-grained classification and judgment of model capabilities and guidance for failure sample reconstruction. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a flowchart of a real-time anxiety state assessment method based on multimodal fusion proposed in this invention;

[0055] Figure 2 This is a flowchart illustrating the structure of a real-time anxiety state assessment method based on multimodal fusion proposed in this invention, including detail word recognition, detail descriptor sequence construction, and detail descriptor vector generation. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0057] refer to Figure 1 and Figure 2 A real-time anxiety state assessment method based on multimodal fusion includes the following steps:

[0058] S1. Collect text data containing several rounds of dialogue and construct a set of anxiety emotion evolution triplets;

[0059] S2. Convert the anxiety emotion evolution triplet into natural language prompt text to generate standardized reasoning task input text;

[0060] S3. Perform detail word recognition operation on each task input text, use the behavior detail word list to match keywords, combine the matched keywords and five words above and below to form a subsequence, and generate a set of detail description subsequences.

[0061] S4. Encode the detailed description subsequences with word vectors, and fuse positional embeddings and syntactic dependency relations to generate detailed description vectors;

[0062] S5. Input the task text and detail description vector into the improved HyperCLOVA model containing a detail-aware gating mechanism. In each Transformer layer, gating weights are generated based on the detail description vector to control the activation path of the expert subnetwork, perform anxiety emotion evolution reasoning operations, record the activated expert path index, and output the anxiety emotion reasoning text set.

[0063] S6. Record the expert path index corresponding to each round of reasoning, calculate the path consistency score of the input text of the reasoning task, and generate a path record matrix.

[0064] S7. Perform semantic comparison between the anxiety inference text set and the anxiety termination state in the anxiety evolution triplet, generate an evaluation vector by combining the path consistency score, and output the evaluation result of anxiety evolution reasoning ability and anxiety level label.

[0065] In this embodiment, the construction of the anxiety emotion evolution triple set in step S1 includes: labeling the anxiety emotion start state, descriptive behavior phrases and anxiety emotion termination state in the text data of each dialogue, and jointly constructing the anxiety emotion evolution triple set.

[0066] In this embodiment, the process of converting the anxiety evolution triplet into natural language prompt text in step S2 includes: constructing a rule set based on a preset task prompt template; filling the anxiety start state text, descriptive behavior phrase text, and anxiety termination state text in each triplet into the corresponding template placeholder positions; generating prompt text that matches the task type; and constructing standardized reasoning task input text. The task prompt template and each template text in the rule set contain an anxiety start state placeholder, a descriptive behavior phrase placeholder, and an anxiety termination state placeholder.

[0067] In this embodiment, the detail word identification and keyword matching in step S3 includes the following steps:

[0068] Construct a behavioral detail vocabulary, and divide the vocabulary into sets of entries according to action, gaze, breathing, tone and posture. Establish a synonym mapping rule for each entry, and map different word forms with the same or similar meanings to a unified standard entry number. Establish a word form normalization rule to restore different parts of speech, tenses or voice changes to standard word forms, and ensure that matching is not affected by morphological changes.

[0069] The extended description of the synonym mapping rule is as follows:

[0070] Synonym mapping rules are used to uniformly map terms with similar semantics but different word forms in actual expression to a standardized term number, thereby improving semantic consistency and matching accuracy in the detailed word recognition process. Synonym mapping rules are constructed based on the following structured operation process:

[0071] First, a multi-source thesaurus and the word embedding space of the language model are invoked to identify the semantic neighborhood word set for each standard term. Based on the cosine similarity between word vectors, a similarity threshold is set to retain candidate word groups semantically close to the standard term. Second, lexical normalization is performed on the candidate word groups, unifying them to their original forms to avoid changes in tense, voice, or number affecting semantic equivalence judgment. Then, a one-to-one mapping relationship is constructed between the standard term and its semantically similar words, forming a mapping dictionary structure. The key of the mapping dictionary is the unified standard term number, and the value is the set of all synonyms that can be assigned to that number.

[0072] During operation, when words in the input text do not completely match the entries in the behavior detail vocabulary, the system will call the synonym mapping rule and search for matching items in the synonym set based on the normalized word form. If there are multiple candidate matching items, the optimal match will be selected based on the word vector similarity ranking to ensure that all semantically equivalent entries can be merged into the standard word number under the same recognition path, thereby improving the ability to recognize detail consistency under diverse expressions.

[0073] The task input text is processed into sentence segmentation and word segmentation, preserving the original word order and punctuation position information. Each word segment is labeled with part-of-speech tags based on natural language processing tools, and the subject-predicate, verb-object, and modification relationships between word segments are determined based on dependency parsing. In particular, a structured natural language processing toolchain is used in the process of sentence segmentation and word segmentation of the task input text to ensure the accuracy and consistency of subsequent detail word recognition and grammatical structure analysis.

[0074] First, in the text preprocessing stage, the open-source natural language processing toolkit LTP for Chinese or tools optimized by deep learning, such as HanLP, are called to perform high-precision sentence segmentation and word segmentation. Sentence segmentation is based on rule bases and statistical models to identify punctuation marks, such as periods, commas, and semicolons, as well as semantic pauses, and generate the smallest semantic segments. The word segmentation module combines dictionary matching and word boundary probability models to segment continuous characters in the text, identify complete word units, and retain the original word order index and punctuation position information in the sentence for each word to support accurate range extraction based on context in subsequent subsequence construction.

[0075] After word segmentation, part-of-speech tagging is performed on each word. The part-of-speech tagging module adopts a bidirectional LSTM-CRF sequence tagging model, which combines contextual dependencies to predict the part-of-speech tag of each word (such as noun, verb, adjective, adverb, etc.), providing a grammatical basis for syntactic structure recognition.

[0076] Subsequently, based on the segmented words and part-of-speech tagging, dependency parsing is performed. The dependency parsing module uses a high-performance dependency parser based on graph neural networks or transition prediction structures, such as the LTP dependency module or BiaffineParser, to model the syntactic dependency relationships between each word and other words in the sentence. The system generates a dependency structure graph, where each node corresponds to a word, edges represent dependency relationships, and each edge is labeled with the dependency type, such as subject-verb relationship (SBV), verb-object relationship (VOB), attributive-head relationship (ATT), etc., as well as the dependency direction, clarifying the subordinate relationship between the current word and its head word. This dependency structure graph will serve as one of the core input data for subsequent structural information fusion and subsequence semantic range correction operations, providing technical support for accurately understanding the grammatical hierarchy and the interaction relationships between words.

[0077] In the word segmentation results, the behavior detail vocabulary is called to perform matching, and the longest priority matching strategy is adopted: when two or more words have matching conflicts in the same position, the word with the longest length is selected first; if the lengths are the same, the word with the highest frequency in the vocabulary is selected; if the length and frequency are the same, the word with the earliest position in the sentence is selected first.

[0078] Perform semantic range correction on the initial matching results:

[0079] Detect negative words (such as "not" or "no") before the matching word. If the matching word is completely within the scope of the negative word, the match is removed. Detect degree adverbs (such as "slightly" or "extremely") before the matching word, attach an intensity label to the matching word, and record the intensity coefficient in the label.

[0080] Centered on the word order position of each retained matching word in the sentence, five candidate subsequences are extracted to the left and right of each of the five adjacent non-punctuation words in the same sentence. When encountering the beginning or end of the sentence, the subsequences are supplemented with the actual extractable words, excluding punctuation marks.

[0081] The candidate subsequences undergo overlap merging and deduplication processing to ensure that only one continuous subsequence is retained within the same range. Based on word order continuity and minimum length constraints, a final set of detailed description subsequences is generated. A one-to-one index relationship is established between each subsequence in the set and its corresponding matching word. Specifically, for the initial results after matching behavioral detail keywords, to ensure semantic accuracy and contextual consistency, semantic range correction and contextual subsequence extraction operations are performed on the matching results. These steps include the following:

[0082] First, the system performs a negation word detection operation on the position of each matching word in the sentence. The system calls a negation word dictionary (containing negative adverbs or verbs such as "not", "without", "never", and "without") and combines it with the dependency parsing results to identify whether there is a direct or indirect dependency connection between the negation word and the matching word. If there is a negation word that directly affects the matching word semantically (for example, forming a combination structure such as "without bowing his head" or "smiling uneasily"), and the matching word is within the scope of this negation, then the matching is considered to lack the effect of expressing a real emotional action, and the system will automatically remove the matching word and its related subsequences from the subsequent analysis.

[0083] Secondly, the system performs degree adverb recognition within the range of three words to the left of each retained matching word. The system uses a pre-built degree adverb table (such as "slightly", "very", "extremely", "intense", etc.) to detect whether there are degree words modifying the matching word. Based on their semantic strength, the system sets the corresponding strength label. For example, "slightly nod" will be marked as a low-intensity action with a corresponding value of 0.3; "extremely angry" will be marked as a high-intensity action with a corresponding value of 0.9. This strength label will serve as an additional feature dimension for the matching word in the process of generating detailed description vectors and participate in the gating mechanism modeling.

[0084] Subsequently, the system uses the word order number of each retained matching word in the original sentence as the center to perform candidate context subsequence extraction. Centered on the matching word, it extracts up to five adjacent non-punctuation words to the left and right to form candidate context regions. The system ensures that the extraction process is limited to the sentence containing the matching word, avoiding cross-sentence interference. If there are fewer than five words due to the beginning or end of a sentence, the extraction is padded to the actual extractable range. The extraction result will form a preliminary subsequence set with a length not exceeding 11 words and containing the center word.

[0085] To avoid redundant overlap in context subsequences corresponding to different matched words, the system performs an overlap merging operation on all extracted subsequences. When two subsequences have overlapping portions of three or more consecutive words, and the overlap does not cross a period (full stop) punctuation mark, they are merged into an extended subsequence. The merged subsequence set is then deduplicated, retaining only unique subsequence instances.

[0086] Finally, the system checks whether each subsequence meets the minimum length requirement (e.g., no less than 5 words) based on word order continuity. Subsequences that do not meet the structural integrity requirement are removed. For each retained subsequence, the system establishes a one-to-one mapping relationship with the original matching words. This processing flow can improve the semantic accuracy and contextual integrity of behavioral detail extraction and enhance the ability to model emotional action details within sentences.

[0087] In this embodiment, step S4 specifically includes:

[0088] For each word in the detailed description subsequence, perform an index lookup operation, call the word vector table that matches the language model used to generate the natural language prompt text, retrieve the fixed-length word vector corresponding to the word, and combine them into a word vector matrix according to the order of the words in the detailed description subsequence;

[0089] Each word in the detailed description subsequence is assigned an integer number starting from zero and incrementing. A preset positional encoding function is called to calculate the corresponding positional embedding vector based on the number. The positional embedding vector is then added dimension-wise with the current word vector along the same dimension, replacing the original word vector, and generating a representation matrix with fused positional embeddings. The "preset positional encoding function" is used to convert the word order number of each word in the detailed description subsequence into a positional information representation consistent with the word vector dimension, thereby enhancing the model's ability to perceive word order structure. The preset positional encoding function adopts an absolute positional encoding method based on sine and cosine transforms, ensuring that a unique positional embedding vector with sequence pattern awareness is generated for each word without parameter training. The specific operation process is as follows:

[0090] Let the word order number of each word in the subsequence be . Starting from 0 and increasing sequentially; let the dimension of each word vector be... For dimension indexes as The location embedding vector for each dimension is calculated as follows:

[0091] In the even-numbered index dimension (i.e.) On the ), the position code value is:

[0092] ;

[0093] In odd index dimension (i.e.) On the ), the position code value is:

[0094] ;

[0095] The aforementioned positional encoding function, without introducing learnable parameters, maps positional information to the same vector space as word vectors through sine and cosine periodic functions, achieving the goal of having unique encoded representations for different words at different positions. This can be efficiently utilized by the model to capture the sequential structural differences in the sequence.

[0096] After calculating the position embedding vector of each word, the system performs element-wise addition of the position embedding vector and the static word vector corresponding to the current word along the dimension. The result is used as the final fusion representation vector of the word, which is used in the subsequent syntactic information fusion and vector aggregation process. This method can improve the model's ability to model position patterns without relying on external training and enhance the positioning accuracy of detail description vectors in inference path control.

[0097] The analysis results obtained by performing dependency parsing on the original sentence are called, and the connection information between each word and the dependency headword is found in the detailed description subsequence. The relative index position of the current word and the dependency headword and the corresponding dependency label are extracted.

[0098] Based on the dependency tag lookup, the preset tag weights are obtained. The vector of the center word is multiplied with the tag weights in a dimension-wise manner, and then added with the vector of the current word in a dimension-wise manner. The weighted superposition result is used to replace the current word vector, thus completing the structural information fusion processing based on dependency relationship.

[0099] One-dimensional average pooling and max pooling operations are performed on the word vectors after structural information fusion processing along the word order of the detail description subsequence, respectively. The two pooling results are then concatenated according to the dimension to construct the detail description vector.

[0100] In this embodiment, the improved HyperCLOVA model including the detail-aware gating mechanism in step S5 has the following structure and operation process:

[0101] In each layer of the Transformer structure, several expert subnetworks are set up. All expert subnetworks do not share parameters and remain independent.

[0102] The detail description vector is input into the gate weight generation unit, and a linear transformation and activation function operation are performed to generate a gate control vector equal to the number of expert subnetworks. The process of performing the linear transformation and activation function operation includes: first, inputting the detail description vector into a linear mapping module, converting the detail description vector into an intermediate representation vector through preset weight parameters and bias parameters, and then applying a nonlinear activation function, such as GELU or ReLU, to the intermediate representation to enhance its expressive power. The output result is used as a gate control vector matching the number of expert subnetworks to guide the selection of expert paths.

[0103] Perform dot product calculation on the gating control vector and the weight vector corresponding to the expert subnetwork, and output the nonnormalized response score for each expert.

[0104] Normalize all expert response scores to form an expert selection probability distribution;

[0105] In each layer, the expert subnetwork with the highest response value is selected according to the expert selection probability distribution. The output of the selected expert subnetwork is weighted and summed according to the corresponding weight in the probability distribution, and then replaced with the original feedforward network output of this layer.

[0106] During the model inference process, the number sequence and corresponding weight value of each activated expert subnetwork are recorded to form an expert path index.

[0107] In this embodiment, the generation of the path record matrix in step S6 specifically includes:

[0108] In each round of anxiety evolution inference, the activated expert subnetwork numbers and corresponding gating weights of each layer in the improved HyperCLOVA model are recorded. Expert path vectors are constructed in hierarchical order. The expert path vectors corresponding to all inference task input texts are arranged in order to form a path record matrix. A path consistency score is calculated for each path vector. Specifically, in the anxiety evolution triplet type set with the same source as the inference task input text, the mean similarity between the expert path vector and other path vectors is calculated based on normalized cosine similarity. The obtained mean similarity is used as the path consistency score and appended to the path record of the corresponding text to form a path record matrix containing the expert path number sequence and the path consistency score.

[0109] In this embodiment, the process of generating the evaluation vector in step S7 includes: for each text in the anxiety inference text set, calling the anxiety termination state text in the corresponding anxiety evolution triplet, inputting it into the semantic encoding model consistent with the model's pre-training corpus, obtaining the anxiety inference text vector and the anxiety termination state vector, and calculating the semantic similarity score between the anxiety inference text vector and the anxiety termination state vector; weighting and fusing the semantic similarity score with the path consistency score generated by the corresponding task input text according to a fixed ratio to generate an evaluation value reflecting semantic accuracy and inference path stability; using the evaluation value as the main component of the evaluation vector, combining the task input text identifier, the original semantic similarity score, the path consistency score, and the fusion ratio parameter to form the evaluation vector. The process of calculating the semantic similarity score between the anxiety inference text vector and the anxiety termination state vector includes the following operations: First, a semantic encoding model consistent with the pre-training corpus of the large language model is invoked, such as Sentence-BERT based on the Transformer architecture or other unsupervised semantic representation models, to encode the anxiety inference text and the anxiety termination state text respectively, obtaining two embedded vector representations with consistent dimensions. Then, in the same vector space, the similarity is calculated on the two vectors based on the normalized cosine similarity method. Specifically, the magnitude of the two vectors is calculated and normalized, and then a dot product operation is performed to finally obtain a similarity score in the interval [0,1]. The score reflects the degree of semantic consistency between the anxiety inference output and the target anxiety state, and is one of the core indicators for evaluating the effectiveness of inference.

[0110] In this embodiment, the output of the anxiety emotion evolution reasoning ability assessment results includes a partitioning strategy based on the fusion assessment score, generating structured result sets respectively, specifically including:

[0111] When the semantic similarity score is higher than the first threshold and the path consistency score is higher than the second threshold, it is marked as a high consistency and high matching class, and the comprehensive score label is directly output as "inference accurate-stable".

[0112] When the semantic similarity score is higher than the first threshold and the path consistency score is lower than the second threshold, it is marked as semantically reliable but path unstable, and the output comprehensive score label is "reasoning reliable - path scattered", with an additional path perturbation warning label;

[0113] When the semantic similarity score is lower than the first threshold and the path consistency score is higher than the second threshold, it is marked as a path-stable but semantically deviated class, and the output comprehensive score label is "reasoning deviation - path stability" with additional semantic drift hints;

[0114] When both the semantic similarity score and the path consistency score are below the corresponding threshold, they are marked as inference failures, and the comprehensive score label is "inference failure - needs reconstruction", which is recorded as a model fine-tuning re-examination sample.

[0115] All sample evaluation results are summarized to generate a structured evaluation report that includes inference type distribution, path activation mode statistics, and an index of inference failure samples.

[0116] The semantic similarity score and the path consistency score are weighted and fused according to a weighted fusion rule to obtain the fusion score;

[0117] The fusion score is used to comprehensively reflect the evaluation results of semantic similarity score and path consistency score. The value of the fusion score ranges from zero to one. The higher the fusion score, the higher the evaluation results of semantic similarity score and path consistency score. The fusion score is obtained by weighting and summing the semantic similarity score and path consistency score by preset fusion weight parameters.

[0118] Anxiety level labels are output based on the range of the fusion score. These labels are used to identify the current anxiety state of the evaluated individual. The output rules for the anxiety level labels are as follows:

[0119] When the fusion score is less than or equal to the first preset threshold, a low anxiety label is output.

[0120] When the fusion score is greater than the first preset threshold and less than or equal to the second preset threshold, a moderate anxiety label is output.

[0121] When the fusion score is greater than the second preset threshold, a high anxiety label is output.

[0122] The first preset threshold and the second preset threshold are used to distinguish different anxiety levels. The first preset threshold is less than the second preset threshold, and both are real numbers between zero and one.

[0123] Example 1:

[0124] In recent years, with the increasing academic pressure, employment anxiety, and difficulty in social adaptation among college students, the incidence of anxiety among students has been rising year by year. Traditional mental health management mainly relies on regular scale tests or face-to-face interviews, which have limitations such as slow response, strong subjectivity, difficulty in capturing short-term fluctuations and detailed behaviors during the conversation. This implementation plan was launched at the mental health counseling center of a key university in a certain city to serve daily psychological counseling and intervention.

[0125] At the psychological counseling center of a key university in a certain city, students can have multiple rounds of dialogue with psychological counselors online via text and voice. The intelligent system automatically helps to identify and assess anxiety status in real time and promptly pushes personalized intervention suggestions. The assessment system is connected to the campus data platform, which can collect and process the dialogue content between students and psychological counselors in real time.

[0126] In May 2024, the consultation took place on the remote counseling platform of the Psychological Health Center of XX University. The student seeking counseling, Xiao Li (a 21-year-old male junior), had recently been experiencing significant stress from postgraduate entrance exam preparation, strained relationships with classmates, and marked mood swings, and thus scheduled an online consultation.

[0127] The system first collects a textual dialogue between Xiao Li and the consultant. The text consists of 6 rounds and contains a total of 847 words. An example excerpt is shown in Example 1 below:

[0128] The student's initial statement: "I've been feeling very nervous lately. Sometimes my heart races for no reason. I often can't sleep well at night and I have no energy during the day."

[0129] Detailed description of behavior: "He kept his hands clenched while talking, had difficulty concentrating, often spaced out, and kept rubbing his fingers."

[0130] At the end of the consultation, the counselor asked, "What is the most difficult feeling for you right now?" The student replied, "It's the uncontrollable anxiety; my mind is filled with worries."

[0131] The annotator automatically generates anxiety emotion evolution triplets based on the content. Each triplet includes the initial state of anxiety emotion ("nervous, heartbeat" in Example 1), descriptive behavioral phrases ("shaking hands, rubbing fingers, daydreaming" in Example 1), and the terminating state of anxiety emotion ("uncontrollable anxiety" in Example 1).

[0132] The following standard inference task input is automatically generated according to the system's preset template in Example 1:

[0133] "The student's initial state is tension and rapid heartbeat, during which they repeatedly exhibit behaviors such as shaking hands and rubbing their fingers, ultimately described as uncontrollable anxiety. Please infer the student's level of anxiety and the impact of their behavior." This forms a standardized task input.

[0134] The system calls a behavior detail vocabulary (including action, gaze, breathing, tone, and posture categories) to perform sentence segmentation, word segmentation, and dependency parsing on the text.

[0135] The system accurately identifies behavioral detail words such as "handshake", "rubbing fingers", "daydreaming", and "difficulty concentrating" and establishes an intensity label for each word (in Example 1, the word "frequently" before "rubbing fingers" is marked as high intensity).

[0136] For each detail word, a subsequence centered on five words in the context is automatically extracted (in Example 1, it is "holding hands while speaking") to construct a detail description subsequence.

[0137] The system encodes the words in the subsequence using a word vector table, and combines positional information and dependency structure to generate detailed description vectors, preparing for subsequent model inference and gating mechanisms.

[0138] The detailed description vector and the task input are fed together into the improved HyperCLOVA large model.

[0139] In each Transformer layer, the gating mechanism dynamically controls the activation path of the expert subnetwork based on the detail description vector (in Example 1, the third layer selects the subnetwork most relevant to the action class). Each round of inference automatically records the activated expert path index and weight, forming an expert path record. The system generates the following inference text: "The student exhibits significant anxiety; both physical behaviors (shaking hands, rubbing fingers) and cognitive behaviors (daydreaming, difficulty concentrating) are characteristic of high anxiety levels. The current state is moderate to severe anxiety."

[0140] For the expert paths recorded during the reasoning process, a path consistency score is automatically calculated to reflect the stability of expert path activation in multi-turn dialogues (the score in this case is 0.81, which is relatively high). A semantic encoding model consistent with the pre-training corpus (Sentence-BERT in Example 1) is used to encode semantic vectors for the reasoning text and the triple termination state text respectively, and the semantic similarity score between the two is calculated (the score in this case is 0.87, which is extremely high).

[0141] The system weights and fuses the path consistency score and semantic similarity score according to their respective weights to arrive at a final evaluation score (fusion score of 0.85). Based on the score range settings (e.g., less than or equal to 0.4 for low anxiety, 0.4-0.7 for moderate anxiety, and greater than 0.7 for high anxiety in Example 1), the system outputs a "high anxiety" label in real time. The system then pushes personalized intervention suggestions to counselors and students; in Example 1, this includes arranging further psychological counseling, behavioral relaxation training, and focused follow-up visits.

[0142] This embodiment simulates real data collected by a university psychological center in May 2024, including a total of 210 student online counseling records, divided into "the method of this invention" group and "the traditional method group" with 105 cases each.

[0143] Table 1. Training Sample Data (Partial Display)

[0144] Student ID initial state Behavioral details Termination status Expert Tags Actual level A001 Nervous, heart racing Shake hands, rub fingers My mind is filled with worries. High anxiety high A002 Poor sleep, nightmares Shifty eyes, nail-biting I've been worried about my exam results Moderate anxiety moderate A003 Depressed mood, decreased appetite Pause, sigh Unable to concentrate High anxiety high A004 Irritability, headache Rubbing temples, tugging at the hem of clothing Nervous and lacking confidence Moderate anxiety moderate ... ... ... ... ... ...

[0145] Comparison of performance data:

[0146] Reasoning accuracy (using expert human labeling as the gold standard):

[0147] The method group of this invention: 94.8% (99 / 105 consistent with expert labels), the conventional method group: 78.1% (82 / 105 consistent with expert labels).

[0148] High consistency and high matching rate (i.e., accurate inference - percentage of stable path labels):

[0149] The percentage of the method group of this invention was 88.6%, while that of the conventional method group was 65.7%.

[0150] Average evaluation latency (from input to output labels, in seconds):

[0151] The time taken by the method group of this invention is 2.5 seconds, while that of the conventional method group is 12.2 seconds.

[0152] Lead time for high-risk anxiety warnings (average time difference between the first system warning and manual detection by a counselor, in minutes):

[0153] The method group of this invention: 16.3 minutes earlier; the traditional method group: 5.6 minutes earlier.

[0154] Satisfaction survey on the interpretability of reasoning results (anonymous ratings by students and counselors, on a 5-point scale):

[0155] The method group of this invention: 4.7 points, the traditional method group: 3.5 points.

[0156] Taking student A001 as an example, after the original text is processed by the system, high-risk anxiety details such as "nervousness", "handshake", and "finger rubbing" are automatically identified. The semantic similarity score between the inferred text and the anxiety termination state is 0.91, the expert path consistency score is 0.83, and the fusion score is 0.87. The system automatically outputs "high anxiety" and generates intervention suggestions, giving a high-risk reminder 21 minutes in advance of the counselor's manual assessment.

[0157] Compared to traditional methods, which require manual recording of each sentence and rely on subjective experience for evaluation, taking an average of 15 minutes, and where detailed behaviors are not standardized and incorporated into decision-making, it is easy to overlook short-term fluctuations or implicit details, and the evaluation level is mostly a delayed feedback.

[0158] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A real-time assessment method for anxiety states based on multimodal fusion, characterized in that, Includes the following steps: S1. Collect text data containing several rounds of dialogue and construct a set of anxiety emotion evolution triplets; S2. Convert the anxiety emotion evolution triplet into natural language prompt text to generate standardized reasoning task input text; S3. Perform detail word recognition operation on each task input text, use the behavior detail word list to match keywords, combine the matched keywords and five words above and below to form a subsequence, and generate a set of detail description subsequences. S4. Encode the detailed description subsequences with word vectors, and fuse positional embeddings and syntactic dependency relations to generate detailed description vectors; S5. Input the task text and detail description vector into the improved HyperCLOVA model containing a detail-aware gating mechanism. In each Transformer layer, gating weights are generated based on the detail description vector to control the activation path of the expert subnetwork, perform anxiety emotion evolution reasoning operations, record the activated expert path index, and output the anxiety emotion reasoning text set. The improved HyperCLOVA model, which includes a detail-aware gating mechanism in step S5, has the following structure and operation process: In each layer of the Transformer structure, several expert subnetworks are set up. All expert subnetworks do not share parameters and remain independent. The detailed description vector is input into the gate weight generation unit, and a linear transformation and activation function operation are performed to generate a gate control vector equal to the number of expert subnetworks. Perform dot product calculation on the gating control vector and the weight vector corresponding to the expert subnetwork, and output the nonnormalized response score for each expert. Normalize all expert response scores to form an expert selection probability distribution; In each layer, the expert subnetwork with the highest response value is selected according to the expert selection probability distribution. The output of the selected expert subnetwork is weighted and summed according to the corresponding weight in the probability distribution, and then replaced with the original feedforward network output of this layer. During the model inference process, the number sequence and corresponding weight value of each activated expert subnetwork are recorded to form an expert path index; S6. Record the expert path index corresponding to each round of reasoning, calculate the path consistency score of the input text of the reasoning task, and generate a path record matrix. S7. Perform semantic comparison between the anxiety inference text set and the anxiety termination state in the anxiety evolution triplet, generate an evaluation vector by combining the path consistency score, and output the evaluation result of anxiety evolution reasoning ability and anxiety level label.

2. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 1, characterized in that, The construction of the anxiety evolution triple set in step S1 includes: labeling the anxiety start state, descriptive behavior phrases and anxiety termination state in the text data of each dialogue, and jointly constructing the anxiety evolution triple set.

3. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 2, characterized in that, The process of converting the anxiety evolution triplet into natural language prompt text in step S2 includes: constructing a rule set based on a preset task prompt template; filling the anxiety start state text, descriptive behavior phrase text, and anxiety termination state text in each triplet into the corresponding template placeholder positions; generating prompt text that matches the task type; and constructing standardized inference task input text. The task prompt template and the rule set each contain an anxiety start state placeholder, a descriptive behavior phrase placeholder, and an anxiety termination state placeholder.

4. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 3, characterized in that, The detail word identification and keyword matching in step S3 includes the following steps: Construct a behavioral detail vocabulary, and divide the vocabulary into sets of entries according to action, gaze, breathing, tone and posture. Establish a synonym mapping rule for each entry, map different word forms with the same or similar meanings to a unified standard entry number, and establish a word form normalization rule to restore different parts of speech, tenses or voice changes to standard word forms. The task input text is processed by sentence segmentation and word segmentation, preserving the original word order and punctuation position information. Each word segment is labeled with part-of-speech tags based on natural language processing tools, and the subject-predicate, verb-object, and modifier relationships between word segments are determined based on dependency parsing. In the word segmentation results, the behavior detail vocabulary is called to perform matching, and the longest priority matching strategy is adopted: when two or more words have matching conflicts in the same position, the word with the longest length is selected first; if the lengths are the same, the word with the highest frequency in the vocabulary is selected; if the length and frequency are the same, the word with the earliest position in the sentence is selected first. The initial matching results are semantically corrected: negative words before the matching words are detected, and if the matching words are completely within the scope of the negative words, the matching is removed; degree adverbs before the matching words are detected, and intensity labels are attached to the matching words, with the intensity coefficient recorded in the labels; Centered on the word order position of each retained matching word in the sentence, five candidate subsequences are extracted to the left and right of each of the five adjacent non-punctuation words in the same sentence. When encountering the beginning or end of the sentence, the subsequences are supplemented with the actual extractable words, excluding punctuation marks. The candidate subsequences are subjected to overlap merging and deduplication processing to ensure that only one continuous subsequence is retained within the same range. The final detailed description subsequence set is generated based on word order continuity and minimum length constraints. A one-to-one index relationship between each subsequence in the set and the corresponding matching word is established.

5. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 4, characterized in that, Step S4 specifically includes: For each word in the detailed description subsequence, perform an index lookup operation, call the word vector table that matches the language model used to generate the natural language prompt text, retrieve the fixed-length word vector corresponding to the word, and combine them into a word vector matrix according to the order of the words in the detailed description subsequence; Assign an integer number starting from zero to each word in the detailed description subsequence, call the preset position encoding function, calculate the corresponding position embedding vector according to the number, perform a dimension-wise addition between the position embedding vector and the current word vector in the same dimension, replace the original word vector, and generate a representation matrix with fused position embeddings; The analysis results obtained by performing dependency parsing on the original sentence are called, and the connection information between each word and the dependency headword is found in the detailed description subsequence. The relative index position of the current word and the dependency headword and the corresponding dependency label are extracted. Based on the dependency tag lookup, the preset tag weights are obtained. The vector of the center word is multiplied with the tag weights in a dimension-wise manner, and then added with the vector of the current word in a dimension-wise manner. The weighted superposition result is used to replace the current word vector, thus completing the structural information fusion processing based on dependency relationship. One-dimensional average pooling and max pooling operations are performed on the word vectors after structural information fusion processing along the word order of the detail description subsequence, respectively. The two pooling results are then concatenated according to the dimension to construct the detail description vector.

6. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 5, characterized in that, The generation of the path record matrix in step S6 specifically includes: In each round of anxiety evolution inference, the activated expert subnetwork numbers and corresponding gating weights of each layer in the improved HyperCLOVA model are recorded. Expert path vectors are constructed in hierarchical order. The expert path vectors corresponding to all inference task input texts are arranged in order to form a path record matrix. A path consistency score is calculated for each path vector. Specifically, in the anxiety evolution triplet type set with the same source as the inference task input text, the mean similarity between the expert path vector and other path vectors is calculated based on normalized cosine similarity. The obtained mean similarity is used as the path consistency score and appended to the path record of the corresponding text to form a path record matrix containing the expert path number sequence and the path consistency score.

7. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 6, characterized in that, The process of generating the evaluation vector in step S7 includes: for each text in the anxiety inference text set, calling the anxiety termination state text in the corresponding anxiety evolution triplet, inputting it into the semantic encoding model consistent with the model's pre-training corpus, obtaining the anxiety inference text vector and the anxiety termination state vector, and calculating the semantic similarity score between the anxiety inference text vector and the anxiety termination state vector; weighting and fusing the semantic similarity score with the path consistency score generated from the corresponding task input text according to a fixed ratio to generate an evaluation value reflecting semantic accuracy and inference path stability; using the evaluation value as the main component of the evaluation vector, combining it with the task input text identifier, the original semantic similarity score, the path consistency score, and the fusion ratio parameter to form the evaluation vector.

8. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 7, characterized in that, The output of the anxiety evolution reasoning ability assessment results includes a partitioning strategy based on the fusion assessment score, generating structured result sets, specifically including: When the semantic similarity score is higher than the first threshold and the path consistency score is higher than the second threshold, it is marked as a high consistency and high matching class, and the comprehensive score label is directly output as "inference accurate-stable". When the semantic similarity score is higher than the first threshold and the path consistency score is lower than the second threshold, it is marked as semantically reliable but path unstable, and the output comprehensive score label is "reasoning reliable - path scattered", with an additional path perturbation warning label; When the semantic similarity score is below the first threshold and the path consistency score is above the second threshold, it is marked as a path-stable but semantically deviated class, and the output comprehensive score label is "reasoning deviation - path stability", with additional semantic drift hints; When both the semantic similarity score and the path consistency score are below the corresponding threshold, they are marked as inference failures, and the comprehensive score label is "inference failure - needs reconstruction", which is recorded as a model fine-tuning re-examination sample. All sample evaluation results are summarized to generate a structured evaluation report that includes inference type distribution, path activation mode statistics, and an index of inference failure samples.

9. The method for real-time assessment of anxiety state based on multimodal fusion according to claim 8, characterized in that, The rules for the anxiety level labeling include: The semantic similarity score and the path consistency score are weighted and fused according to a weighted fusion rule to obtain the fusion score; Anxiety level labels are output based on the range of the fusion score. These labels are used to identify the current anxiety state of the evaluated individual. The output rules for the anxiety level labels are as follows: When the fusion score is less than or equal to the first preset threshold, a low anxiety label is output. When the fusion score is greater than the first preset threshold and less than or equal to the second preset threshold, a moderate anxiety label is output. When the fusion score is greater than the second preset threshold, a high anxiety label is output.

Images