Multi-modal interview automatic quality analysis and evaluation method and system based on large model

Abstract

The invention discloses a multi-modal interview automatic quality analysis and evaluation method and system based on a large model, and the method comprises the steps: collecting and storing multi-modal data, such as texts, audios, videos and behavior interaction, and carrying out the preprocessing of the multi-modal data to form a standardized data set; utilizing a preset interview structure and a large model to dynamically guide the process, adjusting the topic rhythm according to real-time feedback, and recording stage conversion information to form logic trajectory data for process coherence management; automatically coding text data through a large language model, extracting features such as keywords and performing topic clustering, performing cross validation and semantic fusion in combination with data analysis results of each modal, and generating deep analysis results such as psychological states; and generating a comprehensive assessment report containing qualitative description, quantitative score and psychological abnormality or cognitive disorder risk prompts based on a deep analysis result, thereby providing a basis for psychological health assessment and cognitive competence evaluation. According to the method, automatic analysis of multi-modal data is realized, and evaluation scientificity and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for automated qualitative analysis and evaluation of multimodal interviews based on large models. Background Technology

[0002] In traditional psychological assessment methods, experts collect data on an individual's psychology, cognition, and emotions through interviews, and then analyze and evaluate the results. This process typically relies on manual qualitative analysis, where experts extract underlying psychological characteristics and cognitive states by analyzing multimodal data such as interview texts, speech, and body language, based on their personal experience and professional knowledge. However, traditional interview and qualitative analysis methods have several problems: Human analysis is inherently subjective and biased. Traditional psychological interviews rely on the subjective judgment of experts, and the analysis process can be influenced by factors such as the expert's personal experience, emotional state, and cognitive biases, leading to inconsistencies and poor repeatability in the assessment results. This subjectivity and bias make it difficult for psychological test results to meet the standards of academic and industry acceptance in certain complex assessment scenarios.

[0003] Data processing is inefficient; traditional interview data analysis is cumbersome and time-consuming. Experts need to read, understand, and analyze each interview response individually, manually coding, classifying, and summarizing them. This process is not only labor-intensive but also slow and inefficient. Especially when faced with large amounts of interview data, manual analysis becomes almost impractical, and the timeliness of the results cannot be guaranteed.

[0004] Multimodal data analysis is complex. Modern psychological assessments rely not only on textual data but also on multimodal data such as audio, video, emotional expressions, and behavioral observations. However, current technologies have limited capabilities in integrating and analyzing multimodal data. This is especially true when different forms of data need to be considered comprehensively during interviews to arrive at a holistic assessment conclusion; traditional methods often struggle to process this data efficiently and accurately.

[0005] Large-scale assessments have limitations. Traditional interview-based assessment methods require significant human intervention, especially in applications such as mental health and large-scale cognitive ability testing, where their scope and feasibility are severely restricted. The lack of automated and intelligent analytical tools further complicates their implementation on a large scale.

[0006] Currently, some fields of psychological assessment are attempting to introduce artificial intelligence technologies, especially natural language processing (NLP) and machine learning methods, to automate the analysis of interview data. Existing AI-based mental health assessment systems typically rely on simple text analysis or voice emotion recognition to determine emotional attitudes. However, these systems still have the following drawbacks: Lacking in-depth multimodal analysis capabilities, most existing systems can only process single data types (such as text or voice) and cannot simultaneously consider multiple data sources in the interview (such as voice, facial expressions, behavioral patterns, etc.), thus failing to conduct a comprehensive psychological and cognitive assessment of the evaluated individuals from multiple perspectives.

[0007] The evaluation results lack depth. Existing AI systems typically analyze data using superficial statistical methods. These systems are insufficient in their in-depth analysis of individual psychological states, emotional changes, and cognitive biases, and cannot provide professional psychologists with actionable in-depth analysis.

[0008] There is a problem with the integration with traditional psychological theories. Although some AI-based assessment tools have appeared on the market, most of them fail to combine traditional psychological theories for comprehensive analysis, which makes their assessment results often lack academic support and recognition. Summary of the Invention

[0009] In view of this, the present invention provides an automated qualitative analysis and evaluation method and system for multimodal interviews based on large models to solve the above-mentioned technical problems.

[0010] To achieve the above objectives, firstly, a method for automated qualitative analysis and evaluation of multimodal interviews based on a large model is provided, which includes the following steps: S10: Automatically collect and store multimodal data of the interviewees during the interview process. The multimodal data includes text data, audio and speech data, video data, and behavioral interaction data. Preprocess the multimodal data to obtain a standardized multimodal dataset. S20: Dynamically guide the interview process using a pre-configured interview structure and a large artificial intelligence model; automatically adjust the direction of the interview topic and the pace of the conversation based on the real-time feedback of the interviewee and the predefined interview strategy; record the interview stage transition information in real time; and form logical trajectory data for interview process control based on the interview stage transition information. The logical trajectory data is used for the coherence management of the interview process. S30: Based on a pre-trained large language model, the text data in the standardized multimodal dataset is automatically encoded, and the encoding results, including keywords, emotional features, and psychological state features, are extracted; the encoding results are subjected to topic clustering to identify topic clustering results, including core topics and secondary topics; the topic clustering results, the acoustic features and speech emotion analysis results of the audio speech data, the facial expression and body movement feature analysis results of the video data, and the quantitative scoring results of the behavioral interaction data are cross-validated and semantically fused to generate in-depth analysis results including psychological state, cognitive pattern, and emotional stability; S40: Based on the in-depth analysis results, a comprehensive assessment report is generated, which includes qualitative descriptions, quantitative scores, and risk warnings. The qualitative descriptions are used to describe the psychological, cognitive, and emotional characteristics of the interviewees. The quantitative scores quantify the psychological or cognitive state based on preset psychological scale scoring standards and machine learning classification algorithms. The risk warnings are used to identify abnormal psychological states or cognitive impairment risks. The comprehensive assessment report provides a basis for the psychological health assessment or cognitive ability evaluation of the interviewees.

[0011] Secondly, an automated qualitative analysis and evaluation system for multimodal interviews based on a large model is provided, which includes: The multimodal data acquisition module is configured to automatically collect and store the multimodal data of the interviewee during the interview process. The multimodal data includes text data, audio and speech data, video data, and behavioral interaction data. The data preprocessing module is configured to preprocess the multimodal data to generate a standardized multimodal dataset; The interview process control module is configured to dynamically guide the interview process using a pre-configured interview structure and a large artificial intelligence model, adjust the dialogue process based on real-time feedback, and generate logical trajectory data for interview process control. The multimodal data analysis module is configured to process the standardized multimodal dataset based on a pre-trained large language model, including automatic text data encoding, topic clustering analysis, cross-validation of multimodal data, and semantic fusion, to generate in-depth analysis results. The assessment report generation module is configured to generate a comprehensive assessment report based on the in-depth analysis results, including qualitative descriptions, quantitative scores, and risk warnings, providing a basis for mental health assessments or cognitive ability tests.

[0012] The above technical solution has the following beneficial technical effects: This method automatically collects and preprocesses multimodal data (text, audio, video, and behavioral interaction data) from interviewees to ensure data standardization and integrity. It utilizes a large-scale artificial intelligence model to dynamically guide the interview process, intelligently adjusting the rhythm and topic direction of the dialogue to ensure the continuity of data collection. A pre-trained large-scale language model is used to automatically encode and cluster text data, and the results of multimodal data analysis are combined with cross-validation and semantic fusion to deeply explore characteristics such as psychological state, cognitive patterns, and emotional stability, improving the objectivity and comprehensiveness of the analysis. Finally, a comprehensive assessment report containing qualitative descriptions, quantitative scores, and risk warnings is generated, providing a scientific and accurate basis for mental health assessment and cognitive ability evaluation. This achieves automated qualitative analysis driven by multimodal data, effectively improving assessment efficiency and scientific rigor, reducing labor costs, and facilitating the early identification and intervention of abnormal states. Attached Figure Description

[0013] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a flowchart of the automated qualitative analysis and evaluation method for multimodal interviews based on a large model, according to an embodiment of the present invention. Figure 2 This is a flowchart of the automatic encoding and multimodal data cross-validation process according to an embodiment of the present invention; Figure 3 This is a flowchart of the multimodal psychological cognitive risk classification assessment according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the reliability test in an embodiment of the present invention. Figure 5 This is the feedback data-driven model optimization and fine-tuning process of this invention embodiment; Figure 6 This is a functional block diagram of an automated qualitative analysis and evaluation system for multimodal interviews based on a large model, according to an embodiment of the present invention. Figure 7 This is another detailed functional block diagram of the automated qualitative analysis and evaluation system for multimodal interviews based on a large model, according to an embodiment of the present invention. Figure 8 This is a schematic diagram of the structure of a computer system according to an embodiment of the present invention. Detailed Implementation

[0014] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details of these embodiments to aid understanding, and should be considered merely exemplary.

[0015] This invention relates to the field of artificial intelligence technology, specifically an automated qualitative analysis method based on a large language model and its application in multimodal interview assessment. This method analyzes various input data (including text, speech, multiple-choice questions, calculation questions, etc.) during the interview process to automatically extract and assess the interviewee's psychological, cognitive, and emotional states, as well as related risk factors. It is widely used in assessment fields such as mental health assessment, cognitive ability assessment, employability assessment, and interpersonal relationship assessment.

[0016] This invention combines multimodal data input, automated qualitative analysis using large-scale artificial intelligence models, and assessment report generation technology to improve the efficiency, accuracy, and personalization of the assessment process. It addresses the problems of strong subjectivity, time-consuming nature, and difficulty in large-scale application in traditional psychological interview assessments, providing a reliable automated interview analysis tool.

[0017] In view of the problems and limitations of the existing technology, the purpose of this invention is to provide an automated qualitative analysis and evaluation method and system for multimodal interviews based on a large model, so as to realize automated processing and in-depth analysis of interview data and improve the objectivity, accuracy and practicality of the evaluation results.

[0018] This invention discloses an automated qualitative analysis and assessment method and system for multimodal interviews based on a large model, belonging to the field of artificial intelligence assessment technology. The method first automatically collects text, audio, video, and interactive data during the interview process and standardizes the data. Then, it automatically performs qualitative analysis of the interview data using a pre-trained large language model, including automatic coding, topic clustering analysis, deep semantic analysis, and cross-modal data fusion analysis, accurately extracting the psychological, cognitive, emotional, or ability characteristics of the interviewees. Finally, the system automatically generates personalized assessment reports based on the analysis results, including qualitative descriptions, quantitative scores, and risk warning information, and provides corresponding intervention suggestions. Furthermore, this invention provides a feedback optimization mechanism to continuously optimize the analysis model through user and expert feedback, improving assessment accuracy and practicality. This invention achieves automation, standardization, and efficiency in the interview assessment process, and is applicable to multiple fields such as mental health assessment, cognitive ability screening, employability assessment, and interpersonal relationship analysis, with broad application prospects and promotional value.

[0019] like Figure 1 As shown, an automated qualitative analysis and evaluation method for multimodal interviews based on a large model includes the following steps: S10: Automatically collect and store multimodal data of the interviewees during the interview process. The multimodal data includes text data, audio and speech data, video data, and behavioral interaction data. Preprocess the multimodal data to obtain a standardized multimodal dataset. S20: Dynamically guide the interview process using a pre-configured interview structure and a large artificial intelligence model; automatically adjust the direction of the interview topic and the pace of the conversation based on the real-time feedback of the interviewee and the predefined interview strategy; record the interview stage transition information in real time; and form logical trajectory data for interview process control based on the interview stage transition information. The logical trajectory data is used for the coherence management of the interview process. S30: Based on a pre-trained large language model, the text data in the standardized multimodal dataset is automatically encoded, and the encoding results, including keywords, emotional features, and psychological state features, are extracted; the encoding results are subjected to topic clustering to identify topic clustering results, including core topics and secondary topics; the topic clustering results, the acoustic features and speech emotion analysis results of the audio speech data, the facial expression and body movement feature analysis results of the video data, and the quantitative scoring results of the behavioral interaction data are cross-validated and semantically fused to generate in-depth analysis results including psychological state, cognitive pattern, and emotional stability; S40: Based on the in-depth analysis results, a comprehensive assessment report is generated, which includes qualitative descriptions, quantitative scores, and risk warnings. The qualitative descriptions are used to describe the psychological, cognitive, and emotional characteristics of the interviewees. The quantitative scores quantify the psychological or cognitive state based on preset psychological scale scoring standards and machine learning classification algorithms. The risk warnings are used to identify abnormal psychological states or cognitive impairment risks. The comprehensive assessment report provides a basis for the psychological health assessment or cognitive ability evaluation of the interviewees.

[0020] In some embodiments, in step S10, the text data is obtained by converting interview speech into text using real-time speech recognition technology; the audio speech data is obtained by extracting acoustic features including fundamental frequency, spectral features, speech rate, and pause patterns, and generating emotional state indicators through a speech emotion recognition model; the video data is obtained by recognizing facial movement units, eye contact patterns, micro-expressions, and the frequency and amplitude of body movements; the behavioral interaction data includes the result data of multiple-choice questions, true / false questions, calculation questions, subconscious projection card interactions, and connection interactions.

[0021] In some embodiments, in step S30, the automatic encoding is implemented by constructing an encoding prompt word template through a large language model; the topic clustering is implemented by performing semantic similarity calculation and clustering grouping on the encoded labels through an unsupervised clustering algorithm, wherein the encoded labels are digital representations of the keywords, the emotional features, and the psychological state features; the cross-validation of multimodal data evaluates the consistency of each modality by calculating cross-modal consistency scores, and marks abnormal data with consistency scores below a threshold.

[0022] like Figure 2As shown, specifically, in the specific implementation of step S30, the automatic encoding process is implemented by constructing a domain-specific encoding prompt word template through a pre-trained large language model (e.g., GPT-4, LLaMA). For example, for a mental health interview scenario, the prompt word template can be designed as follows: Please extract 5 core keywords from the following text, identify the main emotional features (e.g., anxiety, depression, calmness) and the corresponding psychological state features (e.g., cognitive dissonance, emotional repression, self-identity), and output them in JSON format: {keywords:[], emotional features:[], psychological state features:[]}”. The model processes the text data in the standardized multimodal dataset based on this template, generates an encoding result containing keywords, emotional features, and psychological state features, and converts it into numerical encoding labels (e.g., through One-Hot encoding or word vector embedding technology).

[0023] In the topic clustering stage, unsupervised clustering algorithms (such as K-means, DBSCAN, or hierarchical clustering) are used to calculate the semantic similarity of the above-mentioned coded labels. The specific steps include: first, mapping each coded label to a d-dimensional feature vector (d is the feature dimension), measuring the semantic similarity between samples by cosine similarity or Euclidean distance, and then dividing the feature vector into core topics and secondary topics according to the preset number of clusters (e.g., determined by the elbow rule). For example, high-frequency labels such as "anxiety", "sleep disorder" and "work pressure" are clustered into the core topic of "occupational mental health".

[0024] The cross-validation stage for multimodal data assesses the consistency of text, audio, video, and behavioral interaction data by calculating cross-modal consistency scores. First, acoustic features such as fundamental frequency, speech rate, and Mel-frequency cepstral coefficients (MFCC) are extracted from the audio speech data, and an emotional state probability vector (e.g., [anxiety: 0.8, calm: 0.2]) is output using a pre-trained speech emotion model. For video data, features such as AU unit strength, eye contact duration, and body movement amplitude corresponding to the Facial Action Coding System (FACS) are extracted. For behavioral interaction data, quantitative indicators such as accuracy, reaction time, and card selection mode are generated. Then, features from each modality are... The system standardizes the data using Z-score or Min-Max normalization to achieve a uniform scale. It then calculates the consistency score (range [-1,1] or [0,1]) of any two modal feature vectors using Pearson correlation coefficient, cosine similarity, or canonical correlation analysis (CCA). When the average consistency score of a modality's data with other modalities is lower than a preset threshold (e.g., 0.6, which can be determined through domain expert experience or experiments), an anomaly marking mechanism is triggered, generating a log containing the location and type of anomalies for subsequent manual verification or automatic cleaning. This ensures the reliability and consistency of the analysis basis for multimodal data.

[0025] The aforementioned technical solution automatically encodes text data by constructing encoding prompt word templates using a large language model. This transforms unstructured text into structured encoded labels containing keywords, emotional features, and psychological state characteristics, effectively improving the automation level of data processing and the standardization of feature extraction, while reducing the subjectivity and time-consuming costs of manual encoding. Unsupervised clustering algorithms are used to calculate semantic similarity and group encoded labels, automatically identifying core and secondary themes based on the inherent semantic relationships within the data. This provides a hierarchical semantic framework for subsequent multimodal fusion analysis, enhancing the systematic and logical nature of thematic analysis. By calculating cross-modal consistency scores and marking anomalous data, a quantitative assessment of the consistency of multimodal data such as text, audio, video, and behavioral interactions is achieved. This effectively filters out noise information involving feature conflicts or data anomalies between modalities, ensuring the reliability of multimodal data fusion and providing a data foundation for generating scientific and accurate in-depth analysis results. Ultimately, this improves the comprehensiveness and accuracy of the entire evaluation method in analyzing psychological states and cognitive patterns.

[0026] In some embodiments, in step S40, the quantitative scoring includes a mental health scale score or a cognitive function assessment scale score; the risk warning is based on the cross-modal consistency score and preset abnormal state judgment rules to identify the risk of abnormal mental states or cognitive impairment. By quantifying mental / cognitive states through standardized scales and combining them with multimodal data consistency analysis to identify abnormal risks, the assessment results are quantified, objective, and multi-dimensionally risk-aware, thereby improving the scientific nature of mental health and cognitive ability assessments and the accuracy of abnormal state identification.

[0027] like Figure 3 As shown, specifically, in the specific implementation of step S40, the quantitative scoring is achieved by integrating standardized psychological scales and machine learning algorithms: For mental health assessment, the system presets assessment tools such as the PHQ-9 Depression Scale, the GAD-7 Anxiety Scale, and the PTSD-17 Post-Traumatic Stress Disorder Scale, and maps multimodal features such as text encoding labels (e.g., frequency of keywords like sadness and helplessness), voice emotion analysis results (e.g., probability value of anxiety), and video micro-expression intensity (e.g., duration of drooping corners of the mouth) generated in S30 to the scale scoring system. For example, the number of times the keyword "suicidal thoughts" appears in the text is directly associated with the score of question 9 of the PHQ-9. For cognitive function assessment, the MMSE Mini-Mental State Examination and the MoCA Montreal Cognitive Assessment Scale are used, combined with quantitative indicators such as the accuracy rate of calculation questions, the completion time of matching questions, and the number of words recalled in the behavioral interaction data, and a weighted average algorithm is used to generate a cognitive function score.

[0028] The risk warning mechanism is based on cross-modal consistency scores and a preset rule engine. First, abnormal state judgment rules are defined, such as: "When the anxiety feature score of the text encoding is >75th percentile, the standard deviation of the speech fundamental frequency is >2 times the normal range standard deviation, the frequency of furrowed brows in the video is >15 times / minute, and the cross-modal consistency score of these three factors is <0.6, it is judged as moderate anxiety tendency"; or "If the cross-modal consistency score of three consecutive interview stages is <0.5, and the MMSE score is <24, then a cognitive impairment risk warning is triggered." The system analyzes the cross-modal consistency score generated by S30 (e.g., the text-speech emotion correlation value calculated using the Pearson correlation coefficient) and abnormal features of each modality (e.g., abnormal amplitude of video body movements detected based on Z-score) through the rule engine. When the preset rules are met, the system automatically matches the corresponding abnormal state type (e.g., anxiety disorder or mild cognitive impairment in the DSM-5 standard) and generates a risk level (low / medium / high). For example, if text analysis shows that the keyword "memory error" appears frequently, the correct answer rate for multiple-choice questions in behavioral interaction data is <60%, and the consistency score between these two factors and the video gaze-wandering feature is 0.45 (< the preset threshold of 0.6), then it is marked as a moderate risk of memory decline. Simultaneously, it is correlated with relevant diagnostic criteria in the Diagnostic and Statistical Manual of Mental Disorders (DSM) to provide a clinical basis for subsequent intervention. Finally, the quantitative scores and risk warning results are integrated into a comprehensive assessment report in the form of visual charts, such as displaying the scores of each dimension of the scale using a radar chart, highlighting abnormal risk items in red, forming a structured assessment conclusion.

[0029] In some embodiments, the method further includes the following steps: S50: receiving feedback data from external sources on the comprehensive evaluation report; dynamically fine-tuning and optimizing the analysis parameters of the large language model based on the feedback data.

[0030] like Figure 5As shown, specifically, in the implementation of step S50, the system receives external feedback data through a front-end interactive interface or API interface. This feedback data includes, but is not limited to: user suggestions for correcting qualitative descriptions in the assessment report (e.g., adjusting mild anxiety to moderate anxiety), abnormal state labels annotated by experts, and comparisons between historical assessment reports and actual clinical diagnoses. After format parsing (e.g., converting natural language feedback into structured feature-label pairs), the feedback data first filters out invalid information (e.g., duplicate feedback, incorrectly formatted data) through a data cleaning module, and then inputs it into the model optimization engine. The optimization engine constructs a fine-tuning training set based on the feedback data and uses transfer learning techniques to dynamically adjust the analysis parameters of the large language model: for the text encoding module, it calculates cross-entropy loss and backpropagates to update the word vector layer parameters by comparing the keyword / emotion features output by the model with the manually annotated results; for the cross-modal fusion module, it adjusts the weight matrix of the attention mechanism using multimodal consistency labels (e.g., "correct risk level") in the feedback data. In practice, an incremental fine-tuning strategy can be set. When the amount of feedback data reaches a preset threshold (e.g., 50 data points / day), the model fine-tuning process is triggered. The Adam optimizer (learning rate 1e-5) is used to train the parameters of the last three Transformer layers of the model for 10 epochs, and overfitting is monitored using a validation set (retaining 20% ​​of historical feedback data). For example, if a user reports that a report "did not identify keywords related to post-traumatic stress disorder," the system extracts the text data, model encoding results, and correct labels for that case, generates training samples, and updates the weights of the encoding prompt word template, enabling the model to prioritize the extraction of relevant features in subsequent analyses.

[0031] Its advantage lies in the fact that by constructing a feedback closed loop to dynamically optimize model parameters, the system can continuously learn from the evaluation experience in practical applications, improve the adaptability of the large language model to specific interview scenarios, and ensure that the evaluation method maintains accuracy and robustness in long-term use.

[0032] In some embodiments, in step S10, the preprocessing of the multimodal data includes filling missing values ​​and removing outliers from the multimodal data to obtain complete and clean data; the complete and clean data is normalized or standardized to unify the data format and scale, resulting in a standardized multimodal dataset; the standardized multimodal dataset is stored in a storage system, which is a distributed database or a document-oriented database.

[0033] In some embodiments, in step S20, the automatic control of the interview process defines the interview structure through a preset JSON configuration file, which includes the order of interview topics, the type of interactive segments, and the jump logic; the artificial intelligence big data model dynamically calls the interview strategy in the JSON configuration file based on the real-time feedback of the interviewees, and adjusts the interview pace and topic direction.

[0034] Specifically, in the implementation of step S20, the automatic control of the interview process is achieved in the following way: First, a domain expert designs a JSON configuration file based on the interview objectives (e.g., mental health screening, career ability assessment), the structure of which is as follows: json { "interview_structure": [ { "stage_id": "stage_1", "stage_name": "Opening Remarks", "questions": [ { "question_id": "q1", "question_text": "Hello, welcome to this interview. How have you been feeling in the past week?" "interaction_type": "open_text", / / Open-ended text response "next_stage": { "trigger_keywords": ["anxiety|stress|insomnia"], / / Regular expression for triggering keywords "jump_to": "stage_2_mental" / / Jump to the core stage of mental health } } ] }, { "stage_id": "stage_2_mental", "stage_name": "Mental Health Assessment", "questions": [ { "question_id": "q2", "question_text": "Do you often feel down and lose interest in things?" "interaction_type": "multiple_choice", / / Multiple choice question "options": ["Never", "Occasionally", "Often", "Almost Every Day"], "next_stage": { "score_threshold": 3, / / Score value for the option (1-4 points), skip to the next section if the score exceeds the threshold. "jump_to": "stage_3_risk" / / Jump to the risk warning stage } } ] } ] } The JSON configuration file is stored in the system's distributed configuration center (e.g., Apache ZooKeeper) and supports real-time hot updates. When the interviewee provides an answer (e.g., replying to q1 "I've been under a lot of stress lately and often have insomnia"), the AI ​​big data model (e.g., a finely tuned LLaMA model) extracts the trigger keywords "stress | insomnia" using natural language processing techniques, matches them to the trigger_keywords rule in stage_1 of the JSON, and automatically calls the jump_to:stage_2_mental interview strategy, skipping the regular interview stage and directly proceeding to the core mental health issues. If the answer is a non-trigger keyword (e.g., "emotionally stable"), the system proceeds to the next default stage in sequence.

[0035] During the interview, the big data model analyzes the emotional polarity of the answers (e.g., through the VADER sentiment analysis algorithm), keyword density, and other features in real time, and dynamically adjusts the order of question presentation. For example, if the frequency of the word "anxiety" in the answer is found to be greater than 2 times per minute, a supplementary question, "Can you describe in detail the event that makes you anxious?" is automatically inserted, and the list of questions in the JSON configuration is temporarily modified to achieve intelligent control of the interview pace.

[0036] Its advantages lie in the ability to visualize and flexibly adjust the interview process through JSON configuration files. Combined with the real-time semantic analysis capabilities of large models, the interview process can be dynamically adapted according to the characteristics of the interviewees, improving the personalization of the interviews and the efficiency of data collection. At the same time, it reduces the technical threshold for process changes, making it easier for domain experts to quickly iterate interview strategies.

[0037] In some embodiments, the reliability test of the encoding results is implemented through a cross-validation mechanism, which includes randomly sampling the same data segment and submitting it to the large language model multiple times for encoding to obtain multiple independent encoding results; evaluating the consistency of the multiple encoding results by calculating the Cohen-Kappa value or the Fris-Kappa value; and triggering a manual review process to correct the encoding results when the consistency score is lower than a preset threshold.

[0038] like Figure 4 As shown, specifically, in the specific implementation of step S30, the reliability test is implemented in the following way: The system first randomly divides the text data in the standardized multimodal dataset into blocks, for example, dividing 100 consecutive characters into a data segment, forming N non-overlapping data blocks. For each data block, the system uses a random number generator (e.g., Python's `random.seed` combined with timestamps) to set different random seeds, controlling the sampling strategy of the large language model and ensuring the independence of each encoding process. For example, the same data segment is encoded 5 times independently, with different temperature parameters set for each encoding (e.g., 0.7, 0.8, 0.9, 1.0, 1.1), generating 5 sets of encoded results containing keywords, emotion features, and psychological state features.

[0039] Subsequently, the system converts the classification labels (e.g., anxiety, depression, etc. in emotional features) in each group of encoding results into binary matrices and uses the Cohen-Kappa score to evaluate the consistency of pairwise encoding results. When the average Kappa score of a data segment is lower than a preset threshold (e.g., 0.6), the system automatically marks the data segment as low confidence and generates a review work order containing a comparison of the differences between the encoding results. Human experts review the data through the visualization interface provided by the system (e.g., a heatmap showing the label overlap of different encoding results), correct ambiguous labels, and add the correct results as new training samples to the model fine-tuning dataset, achieving closed-loop optimization of encoding quality.

[0040] By constructing a reliability testing mechanism through random sampling and multiple independent coding, and combining statistical methods such as the Coen Kappa value to quantitatively evaluate the consistency of coding results, the reliability of automatic coding of large models is ensured. At the same time, a quality control closed loop is formed through a threshold-triggered manual review mechanism, which improves the accuracy and reliability of psychological feature extraction.

[0041] In some embodiments, the topic clustering results are displayed in two dimensions using visualization tools, including t-distributed random neighborhood embedding (t-SNE), consistent manifold approximation and projection (UMAP), or heatmaps, to intuitively present the distribution characteristics and semantic associations of the label clusters.

[0042] In some embodiments, the cross-validation and semantic fusion are implemented through a cross-modal fusion model based on the Transformer architecture. The cross-modal fusion model unifies text features, speech features, video features, and behavioral interaction features into a numerical vector representation. It dynamically learns the weight allocation of the text features, speech features, video features, and behavioral interaction features in different analysis scenarios through an attention mechanism, generating a fusion feature vector containing multimodal semantic associations for the generation of the deep analysis results. In step S30, the text data in the standardized multimodal dataset is automatically encoded based on the pre-trained large language model to obtain vectorized text features containing keywords, emotional features, and psychological state features. Acoustic features, including fundamental frequency, spectral features, speech rate and pause patterns, are extracted from the preprocessed audio speech data in step S10, and emotional state analysis is performed in combination with a speech emotion recognition model to obtain speech features. By performing facial motion unit encoding, eye contact pattern analysis, micro-expression intensity extraction, and quantification of limb movement frequency and amplitude on the preprocessed video data in step S10, video features are obtained. By quantifying and scoring the behavioral interaction data in step S30, behavioral interaction characteristics are obtained.

[0043] Specifically, in the implementation of step S30, the cross-modal fusion model adopts an improved Transformer architecture to achieve multimodal semantic fusion, and its specific implementation method is as follows: The system first maps the features of each modality to a unified 512-dimensional vector space: text features are extracted using the BERT-base model to obtain 768-dimensional vectors at the [CLS] position, which are then projected to 512 dimensions through a linear layer; audio features are extracted using 64-dimensional acoustic features such as MFCC coefficients and fundamental frequency change rate, which are encoded into 512-dimensional vectors through bidirectional LSTM; video features are extracted using 177-dimensional facial action unit (AU) intensity values ​​from OpenFace, combined with 2048-dimensional visual features extracted from ResNet, and compressed to 512 dimensions through attention pooling; behavioral interaction data are converted into 10-dimensional feature vectors containing accuracy, reaction time, etc., through a logistic regression model, which are also projected to the 512-dimensional space.

[0044] In the specific implementation of step S30, the structure of the cross-modal fusion model includes the following components: First, a modal encoder layer, which designs independent Transformer encoders for each modality—text, speech, video, and behavioral interaction. Each encoder contains multiple attention heads and residual connection layers to learn the semantic relationships and feature representations within each modality. Second, a cross-modal attention layer, which uses a multi-head self-attention mechanism to calculate the correlation between any two modalities, enabling the model to capture the potential connections between features of different modalities. Third, a dynamic weight allocation mechanism, which automatically adjusts the importance weights of each modality according to the current analysis scenario through a gating unit. For example, it assigns higher weights to speech features when analyzing anxiety states, and increases the weight ratio of video features when analyzing depression states. Finally, a fusion feature generation layer, which weights and combines the features of each modality according to the dynamically allocated weights, and generates the final fusion feature vector through a fully connected layer. This vector integrates complementary information from multimodal data and is used to generate subsequent deep analysis results. Through this structure, the model can adaptively handle the feature differences of different modal data and achieve deep semantic fusion of multimodal information.

[0045] In mental health assessment scenarios, when analyzing anxiety, the model assigns higher weights to speech features (such as increased speech rate and higher pitch); while when analyzing depression, the weights of video features (such as stiff facial expressions and reduced body movements) are significantly increased. The system dynamically adjusts the attention weight matrix by calculating the mutual information between multimodal features, achieving accurate capture of different psychological states.

[0046] For example, when processing multimodal data containing text with anxiety keywords ("I've been feeling anxious and unable to sleep lately"), rapid speech (speech rate > 200 words / minute), videos with frequent frowning (AU4 intensity > 0.8), and calculation questions with a high error rate (accuracy < 60%), the model uses an attention mechanism to find that the text and video features have the highest correlation (weight 0.4), followed by speech (weight 0.3) and behavioral interaction (weight 0.3). The resulting fused feature vector accurately represents the "moderate anxiety" state.

[0047] The advantage of this technical solution is that it achieves deep semantic fusion of multimodal features through the Transformer architecture, dynamically learns the importance weights of each modality in different scenarios using the attention mechanism, effectively captures complementary information between multimodal data, improves the accuracy and robustness of psychological state analysis, and can still draw reliable conclusions through cross-modal verification, especially when the information of a single modality is unclear.

[0048] In some embodiments, the deep analysis results are implemented by constructing a cue word template using a large language model. The cue word template is used to guide the model to analyze the psychological mechanisms, cognitive patterns, and behavioral tendencies of the interview topic and output analysis results containing professional terminology. The analysis results are a component of the deep analysis results.

[0049] In some embodiments, the quantitative scoring of the behavioral interaction data includes calculating the accuracy rate of answers to multiple-choice and true / false questions; performing pattern analysis on the selection results of subconscious projection card interactions and matching questions to generate feature vectors that reflect the cognitive preferences or emotional tendencies of the interviewees, and the feature vectors are used for semantic fusion of the multimodal data.

[0050] In some embodiments, the method is implemented by a system deployed on a cloud server, a local server, or an edge computing device, wherein the system employs Docker containerization technology for environment isolation and uses encryption algorithms to encrypt the transmission and storage of the multimodal data.

[0051] In some embodiments, the personalized recommendations in the comprehensive assessment report are generated by a personalized analysis engine. The personalized analysis engine combines the basic information of the interviewee with the results of the in-depth analysis to provide intervention strategies adapted to individual characteristics. The intervention strategies include emotion regulation programs, career planning suggestions, or cognitive training programs.

[0052] Specifically, in the implementation of step S40, the personalized analysis engine first constructs a multi-dimensional feature space containing basic information such as age, gender, occupation, and educational background, along with in-depth analysis results such as psychological state characteristics and cognitive pattern profiles. A clustering algorithm maps users to specific psychological type classifications. The engine incorporates knowledge graphs from multiple fields, including cognitive behavioral therapy, career development planning, and neuroplasticity training. Based on the semantic correlation between user feature vectors and knowledge graph nodes, an initial set of intervention strategies is generated. Through a reinforcement learning mechanism, the system dynamically adjusts the weight parameters of each strategy based on the implementation effects of historical cases. For example, for users with anxiety tendencies, mindfulness training modules are recommended preferentially over cognitive restructuring techniques. The final output intervention strategy adopts a modular structure, including hierarchical planning for short-term goals (emotion regulation), medium-term goals (cognitive optimization), and long-term goals (career transition). This is then transformed into an executable action plan using natural language generation technology, achieving an automated closed loop from diagnostic results to intervention plans.

[0053] The following provides a detailed description of the automated qualitative analysis multimodal interview assessment method proposed in the embodiments of the present invention: 1. Multimodal interview data collection phase During the interviews, the system automatically collected and fully recorded diverse types of data using intelligent methods, constructing a comprehensive information collection system. Text data includes complete transcripts of the interview dialogues, providing foundational corpus for subsequent semantic analysis; voice data not only includes the voice content itself but also meticulously captures paralinguistic features such as tone of voice, speech rate changes, and emotional expression, providing a basis for emotion recognition and expression style analysis; video data uses visual sensors to record subtle changes in facial expressions, body movements, and posture characteristics, enabling the visual capture of nonverbal communication information; and behavioral interaction data includes standardized answer data from multiple-choice, true / false, and calculation questions, as well as process data from open-ended interactive formats such as OH cards and online connections, comprehensively recording the interviewee's cognitive responses and behavioral patterns, laying a data foundation for multi-dimensional analysis.

[0054] 2. Interview Process Control Stage This phase utilizes the technology described in Chinese Patent Publication No. CN119166782A, "A Dialogue Control Method and System Based on an Artificial Intelligence Dialogue Model," to construct an intelligent interview process management mechanism. Through the natural language processing and dialogue management capabilities of the large-scale model, the system can automatically guide multi-round, multi-topic interviews, dynamically adjusting dialogue strategies based on interviewees' responses. This ensures smooth transitions between topics and logical coherence, preventing topic breaks or logical contradictions. Simultaneously, through pre-set process rules and data verification mechanisms, the system monitors the completeness of interview data collection in real time, ensuring no omissions or deficiencies in text, voice, video, and behavioral interaction data. This provides a rich and standardized data source for subsequent analysis, guaranteeing interview quality from a process perspective.

[0055] 3. Automated Qualitative Analysis Stage The system leverages a pre-trained large language model as its core driving force to construct a multi-layered, multi-dimensional qualitative analysis framework. First, in the interview content encoding stage, natural language processing technology automatically extracts core concepts and keywords from the text data. Combined with voice emotion recognition and interactive data features, this forms a structured encoding system encompassing semantic, emotional, and behavioral attributes, achieving standardized transformation of unstructured data. Second, the topic extraction and clustering module, based on the encoded data, uses topic modeling algorithms to identify the core and secondary topics expressed by interviewees. Semantic similarity calculations are used to cluster these topics, clearly presenting the logical structure of the interview content. The deep semantic analysis stage utilizes the deep semantic understanding capabilities of the large model to uncover implicit psychological patterns, cognitive styles, emotional stability, and risk preferences within the interview content, achieving a mapping analysis from surface language to underlying psychological states. Finally, through cross-validation and multimodal data fusion mechanisms, the system performs cross-modal comparisons and verifications of voice emotion recognition results, facial expression analysis conclusions, and multiple-choice question answer data. This eliminates analytical biases from single-modal data, ensuring the consistency and reliability of analytical conclusions and constructing a comprehensive and multi-dimensional qualitative analysis framework.

[0056] 4. Automated assessment report generation stage Based on qualitative analysis results, the system automatically generates a structured comprehensive assessment report using pre-set evaluation models and algorithms. The qualitative description section focuses on the interviewee's psychological characteristics, cognitive style, and emotional expression, detailing their inner psychological state and behavioral patterns in natural language. The quantitative scoring module quantifies key dimensions of the psychological or cognitive state according to pre-defined evaluation criteria, generating a quantifiable and comparable indicator system to provide data support for objective evaluation. The risk warning function automatically scans for drastic emotional fluctuations, cognitive logical contradictions, or behavioral abnormalities during the interview process using pre-set abnormal behavior identification rules and models, marking and generating risk warnings in real time to provide a basis for subsequent intervention decisions.

[0057] like Figure 6 As shown, an automated qualitative analysis and evaluation system for multimodal interviews based on a large model includes: The multimodal data acquisition module is configured to automatically collect and store the multimodal data of the interviewee during the interview process. The multimodal data includes text data, audio and speech data, video data, and behavioral interaction data. The data preprocessing module is configured to preprocess the multimodal data to generate a standardized multimodal dataset; The interview process control module is configured to dynamically guide the interview process using a pre-configured interview structure and a large artificial intelligence model, adjust the dialogue process based on real-time feedback, and generate logical trajectory data for interview process control. The multimodal data analysis module is configured to process the standardized multimodal dataset based on a pre-trained large language model, including automatic text data encoding, topic clustering analysis, cross-validation of multimodal data, and semantic fusion, to generate in-depth analysis results. The assessment report generation module is configured to generate a comprehensive assessment report based on the in-depth analysis results, including qualitative descriptions, quantitative scores, and risk warnings, providing a basis for mental health assessments or cognitive ability tests.

[0058] like Figure 7 As shown, in some embodiments, the multimodal data acquisition module includes: The text data acquisition unit is configured to convert interview speech into text using real-time speech recognition technology to acquire text data. The audio speech processing unit is configured to extract acoustic features such as sound fundamental frequency, spectral features, speech rate and pause patterns, and generate emotional state indicators through a speech emotion recognition model to obtain audio speech data. The video data processing unit is configured to recognize facial motion units, eye contact patterns, micro-expressions, and the frequency and amplitude of body movements to acquire video data. The behavioral interaction data collection unit is configured to collect the results of multiple-choice questions, true / false questions, calculation questions, subconscious projection card interactions, and line-matching interactions to obtain behavioral interaction data.

[0059] In some embodiments, the multimodal data analysis module includes: The text encoding submodule is configured to automatically encode text data using encoding prompt word templates built from a large language model, and extract the encoding results of keywords, emotional features, and psychological state features. The topic clustering submodule is configured to perform semantic similarity calculation and clustering of coded labels (digital representations of keywords, emotional features, and psychological state features) using an unsupervised clustering algorithm to identify core topics and secondary topics; The cross-modal fusion submodule is configured to calculate cross-modal consistency scores to assess the consistency of data across modalities, mark outlier data, and generate in-depth analysis results that include psychological states, cognitive patterns, and emotional stability through semantic fusion.

[0060] In some embodiments, the evaluation report generation module includes: The quantitative scoring unit is configured to generate a quantitative score of psychological or cognitive state based on mental health scale scoring criteria or cognitive function assessment scale, combined with machine learning classification algorithms. The risk identification unit is configured to identify psychological abnormalities or cognitive impairment risks and generate risk warnings based on the cross-modal consistency score and preset abnormal state judgment rules.

[0061] In some embodiments, the system further includes: The feedback receiving module is configured to receive feedback data on the comprehensive evaluation report input from the outside. The model optimization module is configured to dynamically fine-tune and optimize the analysis parameters of the large language model based on the feedback data.

[0062] In some embodiments, the data preprocessing module includes: The data cleaning unit is configured to fill in missing values ​​and remove outliers from multimodal data to generate complete and clean data. The data standardization unit is configured to normalize or standardize the complete clean data, unify the data format and scale, and generate a standardized multimodal dataset. The data storage unit is configured to store the standardized multimodal dataset in a distributed database or a document-oriented database.

[0063] In some embodiments, the interview process control module includes: The structure definition unit is configured to define the interview structure through a preset JSON configuration file, which includes the order of interview topics, the types of interactive sessions, and the jump logic. The dynamic adjustment unit is configured to dynamically call the interview strategy in the JSON configuration file based on the real-time feedback from the artificial intelligence big data model and the interviewees, and adjust the interview pace and topic direction.

[0064] In some embodiments, the system further includes a coding reliability verification module, configured as follows: The same data segment is randomly sampled and submitted multiple times to the large language model for encoding, resulting in multiple independent encoding results. The consistency of the coding results is evaluated by calculating the Cohen-Kappa or Freis-Kappa values. When the consistency score is lower than a preset threshold, a manual review process is triggered.

[0065] In some embodiments, the system further includes a topic visualization module, configured to display the topic clustering results in two dimensions using visualization tools such as t-SNE, UMAP, or heatmaps, presenting the distribution characteristics and semantic associations of the label clusters.

[0066] In some embodiments, the cross-modal fusion submodule includes a Transformer architecture fusion model, configured as follows: Text features (keywords, emotion features, and psychological state features obtained through large language model encoding, vectorized representation), speech features (preprocessed acoustic features and emotion state indicators), video features (facial action unit encoding, micro-expression intensity, and quantified body movement features), and behavioral interaction features (quantified scoring results of behavioral interaction data) are unified into numerical vector representations. By dynamically learning the weight allocation of each modality feature through an attention mechanism, a fused feature vector containing multimodal semantic associations is generated for the generation of deep analysis results.

[0067] In some embodiments, the multimodal data analysis module further includes a deep semantic parsing module, configured to guide the model to analyze the psychological mechanisms, cognitive patterns and behavioral tendencies of the interview topics through prompt word templates constructed by a large language model, and output analysis results containing professional terms as a component of the deep analysis results.

[0068] In some embodiments, the behavior interaction acquisition unit includes an interaction scoring subunit, configured as follows: Calculate the accuracy rate of answers to multiple-choice and true / false questions; Pattern analysis is performed on the selection results of subconscious projection card interactions and matching questions to generate feature vectors that reflect cognitive preferences or emotional tendencies, which are then used for semantic fusion of multimodal data.

[0069] In some embodiments, the system is deployed on a cloud server, a local server, or an edge computing device, uses Docker containerization technology for environment isolation, and uses encryption algorithms to encrypt the transmission and storage of multimodal data.

[0070] In some embodiments, the assessment report generation module further includes a personalized suggestion engine, configured to combine the basic information of the interviewee with the results of in-depth analysis to generate intervention strategies adapted to individual characteristics, including emotion regulation programs, career planning suggestions, or cognitive training plans.

[0071] The following provides a detailed description of a corresponding system architecture based on an embodiment of the present invention, which specifically includes the following modules: The interview data acquisition module, serving as the system's data entry point, constructs a comprehensive, multimodal data acquisition network through four sub-modules. The text recording sub-module utilizes automatic speech recognition technology to convert interview speech into text in real time, simultaneously and completely recording the dialogue content, providing foundational corpus for subsequent semantic analysis. The speech analysis sub-module employs acoustic feature extraction algorithms to accurately capture emotional changes in speech, such as excitement, anxiety, and calmness, while analyzing acoustic features like speech rate and tone to uncover the emotional information behind the speech. The video analysis sub-module uses computer vision technology to perform real-time recognition and analysis of facial expressions and body language during the interview process, extracting nonverbal communication information from details such as micro-expression changes and gesture amplitude. The behavioral interaction recording sub-module focuses on automatically recording the results of interactive questions such as multiple-choice and true / false questions, and deeply analyzing open-ended interactive behaviors such as OH card interactions and online connections, ensuring complete data acquisition for all types of data.

[0072] The dialogue flow control module, built around a large-scale artificial intelligence model, establishes an intelligent interview guidance system. This module's built-in interview control system, based on the model's powerful natural language understanding and dialogue management capabilities, automatically guides the interview process according to pre-set interview objectives and the interviewee's real-time responses. During the interview, the system intelligently manages topic transitions, naturally extending to related fields based on the current topic, and promptly pulling back when it deviates from the theme, ensuring the dialogue always stays focused on the core objectives. Simultaneously, it flexibly adjusts the depth of the dialogue, delving into important topics to uncover key information, while appropriately simplifying secondary content to ensure an efficient interview pace. Through this automated flow control, the module ensures the fluency, logic, and goal achievement of the interview, providing a process guarantee for high-quality interview data collection.

[0073] The automated qualitative analysis module, through the collaborative operation of three main units, achieves in-depth analysis of multimodal data. The data preprocessing unit first cleans the collected raw data, removing noise and invalid information, and then unifies the data format and structure through normalization and standardization. The large-scale qualitative analysis engine is the core processing unit of the module. Its automatic coding unit uses natural language processing technology to extract core concepts, keywords, and emotional features from text, speech, and interactive data, generating structured codes. The topic clustering unit, based on the coded data, uses topic modeling algorithms to extract and cluster topics, revealing the logical structure of the interview content. The deep semantic analysis unit leverages the powerful natural language understanding capabilities of the large-scale model to uncover hidden psychological patterns, cognitive styles, and other deep semantic information behind the data. The multimodal data fusion unit integrates the analysis results from different data sources, eliminating biases from single-modal analysis through consistency verification and outlier detection, achieving complementarity and synergy of multimodal data, and ensuring the comprehensiveness and reliability of the qualitative analysis conclusions.

[0074] The automated assessment report generation module transforms qualitative analysis results into intuitive and practical assessment reports. Based on preset assessment dimensions, the report generation engine automatically generates structured reports containing qualitative descriptions, quantitative scores, and risk warnings. It uses clear and accurate language to describe the interviewee's psychological, cognitive, and emotional characteristics, presents their psychological or cognitive state intuitively through quantitative indicators, and provides timely warnings for abnormal situations. The report personalization engine, based on the individual characteristics of the interviewees and combined with the analysis results, provides targeted, personalized suggestions and interventions. For example, it offers emotion regulation strategies for individuals with significant mood swings, providing assessment services tailored to the actual needs of different users and enhancing the practicality and guidance of the reports.

[0075] The feedback and model optimization module is crucial for the system's continuous evolution. Based on user feedback during actual use and data analysis of system application effects, this module dynamically adjusts and optimizes the automated qualitative analysis model and algorithm parameters. By continuously collecting user feedback on the accuracy of evaluation results and the practicality of reports, and combining this feedback with real-world case studies, it identifies model shortcomings and areas for algorithm improvement. Simultaneously, through a dynamic optimization mechanism, it continuously adjusts model parameters and optimizes algorithm logic, enabling the system to adapt to different application scenarios and data characteristics. This ensures continuously improving evaluation result accuracy and drives continuous optimization and iterative upgrades of system performance.

[0076] Compared with the prior art, the advantages of the embodiments of the present invention are as follows: This invention proposes for the first time to automate qualitative analysis based on large models (such as the DEEPSEEK series), including automatic coding, topic extraction, and deep semantic analysis, thereby reducing the subjectivity and error of manual analysis and improving the consistency of results.

[0077] The embodiments of the present invention realize the integrated analysis of multimodal input of interview data, provide more comprehensive and reliable psychological assessment conclusions, and ensure the reliability and accuracy of the analysis results across multiple data dimensions.

[0078] The entire analysis process in this embodiment of the invention is automatically completed by a large model, significantly improving the efficiency of interview analysis, and is especially suitable for large-scale psychological testing and group research scenarios. The system has good scalability and can adapt to interview analysis tasks in different fields and with different needs.

[0079] This invention deeply integrates qualitative analysis theory from psychology and sociology with the latest AI technology to ensure the scientific validity and professional acceptance of the evaluation results, while also facilitating in-depth application by professionals in the field.

[0080] The multimodal interview assessment method proposed in this invention, with its multimodal data acquisition, intelligent process control, and in-depth qualitative analysis capabilities, is widely applicable to diverse assessment scenarios: In the field of cognitive ability assessment, it can achieve systematic detection of core cognitive functions such as memory and logic through early cognitive impairment screening; in employability assessment, it combines career counseling and career ability testing to provide job seekers with comprehensive analysis including dimensions such as cognitive style and risk preference, assisting in precise career planning; for intimate relationship assessment, it identifies emotional transmission and behavioral response characteristics in communication through multimodal data acquisition and analysis of interpersonal relationship interaction patterns, providing a scientific basis for improving relationships; in psychological state and risk assessment scenarios, it can capture non-verbal signals such as tone of voice and facial expressions of emotion and stress in real time, and combine them with interactive question-and-answer data to achieve intelligent identification and early intervention of mental health risks, comprehensively covering the diverse needs from cognitive function testing to complex psychological state assessment.

[0081] The following examples illustrate the above technical solutions in detail. The specific implementation of the multi-modal interview assessment system of this invention includes the following main steps: Step 1: Multi-mode data acquisition and preprocessing.

[0082] The system interfaces with data acquisition devices through standardized interfaces to obtain multimodal data from interviews: text data is transcribed using real-time automatic speech recognition technology (speech recognition model) to generate real-time text records, with data format including timestamps, interviewee numbers, and dialogue content; speech data records speech signals in real time, and uses acoustic signal processing algorithms such as openSMILE and Librosa libraries to extract fundamental frequency, MFCC spectral features, speech rate and pause patterns, and probability scores of speech emotions (anxiety, depression, etc.); video data records facial expressions and body language using cameras and video capture devices, and analyzes facial movement units, eye contact, micro-expressions, and the frequency and amplitude of body movements using deep learning models such as OpenFace and MediaPipe; interactive behavior data is automatically stored and labeled with data from interactive segments such as multiple-choice questions, calculation problems, OH cards, and matching questions. In the data preprocessing stage, cleaning, missing value imputation, and outlier removal are performed first, and then normalization and standardization are applied to unify the format and scale, finally storing the data in the MongoDB multimodal data storage system to form a complete multimodal dataset.

[0083] Here is an example of a data format: { "time_stamp": "2025-03-28T10:15:30Z", "participant_id": "Interviewee ID", Transcript: "I've been feeling a lot of stress lately, and I haven't been sleeping well." }

[0084] Step 2: Automated control of the interview process.

[0085] Based on "a dialogue control method and system based on an AI-powered dialogue model," this system automatically controls the dialogue process through a pre-set interview structure (e.g., a JSON configuration file). It dynamically adjusts the interview pace and topic direction based on real-time user feedback, automatically records interview data at each stage, and performs logical judgments to determine whether to proceed to the next topic. An example of the process control is as follows: starting with a global opening, proceeding to Topic 1, conducting Interactive Test 1 (e.g., OH card selection), answering questions for Topic 1, automatically judging user feedback, proceeding to Topic 2, and so on until the interview automatically ends and data is recorded, ensuring the logical consistency and goal orientation of the interview process.

[0086] I. Interview Data Auto-coding Module (Automated Open Coding) This module achieves automated open coding based on a pre-trained large language model (such as Deepseek): First, it uses sentence segmentation models such as SpaCy and NLTK to segment the interview dialogue text into semantic units such as sentences or phrases; then, it constructs an automatic coding prompt word template for each semantic unit, such as "Please identify key concepts or labels such as psychological characteristics and cognitive patterns from the following interview text and output them in the form of a list of professional terms"; the large model automatically outputs structured coding results based on the prompt, for example, encoding "I feel particularly stressed at work recently..." as ["work stress", "insomnia", "anxiety"]; finally, it adopts a cross-validation mechanism, submitting the same data fragment to the model multiple times by randomly sampling and calculating the Cohen's Kappa value to perform consistency checks to ensure the reliability of the coding results.

[0087] II. Core Coding and Theme Extraction Module The module achieves automatic axis encoding and topic extraction through the following steps: It uses pre-trained deep semantic models such as Sentence-BERT to perform word embedding representation on automatically generated tags and calculates the semantic similarity matrix between tags; it uses unsupervised clustering algorithms such as DBSCAN and K-means to cluster the embedded tags to discover core topics. For example, "work pressure," "frequent overtime," and "job burnout" are clustered into the "job pressure and burnout topic"; another cluster topic, composed of tags such as "anxiety," "difficulty falling asleep," and "depression," is automatically named "sleep problems and negative emotions topic" by the system. The topic clustering is visualized in two dimensions using visualization tools such as t-SNE or UMAP, facilitating expert review of the clustering quality.

[0088] III. Selective Coding and Deep Semantic Analysis Module Deep semantic analysis is performed based on topic clustering results: a professional deep prompt template is designed, such as "Given the following interview excerpt topics, please analyze the psychological mechanisms, cognitive styles, and possible behavioral response patterns exhibited by the interviewee"; the large model automatically generates deep analysis results based on the prompt, for example, for the topic of "occupational stress and burnout", the analysis finds that the psychological mechanism is chronic stress induced by continuous high pressure, the cognitive pattern is negative cognitive bias, and the behavioral tendency is negative coping and risk of job loss; cross-validation is performed by integrating the analysis results of text, voice emotion, and video action, for example, when the text shows high stress, the voice shows anxiety and tension, and the video action shows fatigue, the characteristics of "high stress and high risk" are confirmed.

[0089] The in-depth analysis results are automatically generated in technical language, as shown in the following example: { Subject: Occupational Stress and Burnout "Psychological Mechanism": "A sustained high-pressure environment induces a long-term stress response mechanism, which may lead to chronic stress and burnout syndrome." "Cognitive pattern": "Exhibits negative cognitive biases, perceiving work tasks as threatening and uncontrollable." "Behavioral tendencies": "May adopt negative coping strategies (such as avoidance, delaying task completion), and there is a tendency to leave the company or a risk of declining performance."

[0090] IV. Multimodal Data Fusion Analysis and Consistency Verification Module The module implements multimodal data fusion in the following steps: By using the Transformer architecture cross-modal fusion model, text features (semantic labels, topics), speech features (speech emotion scores), and video features (action units and micro-expression scores) are unified into numerical vector representations to complete feature-level fusion; the system calculates cross-modal consistency scores (such as cosine similarity) to evaluate the consistency of each modality. When the score is lower than the threshold, the data is automatically marked as abnormal and manual review is recommended to ensure the reliability of the analysis results.

[0091] V. Quantitative and Qualitative Evaluation Module for Automated Qualitative Analysis Results This module automatically converts the results of qualitative analysis into standardized quantitative and qualitative indicators, including the following steps: In the step of automating the generation of qualitative results, the system automatically generates a professional psychological assessment report, including a detailed qualitative description of the interviewee's psychological, cognitive, and behavioral characteristics.

[0092] In the automated calculation and generation of quantitative indicators, the focus is on converting the structured labels output from automated qualitative analysis into two types of quantifiable scores, thus achieving automated calculation and generation of quantitative indicators. The seven-point scale uses 0 as the normative mean, constructing a scoring system from -3 to 3, with lower values ​​indicating a worse state. Through semantic weight analysis and dimensional mapping of the structured labels, a pre-defined algorithm model automatically completes the conversion from labels to specific scores, ensuring that the score distribution conforms to normative characteristics. The five-point scale sets a scoring range of 1 to 5, with lower values ​​indicating a more normal state, and higher values ​​corresponding to a quantitative increase in the risk of abnormalities such as depression and anxiety. This system establishes a mapping relationship between labels and abnormal risk levels, combining parameters such as the frequency and intensity of label occurrences, and uses an automated calculation model to generate corresponding scores, providing intuitive quantitative evidence for risk assessment. Both scoring systems rely on automated data processing workflows to achieve efficient conversion from qualitative labels to quantitative indicators, meeting the quantitative analysis needs of different scenarios.

[0093] The automatic calculation and generation steps for the above quantitative indicators specifically include the following steps: The tag feature vectorization step, for each interview segment and its corresponding multimodal fragment, outputs a tag (topic ID, semantic strength s, sentiment polarity e, frequency / duration f) from the qualitative module. A feature vector of length n is generated for all tags. ,in α, β, and γ are pre-calibrated weights.

[0094] The dimension-item mapping step constructs a "label-factor dictionary" D, where experts annotate to determine which quantitative factor each label belongs to (e.g., low mood, sleep quality, social withdrawal, etc.). Assume there are K target quantitative dimensions, with each dimension j corresponding to a set of label indices Iⱼ.

[0095] The original score calculation steps involve first calculating a weighted sum for any dimension j: ,in For bias terms, These are the model weights. All weights were obtained through iterative training on paired “interview-synchronization scale” data using regularized least squares.

[0096] The scaling and interval clipping steps include: seven-point conversion (–3…3), Score7=ROUND(MIN(MAX(a7·Raw+b7,–3),3)), and five-point conversion (1…5), Score5=ROUND(MIN(MAX(a5·Raw+b5,1),5)), where a7, b7, a5, and b5 are linear stretching-translation coefficients, uniformly calibrated by the normal sample distribution.

[0097] In the automatic correction step for reverse entries, if the j-th dimension has a reverse meaning (the higher the score, the more normal), Rev7 = –Score7 (seven-point scale) and Rev5 = 6 –Score5 (five-point scale).

[0098] In the dimensional aggregation and index output steps, the scores of several single dimensions are summarized into a comprehensive index (e.g., the "Psychological Stress Index PSI"). Weighted summation or principal component analysis can be used. This section only demonstrates linear weighting: PSI = Σ_kv_k·Score7_k (or Score5_k), where v_k is the weight determined by experts or data-driven learning. The system simultaneously retains both single-dimensional scores and the comprehensive index for use in reports from different perspectives.

[0099] All training weights The model is calibrated once during the initial modeling phase, and then automatically fine-tuned based on the newly added "interview-scale" control data on a quarterly basis and subsequent data sample collection after deployment.

[0100] Step 3: Dynamic Optimization of Model and System The system provides an automatic feedback optimization mechanism, allowing users and experts to manually mark the correctness of analysis results, with feedback data entering the database. Based on feedback data such as anomaly markers, the system automatically triggers model fine-tuning and optimization. A monthly optimization cycle is set to continuously improve the accuracy and reliability of the evaluation model, ensuring continuous improvement in system performance.

[0101] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the methods described above. If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0102] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method provided by the present invention. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the computer system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0103] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0104] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for automated qualitative analysis and evaluation based on large-scale multimodal interviews, characterized in that: The following steps are involved: S10: Automatically collect and store multimodal data of the interviewees during the interview process. The multimodal data includes text data, audio and speech data, video data, and behavioral interaction data. Preprocess the multimodal data to obtain a standardized multimodal dataset. S20: Utilizes pre-configured interview structures and large-scale artificial intelligence models to dynamically guide the interview process, automatically adjusting the direction of the interview topics and the pace of the conversation based on real-time feedback from the interviewees and predefined interview strategies. S30: Based on a pre-trained large language model, the text data in the standardized multimodal dataset is automatically encoded, and the encoding results, including keywords, emotional features, and psychological state features, are extracted; the encoding results are subjected to topic clustering, and the topic clustering results are identified; the topic clustering results, the acoustic features and speech emotion analysis results of the audio speech data, the facial expression and body movement feature analysis results of the video data, and the quantitative scoring results of the behavioral interaction data are cross-validated and semantically fused to generate in-depth analysis results including psychological state, cognitive pattern, and emotional stability; S40: Based on the results of the in-depth analysis, generate a comprehensive assessment report that includes qualitative descriptions, quantitative scores, and risk warnings. The comprehensive assessment report provides a basis for the psychological health assessment or cognitive ability assessment of the interviewees.

2. The method according to claim 1, characterized in that, In step S10, the text data is obtained by converting interview speech into text using real-time speech recognition technology; the audio speech data is obtained by extracting acoustic features including fundamental frequency, spectral features, speech rate, and pause patterns, and generating emotional state indicators through a speech emotion recognition model; the video data is obtained by recognizing facial movement units, eye contact patterns, micro-expressions, and the frequency and amplitude of body movements; the behavioral interaction data includes the results of multiple-choice questions, true / false questions, calculation questions, subconscious projection card interactions, and online connection interactions.

3. The method according to claim 1, characterized in that, In step S30, the automatic encoding is achieved by constructing encoding prompt word templates through a large language model; the topic clustering is achieved by performing semantic similarity calculation and clustering grouping on the encoded labels through an unsupervised clustering algorithm, and the encoded labels are digital representations of the keywords, the emotional features, and the psychological state features; the cross-validation of multimodal data evaluates the consistency of each modality by calculating cross-modal consistency scores, and abnormal data with consistency scores below the threshold are marked.

4. The method according to claim 3, characterized in that, In step S40, the quantitative scoring includes a mental health scale score or a cognitive function assessment scale score; the risk warning is based on the cross-modal consistency score and preset abnormal state judgment rules to identify the risk of abnormal mental state or cognitive impairment.

5. The method according to claim 1, characterized in that, It also includes the following steps: S50: receiving feedback data from the outside world on the comprehensive evaluation report; dynamically fine-tuning and optimizing the analysis parameters of the large language model based on the feedback data.

6. The method according to claim 1, characterized in that, In step S10, the preprocessing of the multimodal data includes filling missing values ​​and removing outliers to obtain complete and clean data; the complete and clean data is normalized or standardized to unify the data format and scale, resulting in a standardized multimodal dataset; the standardized multimodal dataset is stored in a storage system, which uses a distributed database or a document-oriented database.

7. The method according to claim 1, characterized in that, In step S20, the automatic control of the interview process is defined by a preset JSON configuration file that includes the interview topic order, interaction type, and jump logic. The artificial intelligence big data model dynamically calls the interview strategy in the JSON configuration file based on the real-time feedback of the interviewee, and adjusts the interview pace and topic direction.

8. The method according to claim 3, characterized in that: The reliability test of the encoding results is achieved through a cross-validation mechanism, which includes randomly sampling the same data segment and submitting it to the large language model multiple times for encoding, resulting in multiple independent encoding results; evaluating the consistency of the multiple encoding results by calculating the Cohen-Kappa value or the Fris-Kappa value; and triggering a manual review process to correct the encoding results when the consistency score is lower than a preset threshold.

9. The method according to claim 3, characterized in that: The cross-validation and semantic fusion are implemented through a cross-modal fusion model based on the Transformer architecture. The cross-modal fusion model unifies text features, speech features, video features, and behavioral interaction features into a numerical vector representation. It dynamically learns the weight allocation of the text features, speech features, video features, and behavioral interaction features in different analysis scenarios through an attention mechanism, generating a fusion feature vector containing multimodal semantic associations for the generation of the deep analysis results. In step S30, the text data in the standardized multimodal dataset is automatically encoded based on the pre-trained large language model to obtain vectorized text features containing keywords, emotional features, and psychological state features. Acoustic features, including fundamental frequency, spectral features, speech rate and pause patterns, are extracted from the preprocessed audio speech data in step S10, and emotional state analysis is performed in combination with a speech emotion recognition model to obtain speech features. By performing facial motion unit encoding, eye contact pattern analysis, micro-expression intensity extraction, and quantification of limb movement frequency and amplitude on the preprocessed video data in step S10, video features are obtained. By quantifying and scoring the behavioral interaction data in step S30, behavioral interaction characteristics are obtained.

10. An automated qualitative analysis and evaluation system for multimodal interviews based on a large model, characterized in that, include: The multimodal data acquisition module is configured to automatically collect and store the multimodal data of the interviewee during the interview process. The multimodal data includes text data, audio and speech data, video data, and behavioral interaction data. The data preprocessing module is configured to preprocess the multimodal data to generate a standardized multimodal dataset; The interview process control module is configured to dynamically guide the interview process using a pre-configured interview structure and a large artificial intelligence model, adjust the dialogue process based on real-time feedback, and generate logical trajectory data for interview process control. The multimodal data analysis module is configured to process the standardized multimodal dataset based on a pre-trained large language model, including automatic text data encoding, topic clustering analysis, cross-validation of multimodal data, and semantic fusion, to generate in-depth analysis results. The assessment report generation module is configured to generate a comprehensive assessment report based on the in-depth analysis results, including qualitative descriptions, quantitative scores, and risk warnings, providing a basis for mental health assessments or cognitive ability tests.

Images