Man-machine cooperation method, device and equipment in psychotherapy

Abstract

The invention belongs to the technical field of large language models, and discloses a man-machine cooperation method, device and equipment in psychotherapy, which introduces an integrated modeling research normal form, proposes a calculation and evaluation framework of MI intervention quality in combination with an MI four-stage theory, and comprises MI quality, appropriate and improper MI strategies and related language clues. And the heuristic effect of the LLM in improving the task capability of motivating interviews (MI) is detected by combining customized CoT prompts and an RAG technology. The calculation evaluation framework provided by the invention provides a quantifiable evaluation framework for MI practice of human and generative AI, and reveals a complex relationship between a treatment strategy and an intervention effect; the ability and limitation of the LLM in the MI field are verified at the practice level, the prompt project based on man-machine cooperation has a remarkable effect on improving the LLM ecological effectiveness, and a technical path is provided for standardization of digital psychological health services and ethical risk prevention and control.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language models, and in particular relates to a method, device and equipment for human-computer collaboration in psychotherapy. Background Art

[0002] Systems such as TREAT (Training and Reporting Expert Assistant for Talk) and CORE-MI primarily simulate MI processes using predefined rules and templates. These systems can partially replicate the structured dialogue format of MI and enable standardized recording and feedback of intervention processes. However, these systems lack semantic understanding and flexible generation capabilities, making them difficult to adapt to complex emotional changes and individualized expressions, limiting their effectiveness in actual psychological intervention scenarios. Especially when dealing with multi-round nonlinear conversations and high empathy requirements, the system-generated content often appears mechanical and fails to meet the core concept of "people-oriented" MI.

[0003] Commercial mental health platforms such as Youper and Wysa integrate pre-trained language models like GPT-3 and BERT to provide services such as daily emotional support and cognitive restructuring, and attempt to introduce MI elements to enhance user engagement and intervention effectiveness. Although these products have initially demonstrated the scalability of LLMs in behavioral health, they are not optimized for MI training and lack systematic modeling of MI technical indicators (such as reflective listening and support for self-efficacy). Intervention performance is uneven, and there are risks such as language bias and inappropriate advice, which can trigger adverse effects, especially during periods of psychological vulnerability.

[0004] Most current research focuses on overall quality ratings or user satisfaction, but a structured evaluation system for core MI skills (such as strengthening language modification and avoiding confrontation) has yet to be established. This lack makes it difficult to quantitatively identify key behavioral manifestations in LLM outputs, hindering its application, certification, and standardization in evidence-based treatments.

[0005] Generic LLMs often exhibit hallucinations, over-direction, or disregard for clients' emotional cues during conversation generation. Existing systems lack transparent and interpretable feedback mechanisms during interventions, making it impossible to monitor and intervene in unexpected behaviors. This "black-box" generation severely limits the clinical credibility and deployment feasibility of these models in the mental health field. Summary of the Invention

[0006] In response to the problems existing in the prior art, the present invention provides a method, device and equipment for human-computer collaboration in psychotherapy.

[0007] The present invention is implemented as follows: a human-computer collaboration system for psychotherapy includes:

[0008] Data collection module, data processing module, predictive modeling module, explainable modeling module, and customization module;

[0009] A data collection module, connected to the data processing module, is used to use the created motivational interview dataset;

[0010] The data processing module is connected to the predictive modeling module and is used to classify and encode the category labels such as therapist strategies in the dataset and extract embedded representations for text features;

[0011] The predictive modeling module, connected with the interpretable modeling module, is used for predictive modeling and uses machine learning and deep learning techniques to develop a computational evaluation framework for assessing the intervention quality of LLMs in MI;

[0012] The explainable modeling module is connected with the customized module to identify key intrinsic indicators that affect MI through explainable modeling and automatically evaluate the LLM language behavior throughout the MI process;

[0013] The customized module, connected with the interpretable modeling module, is used for interpretable modeling, identifying key intrinsic indicators that affect the quality of MI intervention, and automating the evaluation of LLM language behavior throughout the MI process. It focuses on creating MI task-specific prompts and evaluating the effectiveness of the framework in improving LLM performance in MI tasks.

[0014] Furthermore, the data collection:

[0015] A motivational interviewing dataset was created using this method. The dataset contains 133 conversations between therapists and clients, totaling 9,699 conversation turns. The conversations were sourced from MI demonstration videos on YouTube and were transcribed and annotated by experts in the MI field. GPT-4 was used for text data enhancement to obtain an extended dataset containing 582 conversations and a total of 28,738 conversation turns.

[0016] Furthermore, the predictive modeling:

[0017] First, we manually verified the therapist strategies annotated by GPT-4 to ensure the authenticity and reliability of the annotated data, and counted the usage frequency of various strategies and included them in the initial feature set;

[0018] Then, LIWC analysis and pre-trained word embedding models were used to extract language clues from the conversation text, and each was used as an independent feature set for comparison. On this basis, machine learning and deep learning algorithms were further applied to construct a prediction model for MI intervention quality.

[0019] Finally, the McNemar test was used to compare the MI intervention quality of human experts and DeepSeek-V3 at the overall and different treatment topic levels.

[0020] Furthermore, the interpretable modeling:

[0021] First, we selected XGBoost, the optimal machine learning algorithm for the predictive modeling stage, and performed feature refinement and optimization in the REFRESH framework for different stages of MI. We then used the XGBoost-RFE-SHAP algorithm, combined with five-fold cross-validation, to retrain the model and dynamically adjust hyperparameters in each recursive round. We then iteratively optimized the feature subsets to build a refined predictive model.

[0022] The algorithm uses the cooperative game theory tool SHAP for global interpretation. It calculates the contribution of features to model prediction by taking the average absolute value of the SHAP values ​​of all samples, thereby assigning importance values ​​to features. This is then paired with the XGBoost algorithm for recursive feature elimination, achieving superior model performance with fewer feature combinations.

[0023] Then, based on the refined model, we conducted SHAP local interpretability analysis in conjunction with the XAI framework to quantify the cumulative contribution of intrinsic indicators and clarify their directional impact on the quality of MI intervention, thus forming a complete computational evaluation framework.

[0024] Finally, the paired sample t-test was used to compare the performance differences of intrinsic indicators between human experts and DeepSeek-V3 in each stage of MI, providing important basis for optimizing the MI intervention strategy of universal LLMs.

[0025] Furthermore, the evaluation framework:

[0026] The effectiveness of the computational evaluation framework in enhancing the MI performance of DeepSeek-V3 was evaluated through prompt engineering;

[0027] An MI knowledge base was constructed based on human MI conversation texts, the four-stage MI theory, therapist strategy definitions, and relevant knowledge in the MI operation manual. Under the guidance of the computational evaluation framework established in the explanatory modeling stage, based on the Few-shot-CoT prompt strategy, the RAG technology was used to optimize the knowledge retrieval process and develop a customized prompt framework for DeepSeek-V3. At the same time, the MI intervention quality performance of human therapists, the baseline DeepSeek-V3, and the DeepSeek-V3 optimized by prompt engineering were compared for different treatment topics, as well as the specific differences in intrinsic indicators in each MI stage.

[0028] Another object of the present invention is to provide a method for improving human-machine collaboration in psychotherapy, specifically comprising:

[0029] Step 1: The data collection module uses the created motivational interviewing dataset. The dataset is augmented with GPT-4 and contains 582 conversations between therapists and clients, totaling 28,738 conversation turns. The data processing module classifies and encodes the category labels such as therapist strategies in the dataset, and extracts embedded representations of the text features.

[0030] Step 2, develop a framework for evaluating LLMs in MI using machine learning and deep learning techniques through a predictive modeling module;

[0031] Step 3: Identify key intrinsic indicators that affect the quality of MI intervention through an interpretable modeling module and automatically evaluate LLM language behavior throughout the MI process;

[0032] Step 4, customized prompt development through customized modules focuses on creating MI task-specific prompts and evaluates the effectiveness of the framework in improving LLM performance.

[0033] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the human-computer collaboration improvement method in psychotherapy.

[0034] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method for improving human-computer collaboration in psychotherapy.

[0035] Another object of the present invention is to provide an information data processing terminal, which is used to implement the human-computer collaboration system in psychotherapy.

[0036] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0037] In view of the technical problems existing in the above-mentioned prior art and the difficulty of solving these problems, this paper closely combines the technical solutions to be protected by the present invention and the results and data during the research and development process, and analyzes in detail and in depth how the technical solutions of the present invention solve the technical problems and some creative technical effects brought about by solving the problems. The specific description is as follows:

[0038] Based on a dataset of 582 MI conversations, this paper introduces an integrative modeling research paradigm and proposes a computational assessment framework for MI intervention quality based on the four-stage MI theory. This framework includes MI quality, appropriate and inappropriate MI strategies, and related language cues. Furthermore, customized CoT prompts and RAG techniques are used to test their effectiveness in improving LLMs' ability in motivational interviewing (MI) tasks. The results show that the overall MI intervention quality of general LLMs, such as DeepSeek-V3, is significantly lower than that of human therapists (χ 2 =23.15, p<0.001). However, LLMs (e.g., DeepSeek-V3) with customized CoT prompts showed no significant difference in average intervention effect compared to human therapists across all treatment topics (p>0.05). A phased analysis showed that LLMs outperformed baseline models and humans in reflective listening and analytical thinking during the engagement phase, but still exhibited deficiencies in equal collaboration (e.g., limited communication vocabulary) and MIIN behavioral restraint. Reducing warnings about sensitive topics during the focusing phase reduced defensiveness, but overuse of authoritative expressions and complex vocabulary weakened logical focus. Reducing conversational certainty through qualifiers during the evocative phase preserved client autonomy, but insufficient emotional reflection limited self-efficacy. These findings validate the potential of MI strategies and psycholinguistically driven prompt engineering to enhance the ecological validity of LLMs, but further optimization is needed to balance guidance and client autonomy. These findings reveal the capabilities and limitations of LLM in complex psychological interactions, provide a verifiable technical path for the standardization, ethical risk prevention and control, and safety assessment of digital mental health services (such as therapeutic chatbots), and promote the coordinated development of AI and evidence-based psychotherapy based on human-computer collaboration. This paper systematically examines the capabilities and limitations of large language models (LLMs) in motivational interviewing (MI) tasks by constructing a computational evaluation framework based on 582 MI conversations. On a theoretical level, the computational evaluation framework proposed in this paper provides a quantifiable evaluation framework for the MI practice of humans and generative AI, revealing the complex relationship between treatment strategies and intervention effects; on a practical level, it verifies the capabilities and limitations of LLM in the field of MI, and demonstrates the significant role of prompt engineering based on human-computer collaboration in improving the ecological validity of LLM, providing a technical path for the standardization and ethical risk prevention and control of digital mental health services. In addition, the integrative modeling research paradigm adopted provides a methodological reference for understanding and promoting AI behavior. Future research needs to further optimize the performance of LLM in the dimensions of emotional regression and equal cooperation, and at the same time, combine multimodal data with dynamic feedback mechanisms to explore evidence-based psychotherapy models of human-computer collaboration to ensure the safety and explainability of AI interventions and full respect for client autonomy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1This is a structural block diagram of a system for computing and evaluating the capabilities of a large language model in motivational interviews, provided by an embodiment of the present invention.

[0040] Figure 2 This is a flow chart of a method for improving a human-machine collaboration system in psychotherapy provided by an embodiment of the present invention.

[0041] Figure 3 It is a research method and experimental flow chart provided by the embodiment of the present invention.

[0042] Figure 4 This is a performance graph of the top 20 most important features and optimal feature combinations of different numbers provided by an embodiment of the present invention.

[0043] Figure 5 SHAP diagram of the interpretability of MI intervention quality provided by an embodiment of the present invention.

[0044] Figure 6 This is a comparison chart of the proportion of high-quality conversations between DeepSeek-V3 and human therapists on different treatment topics provided by an embodiment of the present invention.

[0045] Figure 1 In: 1. Data collection module; 2. Predictive modeling module; 3. Explainable modeling module; 4. Customization module. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] like Figure 1 As shown, an embodiment of the present invention provides a human-computer collaboration system for psychotherapy, including:

[0048] Data collection module 1, predictive modeling module 2, interpretable modeling module 3, customization module 4;

[0049] Data Collection Module 1, connected to Predictive Modeling Module 2, uses the created motivational interviewing dataset; the dataset contains 133 conversations between therapists and clients, totaling 9,699 conversational turns. The conversations were collected from MI demonstration videos on YouTube and transcribed and annotated by experts in the MI field.

[0050] Data collection module, data processing module, predictive modeling module, explainable modeling module, and customization module;

[0051] The data collection module, connected to the data processing module, is used to use the created motivational interviewing dataset; the dataset contains 133 conversations between therapists and clients, totaling 9,699 conversation turns. The conversations are derived from MI demonstration videos on the YouTube platform and have been transcribed and annotated by experts in the MI field;

[0052] The data processing module is connected to the predictive modeling module and is used to classify and encode the category labels such as therapist strategies in the dataset and extract embedded representations for text features;

[0053] The predictive modeling module, connected with the interpretable modeling module, is used for predictive modeling and uses machine learning and deep learning techniques to develop a computational evaluation framework for assessing the intervention quality of LLMs in MI;

[0054] The explainable modeling module is connected with the customized module to identify key intrinsic indicators that affect MI through explainable modeling and automatically evaluate the LLM language behavior throughout the MI process;

[0055] The customized module, connected with the interpretable modeling module, is used for interpretable modeling, identifying key intrinsic indicators that affect the quality of MI intervention, and automating the evaluation of LLM language behavior throughout the MI process. It focuses on creating MI task-specific prompts and evaluating the effectiveness of the framework in improving LLM performance in MI tasks.

[0056] Furthermore, the data collection:

[0057] A motivational interviewing dataset was created using this method. The dataset contains 133 conversations between therapists and clients, totaling 9,699 conversation turns. The conversations were sourced from MI demonstration videos on YouTube and were transcribed and annotated by experts in the MI field. GPT-4 was used for text data enhancement to obtain an extended dataset containing 582 conversations and a total of 28,738 conversation turns.

[0058] Furthermore, the predictive modeling:

[0059] First, we manually verified the therapist strategies annotated by GPT-4 to ensure the authenticity and reliability of the annotated data, and counted the usage frequency of various strategies and included them in the initial feature set;

[0060] Then, LIWC analysis and pre-trained word embedding models were used to extract language clues from the conversation text, and each was used as an independent feature set for comparison. On this basis, machine learning and deep learning algorithms were further applied to construct a prediction model for MI intervention quality.

[0061] Finally, the McNemar test was used to compare the MI intervention quality of human experts and DeepSeek-V3 at the overall and different treatment topic levels.

[0062] Furthermore, the interpretable modeling:

[0063] First, we selected XGBoost, the optimal machine learning algorithm for the predictive modeling stage, and performed feature refinement and optimization in the REFRESH framework for different stages of MI. We then used the XGBoost-RFE-SHAP algorithm, combined with five-fold cross-validation, to retrain the model and dynamically adjust hyperparameters in each recursive round. We then iteratively optimized the feature subsets to build a refined predictive model.

[0064] The algorithm uses the cooperative game theory tool SHAP for global interpretation. It calculates the contribution of features to model prediction by taking the average absolute value of the SHAP values ​​of all samples, thereby assigning importance values ​​to features. This is then paired with the XGBoost algorithm for recursive feature elimination, achieving superior model performance with fewer feature combinations.

[0065] Then, based on the refined model, we conducted SHAP local interpretability analysis in conjunction with the XAI framework to quantify the cumulative contribution of intrinsic indicators and clarify their directional impact on the quality of MI intervention, thus forming a complete computational evaluation framework.

[0066] Finally, the paired sample t-test was used to compare the performance differences of intrinsic indicators between human experts and DeepSeek-V3 in each stage of MI, providing important basis for optimizing the MI intervention strategy of universal LLMs.

[0067] Furthermore, the evaluation framework:

[0068] The effectiveness of the computational evaluation framework in enhancing the MI performance of DeepSeek-V3 was evaluated through prompt engineering;

[0069] An MI knowledge base was constructed based on human MI conversation texts, the four-stage MI theory, therapist strategy definitions, and relevant knowledge in the MI operation manual. Under the guidance of the computational evaluation framework established in the explanatory modeling stage, based on the Few-shot-CoT prompt strategy, the RAG technology was used to optimize the knowledge retrieval process and develop a customized prompt framework for DeepSeek-V3. At the same time, the MI intervention quality performance of human therapists, the baseline DeepSeek-V3, and the DeepSeek-V3 optimized by prompt engineering were compared for different treatment topics, as well as the specific differences in intrinsic indicators in each MI stage.

[0070] Another object of the present invention is to provide a method for improving human-machine collaboration in psychotherapy, specifically comprising:

[0071] Step 1: The data collection module uses the created motivational interviewing dataset. The dataset is augmented with GPT-4 and contains 582 conversations between therapists and clients, totaling 28,738 conversation turns. The data processing module classifies and encodes the category labels such as therapist strategies in the dataset, and extracts embedded representations of the text features.

[0072] Step 2, develop a framework for evaluating LLMs in MI using machine learning and deep learning techniques through a predictive modeling module;

[0073] Step 3: Identify key intrinsic indicators that affect the quality of MI intervention through an interpretable modeling module and automatically evaluate LLM language behavior throughout the MI process;

[0074] Step 4, customized prompt development through customized modules focuses on creating MI task-specific prompts and evaluates the effectiveness of the framework in improving LLM performance.

[0075] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the human-computer collaboration improvement method in psychotherapy.

[0076] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method for improving human-computer collaboration in psychotherapy.

[0077] Another object of the present invention is to provide an information data processing terminal, which is used to implement the human-computer collaboration system in psychotherapy.

[0078] The present invention is specifically implemented:

[0079] 1. This paper proposes an innovative human-machine collaborative evaluation framework to conduct a human-machine comparison and promotion study of the MI capabilities of LLMs. The research process includes three main steps: first, predictive modeling uses machine learning and deep learning techniques to develop a framework for evaluating LLMs in MI; second, interpretable modeling identifies key intrinsic indicators that affect MI and enables automated evaluation of LLM language behavior throughout the MI process; third, customized prompt development focuses on creating MI task-specific prompts to evaluate the effectiveness of the framework in improving LLM performance. In addition, this paper comprehensively compares the performance of LLMs with that of human experts through these three steps, and deeply analyzes the advantages and limitations of LLMs.

[0080] 2 Methods and Experiments

[0081] Based on the MI dataset constructed by transcribing and annotating MI demonstration videos on the YouTube platform (Section 2.1), this paper uses an integrated modeling approach to evaluate the MI performance of DeepSeek-V3 and verify the effectiveness of the computational evaluation framework centered on the quality of MI intervention. In the predictive modeling stage (Section 3.2), this paper inputs the effective feature set obtained from the preliminary feature screening into the machine learning and deep learning algorithms to obtain the optimal model to predict the quality of MI intervention responded by DeepSeek-V3. In the explanatory modeling stage (Section 3.3), combined with the four-stage theory of MI, this paper uses the XGBoost-RFE-SHAP algorithm in the XAI framework (aimed at improving the transparency and interpretability of machine learning models) (Belle & Papantonis, 2021) to identify key intrinsic indicators of different MI stages, thereby developing a computational evaluation framework. In the verification stage (Section 3.4), this study develops targeted prompting strategies for DeepSeek-V3 based on this framework to optimize its performance in MI tasks. Finally, a comprehensive comparison of the internal and external indicator performance of DeepSeek-V3 and human therapists was conducted, providing theoretical support and practical guidance for optimizing the performance of LLM in MI tasks.

[0082] 2.1 Data Acquisition

[0083] This paper uses the Motivational Interviewing (MI) dataset created by Wu Zixiu's team (Wu et al., 2022, 2023). This dataset contains 133 conversations between therapists and clients, totaling 9,699 conversational turns. These conversations are sourced from MI demonstration videos on YouTube and transcribed and annotated by experts in the MI field, ensuring the professionalism and reliability of the data. Given the limited data size and imbalanced distribution of this dataset (110 high-quality conversations and 23 low-quality conversations), to obtain natural and coherent augmented text and accurate annotation results, this paper uses GPT-4 for text data augmentation and annotation of therapist strategies and conversation phases. The resulting extended dataset consists of 582 conversations totaling 28,738 conversational turns (346 high-quality conversations and 236 low-quality conversations). This method can effectively augment unbalanced datasets, significantly improving the performance of classification models (Fang et al., 2023). The fields and their meanings of this dataset are shown in Table 1. Tables 2 and 3 present basic statistics of the augmented extended dataset. In therapist-client interviews, the Evoking phase had the most conversational turns (17.48 turns), while the Focusing phase had the longest average sentence length (15.29 words). In terms of strategy frequency, open-ended questions were the most common (42.6%), followed by complex reflection (13.2%).

[0084] Table 1 Fields and definitions included in the dataset

[0085]

[0086] Table 2 Descriptive statistics of the MI dataset

[0087]

[0088] Table 3 Descriptive statistics of MI strategy annotation

[0089]

[0090] 2.2 Predictive modeling to evaluate the MI effectiveness of DeepSeek-V3 In the predictive modeling stage, the present invention developed an accurate and robust predictive model to provide an objective, stable and efficient tool for evaluating the quality of DeepSeek-V3 in MI intervention. The model takes the quality of MI intervention as the prediction target, and uses the therapist's strategy usage frequency and language clues as well as the therapist and client as input features, and uses machine learning and deep learning algorithms to build a predictive model. Among them, the language clues of the therapist's conversation are generated through Linguistic Inquiry and Word Count (LIWC) analysis and word vector embedding of pre-trained models (BERT and RoBERTa). Finally, the optimal model is selected to evaluate the MI intervention quality of DeepSeek-V3.

[0091] First, the therapist strategies annotated by GPT-4 were manually verified to ensure the authenticity and reliability of the annotated data. The frequency of use of each strategy was counted and incorporated into the initial feature set. Next, linguistic cues from the conversation texts were extracted using both LIWC analysis and a pre-trained word embedding model. These were used as independent feature sets for comparison. LIWC analysis utilizes a predefined psychological lexicon to calculate the frequency of occurrence of different vocabulary categories to comprehensively assess emotional states, psychological traits, and social interaction patterns. It is a common method for studying the relationship between mental health and language (Alhazzaa & Curcin, 2024) and is widely used in the identification and treatment of mental illnesses such as depression (Shen et al., 2017; Tadesse et al., 2019) and anxiety (Coppersmith et al., 2015; Shen & Rudzicz, 2017). The present invention performs LIWC analysis on the conversation texts between therapists and clients to extract fine-grained linguistic features. On this basis, the Language Style Matching (LSM) between therapists and clients is calculated (Niederhoffer & Pennebaker, 2002). This quantifies the consistency of the language interaction between the two parties and clarifies its role as an implicit indicator of the therapeutic alliance in improving relationship quality and symptom improvement (Aafjes-vanDoorn et al., 2020). In addition, two pre-trained models, BERT and RoBERTa, are used to generate embedding vectors for the conversation texts. BERT and RoBERTa are based on the Transformer architecture and use bidirectional encoders to generate highly context-sensitive high-dimensional vector representations. They can effectively capture the semantics and nonlinear relationships between words and sentences (Greco et al., 2023), and have shown significant advantages in mental illness identification (Kim et al., 2020) and complex emotion capture (Tavabi et al., 2020). Next, we construct an initial feature set based on the extracted features, and calculate feature importance using the XGBoost "gain" method, which has low computational overhead (Wang et al., 2024). We then combine hyperparameter grid search and five-fold cross-validation (GridSearchCV) to gradually eliminate the least important features and select the feature subset that maximizes the ROC-AUC, thereby reducing model complexity and mitigating the risk of overfitting (Islam et al., 2023).Subsequently, the effective feature subsets were input into various classification algorithms, including five machine learning models such as Random Forest (RF) and Ridge Regression (Ridge), and three deep learning models such as Recurrent Neural Network (RNN) and Bidirectional Long Short-Term Memory (BiLSTM). Prediction models were constructed, and model performance was evaluated using five classification metrics: accuracy, precision, recall, F1 score, and ROC-AUC, combined with five-fold cross-validation. The best prediction model was used to evaluate the quality of DeepSeek-V3 in MI intervention. Finally, the McNemar test was used to compare the MI intervention quality of human therapists and DeepSeek-V3 at both the overall and different treatment theme levels to comprehensively analyze their differences and adaptability.

[0092] 2.3 Explanation Modeling to Evaluate DeepSeek-V3’s Psycholinguistic Performance in MI

[0093] Drawing on the four-stage MI theory proposed by Miller and Rollnick, this paper develops a computational evaluation framework centered on MI intervention quality using an interpretable machine learning algorithm based on the XAI framework. This framework is used to explain the specific impact of intrinsic indicators such as therapist strategies and psycholinguistic cues on the quality of MI intervention.

[0094] First, XGBoost, the optimal machine learning algorithm for the predictive modeling stage, is selected. Feature refinement and optimization are performed in the REFRESH framework (combining recursive feature elimination and SHAP value dynamic optimization feature selection) (Sharma et al., 2023) for different stages of MI. Specifically, the present invention uses the XGBoost-RFE-SHAP algorithm, combined with five-fold cross-validation, to retrain the model and dynamically adjust hyperparameters in each round of recursion, iteratively optimize the feature subset to build a refined predictive model. The algorithm uses the cooperative game theory tool Shapley Additive Explanations (SHAP) for global interpretation. The contribution of the feature to the model prediction is calculated by taking the average absolute value of the SHAP value of all samples, thereby assigning importance values ​​to the features (Lundberg & Lee, 2017). It is then paired with the XGBoost algorithm for recursive feature elimination (RFE), which can achieve excellent model performance with fewer feature combinations (Huang et al., 2024). Next, based on the refined model, we conducted a SHAP local interpretability analysis using the Explainable Artificial Intelligence (XAI) framework to quantify the cumulative contribution of intrinsic metrics and clarify their directional impact on the quality of MI intervention, forming a complete computational evaluation framework to improve the model's interpretability (Ullah et al., 2023). Finally, we used paired-sample t-tests to compare the performance of intrinsic metrics between human experts and DeepSeek-V3 at each MI stage, providing important insights for optimizing MI intervention strategies for general LLMs.

[0095] 2.4 Effectiveness of the computational evaluation framework in enhancing DeepSeek-V3’s MI performance through prompt engineering evaluation

[0096] This paper constructs an MI knowledge base based on relevant knowledge such as human MI conversation texts, MI four-stage theory, therapist strategy definitions, and MI operation manuals. Under the guidance of the computational evaluation framework established in the explanatory modeling stage, based on the Few-shot-CoT prompt strategy, the RAG technology is used to optimize the knowledge retrieval process and develop a customized prompt framework for DeepSeek-V3 to optimize its performance in MI tasks. The Few-shot-CoT strategy uses multi-step reasoning technology to guide the model to generate reasoning paths, which can significantly improve the model's completion accuracy in complex tasks while improving the model's generalization ability (Wei et al., 2022). In addition, the introduction of external knowledge sources for enhanced retrieval can effectively improve the diversity and richness of generated text (Lewis et al., 2020) while reducing the generation of hallucinations (Béchard & Ayala, 2024). Next, we compared the MI intervention quality performance of human experts, the benchmark DeepSeek-V3, and the hint-engineered optimized DeepSeek-V3 applied to different treatment topics, as well as the specific differences in intrinsic indicators in each stage of MI, to verify the effectiveness of the computational evaluation framework in improving the MI performance of DeepSeek-V3.

[0097] 2.5 Research methods and experimental procedures

[0098] The research framework of this invention is as follows Figure 3 This study was based on the MI dataset created by Wu Zixiu's team, using GPT-4 for data augmentation and annotation to create a balanced, expanded dataset. During the predictive modeling phase, language features were first extracted using LIWC analysis and a pre-trained model. An initial feature set was constructed based on the therapist's strategy usage frequency and the language style match between the therapist and the client. Initial feature screening was performed using the XGBoost algorithm, and valid features were input into various classification models. Model performance was evaluated using classification metrics. Finally, the optimal model was used to predict extrinsic indicators of DeepSeek-V3 responses, and the McNemar test was used to quantify DeepSeek-V3's performance compared to human experts.

[0099] In the explanatory modeling stage, the XGBoost-RFE-SHAP algorithm was used to streamline the feature set and extract key features of different MI stages. SHAP was then used to perform local explanatory analysis to construct a computational evaluation framework centered on the quality of MI intervention.

[0100] Finally, the performance of DeepSeek-V3 and human experts on internal and external indicators were compared by paired sample t-test to evaluate the ability of DeepSeek-V3 in MI tasks.

[0101] During the validation phase of the evaluation framework, a targeted hinting framework was developed for DeepSeek-V3 using the Few-shot CoT hinting strategy. This was combined with RAG technology to optimize knowledge retrieval, thereby optimizing its performance on MI tasks. Finally, statistical methods were used to compare the internal and external performance of DeepSeek-V3 against human experts, the baseline DeepSeek-V3, and the DeepSeek-V3 optimized with customized CoT hints to verify the effectiveness of the computational evaluation framework.

[0102] 3 Results

[0103] This section will sequentially describe the results of predictive modeling and explanatory modeling, and on this basis systematically introduce the development and effectiveness verification of a comprehensive computational evaluation framework.

[0104] 3.1 Evaluation of DeepSeek-V3’s MI Effectiveness Based on a Prediction Model Focused on MI Intervention Quality

[0105] Using the effective feature subsets obtained through preliminary screening, this paper trained and evaluated multiple classification prediction models and selected the best-performing models to predict the quality of DeepSeek-V3 MI interventions. The performance of DeepSeek-V3 and human therapists was then compared overall and across different themes to assess the gap between DeepSeek-V3's capabilities and those of human therapists in MI interventions and its relative strengths in specific themes.

[0106] 3.1.1 Performance Analysis of MI Intervention Quality Prediction Model

[0107] Based on the results of feature engineering, the present invention constructed two sets of features for model training and evaluation. The first set of features focuses on language features and treatment strategies, including the frequency of therapist strategy use, LIWC indicators, and LSM between therapists and clients, a total of 127 initial features. After preliminary screening by the XGBoost algorithm, 48 valid features were finally obtained and input into the classification model. The second set of features introduced semantic embedding features, including the frequency of therapist strategy use and the conversation text embedding vectors generated by pre-trained models (BERT and RoBERTa) to capture deep semantic information. The performance indicators of various classification models are shown in Table 4. Among the five machine learning models, the XGBoost model based on LIWC language features performed best, with an accuracy of 0.9088 and a ROC-AUC of 0.9672. Among the three deep learning models, the Bilstm model based on LIWC performed best, with an accuracy of 0.7941 and a ROC-AUC of 0.8869. In comparison, the classification model based on pre-trained word embedding vectors performed poorly. Therefore, the LIWC-XGBoost model is used to evaluate the response quality of DeepSeek-V3, and the XGBoost algorithm is used for feature refinement and interpretability analysis.

[0108] Table 4 Classification prediction model performance indicators

[0109]

[0110] 3.1.2 Comparison of MI intervention quality between DeepSeek-V3 and human therapists

[0111] Based on the best-performing LIWC-XGBoost model, this paper predicts the MI intervention quality of DeepSeek-V3 responses and compares the MI performance of human therapists and DeepSeek-V3 using the McNemar test. In addition, the odds ratio (OR) is used to assess the relative probability of good or bad performance between the two groups of samples to further quantify the difference in intervention effect. The results are shown in Table 5. Overall, the proportion of high intervention quality of human therapists is 59.45%, which is significantly higher than that of DeepSeek-V3 at 46.56% (χ 2 =23.15, p<0.001, OR=1.89). Across specific treatment topics, except for lifestyle behavior and nutritional health treatments, DeepSeek-V3 performed insignificantly better than human therapists (p>0.05). However, the "Other" category showed a larger odds ratio, despite a nonsignificant difference. This finding addresses question 1.

[0112] Table 5 Comparison of MI intervention quality between human therapists and DeepSeek-V3

[0113]

[0114] Note: N represents the number of video samples on different topics; human therapists represents the number of samples rated as high-quality by human therapists in conversation videos on different topics and their proportion. For example, 333 (57.22%) means that among all video samples of human therapists, there are 333 high-quality videos, accounting for 57.22% of the total; DeepSeek-V3 represents the number of samples rated as high-quality by DeepSeek-V3 in conversation videos on different topics and their proportion. For example, 271 (46.56%) means that among all video samples of DeepSeek-V3, there are 271 high-quality videos, accounting for 46.56% of the total; χ 2 (McNemar test) is the statistic of the McNemar test, which is used to measure the significant difference between the classification results of two groups in paired samples; p represents the p-value of the McNemar test, reflecting the significance of the difference between the groups. A p-value less than 0.05 indicates a significant difference; The mark indicates that the p-value uses the exact McNemar test, which is used in small sample situations (b+c<10, that is, the number of samples with inconsistent classification results in the paired samples is less than 10) to improve the accuracy of the p-value; OR (95% CI) represents the odds ratio (Odds Ratio) and its 95% confidence interval. The odds ratio measures the relative probability of an event occurring in two groups of paired samples. An OR value greater than 1 indicates that the probability of high-quality conversations in the first group of samples is higher than that in the second group. An OR value less than 1 indicates that the probability of high-quality conversations in the first group of samples is lower than that in the second group. The confidence interval is used to estimate the exact range of the odds ratio. For example, 1.89[1.44,2.50] means that the odds ratio is 1.89, and the confidence interval of the odds ratio at a 95% confidence level is (1.44,2.50). 3.2 Evaluating DeepSeek-V3's MI Effectiveness Based on a Predictive Model Focused on MI Intervention Quality Based on the XAI framework, this paper uses the XGBoost-RFE-SHAP algorithm to refine features for different MI stages. Using SHAP for sensitivity analysis, we develop a comprehensive and efficient computational evaluation framework for MI intervention quality. This framework is used to quantitatively assess the performance of human therapists and DeepSeek-V3 at different stages of MI intervention. Finally, paired-sample t-tests were used to compare the performance of key intrinsic indicators between human therapists and DeepSeek-V3 at different MI stages, thereby quantitatively evaluating DeepSeek-V3's MI capabilities.

[0115] 3.2.1 Feature Refinement Based on Explainable Machine Learning

[0116] First, based on Miller and Rollnick's MI four-stage theory, the XGBoost algorithm is combined to perform recursive feature elimination, and the SHAP value is iteratively calculated to quantify the feature importance, and the model performance is gradually optimized. Specifically, the present invention uses the 48 features screened in the feature engineering stage and ranks them according to SHAP importance, such as Figure 4 (Figures A1, B1, C1, and D1) are shown. Feature pruning is then performed, and finally, a refined model with 17, 13, 20, and 11 features is obtained in the four stages of Engaging, Focusing, Evoking, and Planning, respectively. The feature screening process is shown in Figures A1, B1, C1, and D1. Figure 4 (As shown in Figures A2, B2, C2, and D2).

[0117] 3.2.2 Development of a computational assessment framework for MI intervention quality

[0118] In order to develop a comprehensive and objective computational evaluation framework to systematically evaluate the ability of DeepSeek-V3 in MI treatment intervention, this paper conducts local interpretability analysis based on Miller and Rollnick's four-stage MI theory and combines SHAP values ​​to explore the impact mechanism and relative importance of each feature in the refined model on the quality of MI intervention. First, according to the ranking of the feature contribution to the prediction model, the interpretability SHAP diagrams of different MI stages are drawn, such as Figure 5 As shown in (Figures a, b, c, d), this explains the differential effects of key language cues and therapist strategies on the quality of MI intervention at different MI stages.

[0119] As shown in Table 6, this paper draws on the core mechanisms of the MMI framework (Frey et al., 2021) and details the impact of intrinsic indicators (therapist's use of strategies and key language cues) on extrinsic indicators (MI intervention quality) across four dimensions: technical, relational, motivational interviewing inconsistent behaviors (MIIN), and other. During the engagement phase, therapists need to establish an open and comfortable atmosphere for the interview by building trust and collaboration. Characteristics positively correlated with MI intervention quality include simple reflection on the technical dimension, apostrophe and communication on the relational dimension, and preposition, money, and average words per sentence (WPS) on the other dimension. Conversely, time on the relational dimension, advice and attention on the MI inconsistent behaviors, and total word count (WC) and big words on the other dimension were negatively correlated with MI intervention quality. During the focusing phase, therapists and clients need to jointly identify the focus of the interview. Characteristics positively correlated with MI intervention quality include simple reflection (technical dimension), conjunctions (other dimensions), money (money), and average number of words per sentence (WPS). However, characteristics negatively correlated with MI intervention quality include therapist conversational authority (Clout) (relational dimension), advice (substances), health (health), and physical (physical) (MI inconsistent behaviors), prepositions (other dimensions), total word count (WC), and proportion of long words (Bigwords). During the evoking phase, therapists need to stimulate clients' intrinsic motivation for change and encourage the expression of change language.Characteristics positively correlated with MI intervention quality included complex reflection and simple reflection at the technical level, memory and determiners at the relational level, and average words per sentence (WPS) at the other level. Characteristics negatively correlated included exclamations, time, certitude, and therapist authority at the relational level, advice without permission, health, physical activity, and social references at the MI inconsistent level, and articles at the other level. During the planning phase, therapists need to help clients develop specific action plans to translate motivation for behavior change into actionable commitments. Characteristics positively correlated with MI intervention quality included simple reflection at the technical level, perception at the relational level, and impersonal pronouns in other dimensions. Characteristics negatively correlated included exclamations at the relational level, time, unsolicited advice in MI-inconsistent behaviors, focus present, and articles and big words in other dimensions. Note: MI phases refer to the four phases of MI proposed by Miller and Rollnick: engagement, focusing, arousal, and planning. Internal indicators include the therapist's use of strategies and key language cues, such as simple reflection and substance-related vocabulary.

[0120] Table 6 MI intervention quality calculation and evaluation framework

[0121]

[0122]

[0123]

[0124] 3.2.3 Evaluating DeepSeek-V3’s Performance in MI Using Intrinsic Metrics

[0125] A paired sample t-test was used to compare the differences in the intrinsic indicators of human therapists and DeepSeek-V3 at different MI stages, thereby systematically evaluating the performance of DeepSeek-V3 in MI tasks. The results are shown in Table 7.

[0126] Table 7 Comparison of intrinsic index performance between human therapists and DeepSeek-V3

[0127]

[0128]

[0129]

[0130] Note: n represents the number of video samples at different stages; Cohen's d (human vs. DeepSeek-V3) represents the result of a paired-sample t-test comparing the responses of human therapists and DeepSeek-V3 to intrinsic indicators. Numerical values ​​represent t-values; positive values ​​indicate that human therapists used the corresponding indicator more frequently, while negative values ​​indicate that DeepSeek-V3 used it more frequently. Asterisks indicate the significance level of the difference; * indicates p < 0.05, ** indicates p < 0.01, and *** indicates p < 0.001. In the engagement stage, DeepSeek-V3 significantly outperformed humans in terms of technique, but was significantly weaker than humans in expressions related to apostrophe and communication in the relationship dimension. In the MIIN dimension, DeepSeek-V3 was significantly weaker than humans in refraining from offering advice without permission, but reduced expressions related to attention. In other dimensions, DeepSeek-V3 significantly outperforms humans in the use of prepositions and long sentences (WPS), but is significantly less effective than humans in the total number of words in a single conversation (WC) and the use of long words (Bigwords). Furthermore, DeepSeek-V3 performs similarly to humans in discussions about money and time.

[0131] During the focusing phase, DeepSeek-V3 was significantly weaker than humans in both its use of simple reflection strategies at the technical level and its restraint of conversational authority at the relational level. In the MIIN dimension, DeepSeek-V3 frequently offered advice without permission, but demonstrated some restraint in discussions of substances, health, and physical activities. In other dimensions, DeepSeek-V3 was significantly weaker than humans in its use of conjunctions, prepositions, and long words, but it used more long sentences (WPS), which is beneficial for improving the quality of MI intervention. Furthermore, DeepSeek-V3 performed comparable to humans in its use of monetary rewards.

[0132] During the evocation phase, DeepSeek-V3 demonstrated a technically significant reduction in complex reflection compared to human therapists, impairing the quality of MI intervention. At the relational level, DeepSeek-V3 significantly outperformed humans in the use of interjections (exclam) and qualifiers (det), and in reducing conversational certainty (certitude), but significantly underperformed humans in expressions related to memory and time, as well as in the use of authoritative language (clout). In the MIIN dimension, DeepSeek-V3 had some limitations in warnings regarding health and physical activity, but offered unsolicited advice and references to social pressure more frequently than humans. In other dimensions, DeepSeek-V3 significantly outperformed humans in the use of long sentences (WPS), but underperformed humans in the use of articles (articles). Furthermore, there was no significant difference in the frequency of simple reflection between DeepSeek-V3 and humans.

[0133] During the planning phase, DeepSeek-V3 significantly outperformed humans in relational interjections (Exclam), time constraints (time) in planning tasks, and perceptual expressions (Perception). However, in the MIIN dimension, DeepSeek-V3 made unsolicited suggestions (Advice) and focused on the present time (Focus Present) more frequently than humans, which negatively impacted the quality of MI intervention. In other dimensions, DeepSeek-V3 significantly outperformed humans in the use of impersonal pronouns, but was significantly weaker than humans in the use of articles (Articles) and long words (Bigwords). In addition, DeepSeek-V3 and humans performed similarly in the use of simple reflection strategies. This finding addresses Question 2.

[0134] 3.3 Verification of the comprehensive computational evaluation framework

[0135] Guided by a computational evaluation framework, we developed a customized prompting framework for DeepSeek-V3 based on the Few-shot-CoT prompting strategy and combined it with retrieval enhancement technology. We also systematically analyzed the responses of human therapists, DeepSeek-V3 using customized CoT prompts, and DeepSeek-V3 using standard CoT prompts.

[0136] 3.3.1 Developing Few-Shot CoT Prompts: Optimizing DeepSeek-V3's Responses Using Interpretive Modeling Results Focused on MI Intervention Quality To optimize DeepSeek-V3's performance in MI interventions, we developed a customized CoT prompt framework based on Miller and Rollnick's four-stage MI theory, integrating intrinsic indicators and relevant psycholinguistic knowledge. An overview of the prompt framework is shown in Table 8.

[0137] Table 8. Hint framework for improving the quality of DeepSeek-V3 MI interventions

[0138]

[0139]

[0140]

[0141]

[0142] 3.3.2 Comparative Analysis of MI Effectiveness Between Prompted DeepSeek-V3 and Human Therapists

[0143] Based on the above-mentioned customized CoT prompt framework, the present invention calls DeepSeek API to obtain the response of DeepSeek-V3, and compares the MI intervention quality of human therapists, DeepSeek-V3 using customized CoT prompts, and baseline DeepSeek-V3 through Cochran's Q test. The test results show that there are significant differences in the MI intervention quality of the three (Q=25.30, p<0.001). Based on this, the present invention further uses the McNemar test system to compare the intervention effects of the three groups of responses, and uses the odds ratio to quantify the degree of improvement of customized CoT prompts on MI performance. The overall comparison results of the three are as follows: Figure 6 shown.

[0144] Table 9 shows the comparison results of DeepSeek-V3 responses before and after the optimization of customized prompts. Overall, under the guidance of customized prompts, the quality of MI intervention of DeepSeek-V3 responses is significantly improved (χ 2 =16.25, p<0.001, OR=1.68). Specifically, DeepSeek-V3 significantly improved the quality of MI intervention in emotional and mental health, general health, and lifestyle behaviors (p<0.05). The "Other" category, while not reaching statistical significance, also showed a large effect size. In contrast, customized CoT prompts had a more limited effect on nutritional intervention (p>0.05).

[0145] Table 9 Comparison of MI intervention quality before and after DeepSeek-V3 optimization

[0146]

[0147] Note: * indicates that the OR (odds ratio) value adopts Haldane-Anscombe correction, which is applicable to the zero frequency case to avoid bias in the odds ratio calculation.

[0148] Table 10 compares the intervention effectiveness of DeepSeek-V3 and human therapists after optimizing custom CoT prompts. The results show no significant difference in intervention effectiveness between the two across all treatment topics (p>0.05). Specifically, DeepSeek-V3 demonstrated a modest advantage in treating emotional and psychological well-being and general health (OR>1), but the difference did not reach statistical significance. Overall, the optimized DeepSeek-V3's MI performance has reached a level comparable to that of human therapists.

[0149] Table 10 Comparison of MI intervention quality between optimized DeepSeek-V3 and human therapists

[0150]

[0151] 3.3.3 Comparative Analysis of Intrinsic Indicators of Prompted DeepSeek-V3 and Human Therapists

[0152] To evaluate the effectiveness of customized CoT prompts in optimizing DeepSeek-V3's language cue performance, we used paired sample t-tests to quantify the differences in intrinsic indicator performance among human therapists, the baseline DeepSeek-V3 (DeepSeek-V3), and the customized prompt-optimized DeepSeek-V3 (Prompted DeepSeek-V3) at different MI stages, and used Cohen's d to evaluate the magnitude of the differences.

[0153] In the engagement phase, PromptedDeepSeek-V3 significantly outperformed humans and DeepSeek-V3 in its use of simple reflection at a technical level. At the relational level, Prompted DeepSeek-V3 increased its use of apostrophes compared to DeepSeek-V3. Furthermore, while its performance in communication-related expressions remained below human levels, it was significantly improved compared to DeepSeek-V3. In the MIIN dimension, Prompted DeepSeek-V3 was weaker than humans and DeepSeek-V3 in restraining unauthorized advice and adversarial expressions related to attention, respectively. In other dimensions, Prompted DeepSeek-V3 used more prepositions than humans and DeepSeek-V3, which contributed to improved MI intervention quality, but was significantly weaker than humans in terms of monetary rewards. In terms of text complexity, PromptedDeepSeek-V3 performed significantly worse than humans in terms of total words per conversation (WC), but better than DeepSeek-V3. It significantly outperformed humans in the use of long sentences (WPS), but was less effective than DeepSeek-V3. It also significantly outperformed DeepSeek-V3 in the use of big words. Furthermore, PromptedDeepSeek-V3 showed no significant differences from human therapists in time-related expressions (time), the use of apostrophes, attention-related expressions, and the use of long words. It was similar to DeepSeek-V3 in the use of time-related expressions, unauthorized suggestions, and money-related words.

[0154] During the focus phase, Prompted DeepSeek-V3's use of simple reflection remained technically superior to both humans and DeepSeek-V3. However, at the relationship level, Prompted DeepSeek-V3 was less effective than humans and DeepSeek-V3 in reducing the authoritativeness of the conversation (Clout). In the MIIN dimension, Prompted DeepSeek-V3 reduced the number of unsolicited advice compared to DeepSeek-V3, but still underperformed human therapists. Furthermore, Prompted DeepSeek-V3 was significantly better than humans at refraining from warnings about substances, health, and physical activity. In other dimensions, Prompted DeepSeek-V3 was significantly weaker than humans and DeepSeek-V3 in the use of conjunctions and prepositions. It improved on DeepSeek-V3 in single-round conversation length (WC) and the use of long words (Bigwords), but was still inferior to humans. It was significantly weaker than DeepSeek-V3 in the use of long sentences (WPS), but better than humans. Furthermore, Prompted DeepSeek-V3 showed no significant difference from human therapists in the use of money-related vocabulary (money). It performed on par with DeepSeek-V3 in the use of warnings related to material things, health, and physical activity, as well as money-related vocabulary.

[0155] During the evocation phase, Prompted DeepSeek-V3 significantly increased its use of complex reflection compared to DeepSeek-V3, but still lagged behind humans. It significantly outperformed DeepSeek-V3 and humans in the use of simple reflection. At the relational level, PromptedDeepSeek-V3 significantly outperformed humans in the use of interjections (Exclam), but was significantly weaker than humans and DeepSeek-V3 in expressions related to memory. Regarding participant autonomy, PromptedDeepSeek-V3 outperformed DeepSeek-V3 and humans in avoiding discussions about time and reducing certainty in conversations, respectively. Furthermore, PromptedDeepSeek-V3 significantly outperformed humans in the use of determiners (det), but lagged behind DeepSeek-V3. Furthermore, PromptedDeepSeek-V3 was significantly weaker than humans and DeepSeek-V3 in avoiding authoritative expressions (Clout). In the MIIN dimension, Prompted DeepSeek-V3 significantly improved over DeepSeek-V3 in avoiding unsolicited suggestions, but was still less effective than humans. Furthermore, compared to human therapists, Prompted DeepSeek-V3 had some limitations in its warnings regarding health and physical activity, but was significantly weaker than humans in expressions involving social pressure. In other dimensions, Prompted DeepSeek-V3 significantly outperformed both humans and DeepSeek-V3 in the use of articles. In the use of long sentences (WPS), Prompted DeepSeek-V3, while not as good as DeepSeek-V3, was significantly better than humans. Furthermore, Prompted DeepSeek-V3 did not differ significantly from human therapists in discussions of time; its use of interjections, warnings regarding health and physical activity, and the certainty of the conversation were similar to DeepSeek-V3. During the planning phase, Prompted DeepSeek-V3 significantly outperformed both humans and DeepSeek-V3 in the use of simple reflection. At the relational level, Prompted DeepSeek-V3 significantly outperformed humans in the use of exclamations and perceptual expressions, but was less effective than DeepSeek-V3 in avoiding time constraints on planning tasks.In the MIIN dimension, Prompted DeepSeek-V3 is significantly better than DeepSeek-V3 in avoiding unauthorized advice and focusing on the current time, but it does not reach the human level. In other dimensions, Prompted DeepSeek-V3 is significantly better than humans in the use of impersonal pronouns. At the same time, compared with DeepSeek-V3, it effectively reduces the use of articles and long words, but does not reach the human level. In addition, PromptedDeepSeek-V3 has no significant difference from human therapists in discussing time; it performs comparable to DeepSeek-V3 in the use of interjections, perceptual language, and impersonal pronouns. This finding solves problem 3.

[0156] 4 Discussions

[0157] Based on a dataset of 582 MI conversations, this paper explores the potential and challenges of human-intelligence collaboration in MI. First, a variety of natural language processing techniques are used to extract language features, and machine learning and deep learning algorithms are used to build a predictive model to evaluate the intervention quality of LLM in MI tasks. Then, based on Miller and Rollnick's four-stage MI theory, explanatory modeling is performed to identify the key intrinsic indicators that affect the intervention quality at different MI stages. On this basis, based on the core mechanism of the MMI framework, a computational evaluation framework with MI intervention quality as the core is developed. Finally, based on the evaluation framework, this study customized targeted CoT prompts for LLM, combined with RAG technology to obtain responses enhanced by customized prompts, and comprehensively compared the performance of internal and external indicators of human experts, baseline LLMs, and LLMs optimized with customized CoT prompts to verify the effectiveness of the evaluation framework in optimizing the MI performance of LLMs.

[0158] Results demonstrated that the LLM demonstrated excellent MI performance, particularly with the use of customized CoT prompts, which significantly improved performance. Overall, the approach employed in this paper optimizes LLM MI performance while maintaining transparency and interpretability. By integrating extrinsic and intrinsic indicators, the computational assessment framework developed in this paper provides insight into the complex relationship between therapist strategies and interview outcomes, providing a theoretical basis and systematic approach for understanding and optimizing MI practice.

[0159] 4.1 MI Competency Level of Advanced LLM

[0160] LLMs-driven conversational agents have high compliance in emotional validation and encouraging client behavior change in MI tasks (Gabriel et al., 2024). This technological potential lays a solid foundation for providing accessible and scalable interventions, and promotes the application of LLMs to evidence-based psychotherapy models. Globally, the widespread deployment of chatbots as primary intervention tools by digital mental health platforms has become a significant trend (Casu et al., 2024). At the same time, more and more adolescents tend to choose LLM-based mental health interventions and even regard them as "safe confidants" (Abbasi et al., 2022). However, the results of the present invention show that the quality of MI interventions of general LLMs, such as DeepSeek-V3, is significantly lower than that of human therapists overall (χ 2 =23.15, p<0.001). Especially in lifestyle behavior (χ2=13.41, p<0.001) and nutritional health (χ 2 =4.33, p=0.037). These findings are consistent with experts' concerns about the effectiveness and potential risks of LLMs in mental health interventions (Choudhury et al., 2023; Lawrence et al., 2024), indicating that they still have certain limitations in dealing with complex psychological interactions and emotional resonance, and have not yet reached the professional level of human therapists. This highlights the need for further research to enhance the ability of LLMs to build therapeutic alliances and practice the spirit of MI, while systematically evaluating and addressing their ethical risks in mental health interventions, including issues such as privacy protection and clinical safety. Therefore, AI applications for mental health support must be based on deep insights into clinical paradigms. This invention verifies the role of LLMs in this sensitive field and provides guidance for their technical optimization.

[0161] 4.2 MI strategies used by LLM and their similarities and differences with human experts

[0162] As described in the previous sections, this paper categorizes intrinsic indicators into four dimensions: technical, relational, MIIN behaviors, and other aspects. The relational dimension emphasizes empathy and the spirit of MI. The therapist needs to establish a collaborative relationship, empower the client during treatment, and thus stimulate their willingness to change. The technical dimension emphasizes specific MI strategies used to guide the client to make behavioral changes (Romano & Peters, 2016). MIIN behaviors refer to behaviors that conflict with core MI principles and thus reduce the effectiveness of the intervention (Akinrolie et al., 2024). Other aspects do not fit into any of these categories but still affect the MI intervention process. This paper explores the intrinsic indicator performance of LLMs in the MI process across the four MI phases: engagement, focusing, arousal, and planning. In the engagement phase, DeepSeek-V3 emphasizes the use of simple reflections, prepositions, and long sentences, which facilitates the establishment of a positive doctor-patient relationship (Cole et al., 2023) and conveys rich, multidimensional information through rigorous analytical thinking. Furthermore, avoiding confrontational expressions associated with attention effectively reduces client hostility and resistance (Moyers & Martin, 2006). However, DeepSeek-V3 performed poorly in terms of equal cooperation. The limited use of apostrophes and communication-related vocabulary made the conversations overly formal and restrictive, increasing client anxiety in the early stages of treatment (Lee et al., 2022). Furthermore, the excessive frequency of proactive suggestions hindered client self-exploration. Complex vocabulary and excessive sentences increased the client's cognitive burden, impairing their understanding and acceptance of key treatment information.

[0163] During the focusing phase, DeepSeek-V3 reduces warnings about sensitive topics such as substances, health, or physical activity, effectively lowering clients' defensiveness and preventing them from straying from core treatment goals (Moyers & Martin, 2006). Furthermore, the moderate use of long sentences enhances the hierarchy of information and helps clients accurately identify core issues. However, DeepSeek-V3 limits simple reflection on client questions and overemphasizes authoritative expressions and proactive suggestions. This "expert-driven," directive approach to conversation may undermine client autonomy and foster resistance (Harakas, 2013). Excessive reduction of connectives and increased use of prepositions may lead to loose syntactic structure and ambiguous logical relationships, weakening client and therapist focus and understanding of core issues. Furthermore, complex vocabulary and lengthy expressions may hinder clients' accurate identification of the treatment focus. During the evoking phase, DeepSeek-V3 reduces the use of interjections, avoiding overly emotional salience and thereby rationally tapping into clients' intrinsic motivations. Furthermore, reducing warnings about topics such as health and physical activity helps clients alleviate anxiety and stimulate their willingness to change. Furthermore, reducing conversational certainty and using qualifiers (such as "some" and "any") allows clients more room for independent reflection, fostering sustainable motivation for change. The use of long sentences facilitates clear expression of understanding and empathy for clients. However, DeepSeek-V3's restrictions on complex reflection and emotional reflection may impair in-depth understanding of clients' emotional states, thereby hindering the stimulation of positive emotions and the enhancement of their self-efficacy (Cole et al., 2023; Wagner & Ingersoll, 2008). Furthermore, excessive discussion time, excessive use of authoritative language, and proactive suggestions violate the autonomy-supporting principle of MI, thereby triggering psychological backlash (Harakas, 2013). Excessive use of articles may limit clients' open exploration, while expressions involving social references may exert social pressure and induce motivated silence in clients.

[0164] During the planning phase, DeepSeek-V3 rationally expresses empathy by limiting the use of interjections and increasing perceptual expressions, promoting self-reflection and motivation for change (Miller et al., 2003). Reducing time constraints promotes autonomous goal setting, consistent with the need for autonomy principle of Self-Determination Theory (SDT) (Deci & Ryan, 2000). Furthermore, the use of impersonal pronouns enhances the deindividuation effect, promoting objective reflection on change strategies. However, DeepSeek-V3's excessive use of proactive suggestions and articles, coupled with an overemphasis on the present time, makes goals and plans more deterministic and structured, limiting clients' open exploration. Furthermore, the excessive use of complex vocabulary increases clients' cognitive load and impairs their comprehension of the core plan content.

[0165] 4.3 Impact of human expert strategies on LLM’s MI capabilities and behavioral performance

[0166] Prompt engineering is an effective method to optimize the performance of generative AI in specific tasks. Based on the comprehensive modeling paradigm of computational social science and applying the four-stage theory of MI, this paper develops a comprehensive computational evaluation framework for the quality of MI intervention. At the same time, based on this, customized CoT prompts are introduced and their effectiveness is evaluated. Experimental results show that customized CoT prompts significantly improve the quality of MI intervention of LLM, and there is no significant difference compared with human therapists. This result verifies the effectiveness of CoT prompts in improving the performance of generative AI in performing MI tasks, and provides a precise and feasible solution for the application and optimization of LLM in the fields of medical care, psychological counseling, and health behavior guidance.

[0167] In terms of specific intrinsic performance indicators, LLM demonstrates both advancements and limitations. During the engagement phase, DeepSeek-V3, optimized with customized prompts, outperformed both the baseline DeepSeek-V3 and human therapists in reflective listening and analytical thinking, reflecting its synergistic improvement in empathetic dialogue and logical coherence, which helps build trust. Furthermore, its optimizations in equal cooperation and text complexity foster an atmosphere of equal interaction and alleviate client anxiety. However, it still has certain flaws in the areas of restraint in MIIN behavior and monetary rewards, which may induce early rebellious reactions and weaken clients' intrinsic motivation to change.

[0168] During the focusing phase, DeepSeek-V3, optimized with customized prompts, performed well in the use of simple reflections, helping therapists accurately capture clients' emotional tendencies and focus on core issues. Furthermore, it performed well in reducing proactive suggestions and warnings about sensitive topics, effectively avoiding clients' defensive reactions. At the same time, DeepSeek-V3 reduced complex vocabulary and lengthy expressions, reflecting an improvement in the clarity of its message delivery. However, in terms of equal cooperation and analytical thinking, DeepSeek-V3 with customized prompts was still weaker than the baseline DeepSeek-V3 and human therapists, which caused the intervention to deviate from the "cooperative spirit" at the core of MI and was not conducive to clients' precise focus on core issues.

[0169] During the evocation phase, DeepSeek-V3, optimized with customized prompts, performed well in increasing reflective listening and reducing the use of interjections, facilitating the expression of moderate empathy while avoiding excessive emotional expression. Furthermore, it outperformed human therapists in enhancing participant autonomy and reducing the use of articles, reflecting its ability to construct a language environment for autonomous exploration. Furthermore, it reduced warnings about health and physical activity, helping to alleviate client anxiety and enhance intrinsic motivation for behavioral change. However, DeepSeek-V3 with customized prompts still lagged behind both the baseline DeepSeek-V3 and human therapists in terms of emotional reflection and social referencing, reflecting its inability to strengthen client self-perception and mitigate potential external pressures. Furthermore, although customized prompts reduced the frequency of proactive suggestions, further optimization is needed to ensure that the intervention is more consistent with the autonomy-supportive principles of MI and avoid triggering client resistance.

[0170] During the planning phase, DeepSeek-V3, optimized with customized prompts, excelled in reflective listening and empathic expression. By reducing interjections and enhancing perceptual expressions, it improved clients' cognitive-behavioral consistency, strengthening their willingness and persistence to execute their plans. Furthermore, it outperformed the baseline DeepSeek-V3 in limiting MIIN behaviors, using articles, and using long words, demonstrating improvements in strengthening decision-making autonomy and improving presentation clarity. However, further work is needed to balance guidance and autonomy to support clients in developing more personalized and actionable action plans.

[0171] Overall, the customized Few-shot-CoT prompts effectively narrowed the gap between the general LLM and human therapists in terms of internal and external indicators, indicating that prompt engineering driven by psycholinguistics can significantly improve the ecological validity of artificial intelligence in psychological intervention (Souza et al., 2024), and provide a verifiable optimization paradigm for the standardization of intervention quality, ethical risk prevention and control, and safety assessment of digital mental health services (Pendse et al., 2024).

[0172] 5 Conclusion

[0173] This study systematically examined the capabilities and limitations of large language models (LLMs) in motivational interviewing (MI) tasks by constructing a computational evaluation framework based on 582 MI dialogues. The study found that the quality of MI interventions using general-purpose LLMs (such as DeepSeek-V3) was significantly lower than that of human therapists, particularly in handling sensitive topics, engaging with equal cooperation, and reflecting on emotions. However, informed by the evaluation framework, customized CoT prompts based on psycholinguistics and the four-stage MI theory significantly improved LLM performance, approaching or even exceeding human performance in key indicators such as reflective listening in the engagement phase, information clarity in the focusing phase, and autonomy support in the evocative phase. A phased analysis further revealed potential for improvement in the LLM: reducing sensitive topic warnings and authoritative expressions could reduce defensiveness, while the appropriate use of qualifiers and long sentences could balance information transfer and client autonomy. However, the LLM still has limitations in areas such as emotional resonance, social reference suppression, and behavioral restraint in MI, potentially undermining client self-efficacy and the development of a therapeutic alliance. On a theoretical level, the computational evaluation framework proposed in this paper provides a quantifiable evaluation framework for MI practices of humans and generative AI, revealing the complex relationship between treatment strategies and intervention effects. On a practical level, it verifies the capabilities and limitations of LLM in the field of MI, and demonstrates the significant role of prompt engineering based on human-computer collaboration in improving the ecological validity of LLM, providing a technical path for the standardization of digital mental health services and the prevention and control of ethical risks. In addition, the integrative modeling research paradigm adopted provides a methodological reference for understanding and promoting AI behavior. Future research needs to further optimize the performance of LLM in the dimensions of emotional regression and equal cooperation, and at the same time combine multimodal data with dynamic feedback mechanisms to explore evidence-based psychotherapy models of human-computer collaboration to ensure the safety, explainability and full respect for client autonomy of AI interventions.

[0174] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0175] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A human-computer collaboration system for psychotherapy, characterized in that: Human-computer collaboration in psychotherapy includes: Data collection module, data processing module, predictive modeling module, explainable modeling module, and customization module; A data collection module, connected to the data processing module, is used to use the created motivational interview dataset; The data processing module is connected to the predictive modeling module and is used to classify and encode the category labels such as therapist strategies in the dataset and extract embedded representations for text features; The predictive modeling module, connected with the interpretable modeling module, is used for predictive modeling and uses machine learning and deep learning techniques to develop a computational evaluation framework for assessing the intervention quality of LLMs in MI; The explainable modeling module is connected with the customized module to identify key intrinsic indicators that affect MI through explainable modeling and automatically evaluate the LLM language behavior throughout the MI process; The customized module, connected with the interpretable modeling module, is used for interpretable modeling, identifying key intrinsic indicators that affect the quality of MI intervention, and automating the evaluation of LLM language behavior throughout the MI process. It focuses on creating MI task-specific prompts and evaluating the effectiveness of the framework in improving LLM performance in MI tasks.

2. The human-computer collaboration system for psychotherapy according to claim 1, characterized in that: The data collected: A motivational interviewing dataset was created using this method. The dataset contains 133 conversations between therapists and clients, totaling 9,699 conversation turns. The conversations were sourced from MI demonstration videos on YouTube and were transcribed and annotated by experts in the MI field. GPT-4 was used for text data enhancement to obtain an extended dataset containing 582 conversations and a total of 28,738 conversation turns.

3. The human-computer collaboration system for psychotherapy according to claim 1, characterized in that: The predictive modeling: First, we manually verified the therapist strategies annotated by GPT-4 to ensure the authenticity and reliability of the annotated data, and counted the usage frequency of various strategies and included them in the initial feature set; Then, LIWC analysis and pre-trained word embedding models were used to extract language clues from the conversation text, and each was used as an independent feature set for comparison. On this basis, machine learning and deep learning algorithms were further applied to construct a prediction model for MI intervention quality. Finally, the McNemar test was used to compare the MI intervention quality of human experts and DeepSeek-V3 at the overall and different treatment topic levels.

4. The human-computer collaboration system for psychotherapy according to claim 1, characterized in that: The interpretable modeling: First, we selected XGBoost, the optimal machine learning algorithm for the predictive modeling stage, and performed feature refinement and optimization in the REFRESH framework for different stages of MI. We then used the XGBoost-RFE-SHAP algorithm, combined with five-fold cross-validation, to retrain the model and dynamically adjust hyperparameters in each recursive round. We then iteratively optimized the feature subsets to build a refined predictive model. The algorithm uses the cooperative game theory tool SHAP for global interpretation. It calculates the contribution of features to model prediction by taking the average absolute value of the SHAP values ​​of all samples, thereby assigning importance values ​​to features. This is then paired with the XGBoost algorithm for recursive feature elimination, achieving superior model performance with fewer feature combinations. Then, based on the refined model, we conducted SHAP local interpretability analysis in conjunction with the XAI framework to quantify the cumulative contribution of intrinsic indicators and clarify their directional impact on the quality of MI intervention, thus forming a complete computational evaluation framework. Finally, the paired sample t-test was used to compare the performance differences of intrinsic indicators between human experts and DeepSeek-V3 in each stage of MI, providing important basis for optimizing the MI intervention strategy of universal LLMs.

5. The human-computer collaboration system for psychotherapy according to claim 1, characterized in that: The assessment framework: The effectiveness of the computational evaluation framework in enhancing the MI performance of DeepSeek-V3 was evaluated through prompt engineering; An MI knowledge base was constructed based on human MI conversation texts, the four-stage MI theory, therapist strategy definitions, and relevant knowledge in the MI operation manual. Under the guidance of the computational evaluation framework established in the explanatory modeling stage, based on the Few-shot-CoT prompt strategy, the RAG technology was used to optimize the knowledge retrieval process and develop a customized prompt framework for DeepSeek-V3. At the same time, the MI intervention quality performance of human therapists, the baseline DeepSeek-V3, and the DeepSeek-V3 optimized by prompt engineering were compared for different treatment topics, as well as the specific differences in intrinsic indicators in each MI stage.

6. A method for human-computer collaboration in psychotherapy implementing the human-computer collaboration system in psychotherapy according to any one of claims 1 to 5, characterized in that: include: Step 1: Use the created motivational interview dataset through the data collection module; The dataset has been augmented with GPT-4 and contains 582 conversations between therapists and clients, totaling 28,738 dialogue turns. The data processing module categorizes and encodes the therapist strategies and other category labels in the dataset, extracting embedded representations of the text features. Step 2, develop a framework for evaluating LLMs in MI using machine learning and deep learning techniques through a predictive modeling module; Step 3: Identify key intrinsic indicators that affect the quality of MI intervention through an interpretable modeling module and automatically evaluate LLM language behavior throughout the MI process; Step 4, customized prompt development through customized modules focuses on creating MI task-specific prompts and evaluates the effectiveness of the framework in improving LLM performance.

7. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the method for improving human-computer collaboration in psychotherapy as claimed in claim 7.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method for improving human-computer collaboration in psychotherapy according to claim 7.

9. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the human-computer collaboration system in psychotherapy as described in any one of claims 1-5.

Images