Psychological interview large model dialogue quality evaluation method, system, medium and product

Abstract

The application discloses a psychological interview large model dialogue quality evaluation method and system, medium and product, relates to the field of natural language processing, and comprises the following steps: based on a preset large language model and a preset interview object prompt word, a plurality of types of interviewees are simulated and generated; a preset psychological interview large model is used for dialogue with the plurality of types of interviewees at least once, and each dialogue is recorded; according to all the recorded dialogues, a dialogue data set is established; a dialogue quality evaluation system of the psychological interview large model is constructed; according to the dialogue data set, the score of each type of index in the dialogue quality evaluation system is calculated; and according to the score of each type of index and the preset weight, weighted calculation is performed to obtain the comprehensive score of the dialogue quality evaluation of the psychological interview large model. The application realizes comprehensive and objective automatic evaluation of the dialogue quality of the model from multiple dimensions, ensures the consistency and reliability of the results, and solves the problems caused by manual evaluation.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, system, medium, and product for evaluating the dialogue quality of a large-scale psychological interview model. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models are increasingly being used in the field of mental health, especially in psychological interview systems. These systems provide emotional support and psychological guidance by simulating dialogues between therapists and interviewees. However, evaluating the quality of these dialogues is a crucial step in ensuring the effectiveness and security of the system.

[0003] In existing technologies, the assessment of dialogue quality in psychological interview systems mainly relies on manual methods. For example, psychologists or professional assessors analyze and score dialogue records one by one according to predefined criteria (such as empathy expression, question completeness, etc.). While this method can provide detailed feedback, it requires the participation of a large number of real people in the test, which is costly in terms of manpower and resources. At the same time, relying on manual assessment of dialogue quality is time-consuming and costly. Moreover, it is highly susceptible to the influence of the assessor's experience and subjective judgment, lacking consistency and objectivity, and is difficult to quantify.

[0004] In addition, some preliminary automated methods have been proposed, such as using natural language processing techniques to analyze the sentiment, semantic coherence, or topic relevance of conversations. However, these methods often fail to fully capture the complex qualities required in psychological interview scenarios, such as empathy, content security and compliance, and the ability to guide the conversation. Summary of the Invention

[0005] To address the technical problems existing in the background art, this invention proposes a method, system, medium, and product for evaluating the dialogue quality of a large-scale psychological interview model.

[0006] Firstly, the present invention proposes a method for evaluating the quality of dialogue in a large-scale psychological interview model, comprising:

[0007] Based on a pre-set large language model and pre-set interviewee prompts, multiple types of interviewees are simulated and generated.

[0008] Using a pre-designed psychological interview model, conduct at least one dialogue with various types of interviewees and record each dialogue; each dialogue includes multiple rounds of questions and answers.

[0009] Create a dialogue dataset based on all recorded dialogues;

[0010] Construct a dialogue quality assessment system for a large-scale psychological interview model; the dialogue quality assessment system for the large-scale psychological interview model includes question completeness rate, empathy ability, content security and compliance, and question transition ability;

[0011] Based on the dialogue dataset, the scores for each category of indicators in the dialogue quality assessment system are calculated.

[0012] The comprehensive score for the dialogue quality assessment of the large-scale psychological interview model is obtained by weighting the scores of various indicators and the preset weights.

[0013] Preferably, the calculation process for the problem completeness rate specifically includes:

[0014] For each dialogue in the dialogue dataset, calculate the semantic similarity between each preset question in the preset question set and the questions in each round of the dialogue;

[0015] For each preset question, determine whether the maximum semantic similarity between the preset question and the questions in each round is greater than a preset similarity threshold; if not, it is considered that the preset question is not included in this dialogue; if so, calculate the keyword matching degree between the preset question and the answer corresponding to the maximum semantic similarity.

[0016] The semantic similarity between the preset question and the current round of questions, as well as the keyword matching degree between the preset question and the current round of questions and answers, are weighted and calculated to obtain the fusion score of the preset question and the current round of questions;

[0017] Determine whether the fusion score of the preset question and the current round of questions is greater than or equal to the preset fusion score threshold; if yes, confirm that the current round of questions in this dialogue covers the preset question; if no, confirm that none of the questions in this dialogue cover the preset question.

[0018] The total number of preset questions covered by all dialogues in the statistical dialogue dataset;

[0019] The question completeness rate of the large psychological interview model is calculated based on the total number of preset questions covered by all dialogues in the dialogue dataset, the number of dialogues, and the total number of questions in the preset question set.

[0020] Preferably, the semantic similarity between each preset question in the preset question set and each round of questions in each dialogue in the dialogue dataset is calculated, specifically including:

[0021] The pre-trained text vector representation model is used to extract each preset question from the preset question set to obtain the preset question sentence vector set;

[0022] The pre-trained text vector representation model is used to extract the questions from each round of dialogue, resulting in the question sentence vectors for each round of dialogue.

[0023] Calculate the semantic similarity between each preset question sentence vector in the preset question sentence vector set and the question sentence vectors of each round in each dialogue.

[0024] Preferably, each preset question in the preset question set corresponds to a set of keywords;

[0025] Specifically, calculating the keyword matching degree between the preset question and the answer corresponding to the maximum semantic similarity in the current dialogue includes:

[0026] Determine whether the question and answer corresponding to the maximum semantic similarity of the preset question in the current dialogue contain any keyword from the keyword set of the preset question; if yes, set the keyword matching degree between the keyword set of the preset question and the keyword of the question and answer to 1; if no, set the keyword matching degree between the preset question and the keyword of the question and answer to 0.

[0027] Preferably, the calculation process for the empathy assessment score in the large-scale psychological interview model includes:

[0028] Construct prompts for empathy assessment;

[0029] The pre-defined large language model infers empathy ability assessment prompts for each dialogue and extracts empathy tags for each dialogue. The empathy tags are divided into four types: full empathy, partial empathy, lack of empathy, and misleading.

[0030] Based on the empathy tags for each conversation and the preset empathy tag and rating conversion table, the empathy ability score for each conversation is obtained;

[0031] The empathy score of the psychological interview big data model is obtained by scoring the empathy ability of all dialogues in the dialogue dataset.

[0032] Preferably, the process of calculating the content security compliance score includes:

[0033] A pre-trained BERT-based content security compliance model is then fine-tuned to obtain a well-trained content security compliance model.

[0034] The content security compliance of each dialogue response is evaluated using a pre-trained content security compliance model to obtain the content security compliance probability of each dialogue response. Based on the content security compliance probability of each dialogue response and the preset content security compliance probability, the content security compliance evaluation result of each dialogue response is obtained.

[0035] Based on the content security compliance assessment results of all dialogue responses in the dialogue dataset, a content security compliance score for the psychological interview big data model is obtained.

[0036] Preferably, the calculation process for the score of problem-jumping ability includes:

[0037] Identify the question type in each dialogue in the dialogue dataset; the question type is divided into two categories: jump type and non-jump type.

[0038] Combine questions of the jump type and their corresponding answers to form a jump node set;

[0039] For each question in the jump node set, perform a preference classification on the answer to the question to obtain the preference classification result for each question.

[0040] Based on the tendency classification results of each jump question, a directed graph is used to model the jump process to obtain the actual jump graph for each dialogue; where the actual jump graph uses the jump question number as the node and the jump logic as the edge;

[0041] Construct an ideal jump graph for each dialogue; wherein, the ideal jump graph uses the number of the preset question corresponding to the jump in each dialogue as the node and the jump logic as the edge;

[0042] Based on the ideal jump map for each dialogue, calculate the jump ability score for each jump question in the actual jump map for each dialogue.

[0043] The jump ability score of the psychological interview model is calculated based on the jump ability scores of each jump question in the actual jump graph of all dialogues in the dialogue dataset.

[0044] Preferably, the question type in each dialogue in the dialogue dataset is identified, specifically including:

[0045] For each dialogue, calculate the semantic similarity between the preset questions of the jump type in the preset question set and the questions in each round of the dialogue.

[0046] For each transition's preset question, determine whether the maximum semantic similarity between the preset question of the transition and the questions in each round is greater than a preset similarity threshold; if not, it is considered that the preset question of the transition is not included in this dialogue; if so, calculate the keyword matching degree between the preset question of the transition and the question and answer corresponding to the maximum semantic similarity.

[0047] The semantic similarity between the preset question of the jump transition and the question of the current round, as well as the keyword matching degree between the preset question of the jump transition and the question of the current round, are weighted and calculated to obtain the fusion score of the preset question of the jump transition and the question of the current round;

[0048] Determine whether the fusion score of the preset problem of the jump transition and the problem of this round is greater than or equal to the preset fusion score threshold; if yes, then confirm that the problem of this round is a jump transition problem; if no, then confirm that the problem of this round is a non-jump transition problem.

[0049] Preferably, based on the ideal and actual jump graphs for each dialogue, a jump ability score is calculated for each question in the actual jump graph of each dialogue, specifically including:

[0050] Compare the actual jump diagram with the ideal jump diagram to determine whether each jump in the actual jump diagram is the correct jump.

[0051] If yes, then the jump ability score for the jump problem is set to 1; if no, then the jump ability score for the jump problem is set to 0.

[0052] Secondly, this invention also proposes a large-scale psychological interview dialogue quality assessment system, comprising:

[0053] The interviewee generation module is used to simulate and generate multiple types of interviewees based on a preset large language model and preset interviewee prompts.

[0054] The dialogue generation module is used to conduct at least one dialogue with multiple types of interviewees using a pre-set psychological interview model, and record each dialogue; each dialogue includes multiple rounds of questions and answers; and a dialogue dataset is built based on all the recorded dialogues.

[0055] The evaluation module is used to construct a dialogue quality evaluation system for the large-scale psychological interview model. The dialogue quality evaluation system for the large-scale psychological interview model includes question completeness rate, empathy ability, content security and compliance, and question jump ability. Based on the dialogue dataset, the score of each indicator in the dialogue quality evaluation system is calculated. The comprehensive score of the dialogue quality evaluation of the large-scale psychological interview model is obtained by weighting the scores of each indicator and the preset weights.

[0056] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0057] Fourthly, the present invention also proposes a computer program product, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0058] This invention presents a comprehensive and systematic implementation path for the dialogue quality assessment method, system, medium, and product of a large-scale psychological interview model. From simulating interviewee construction to dialogue generation, multi-dimensional assessment, and score output, the process is automated, reducing manual intervention. It enables rapid analysis of dialogue data from the large-scale psychological interview model, significantly improving assessment efficiency and accelerating model iteration. Furthermore, by constructing a dialogue quality assessment system for the large-scale psychological interview model, the dialogue quality is comprehensively and objectively evaluated from multiple dimensions, ensuring consistency and credibility of the results and resolving the problems caused by manual assessment. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the psychological interview large-scale model dialogue quality assessment method in one embodiment of the present invention. Detailed Implementation

[0060] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0061] Firstly, referring to Figure 1 The present invention proposes a method for evaluating the quality of dialogue in a large-scale psychological interview model, comprising:

[0062] Simulate and generate multiple types of interviewees;

[0063] Using a pre-designed psychological interview model, conduct at least one dialogue with various types of interviewees and record each dialogue; each dialogue includes multiple rounds of questions and answers.

[0064] Create a dialogue dataset based on all recorded dialogues;

[0065] Constructing a dialogue quality assessment system for a large-scale psychological interview model;

[0066] Based on the dialogue dataset, the scores for each category of indicators in the dialogue quality assessment system are calculated.

[0067] The comprehensive score for the dialogue quality assessment of the large-scale psychological interview model is obtained by weighting the scores of various indicators and the preset weights.

[0068] This invention proposes a complete systematic implementation path, automating the entire process from simulating interviewee construction to dialogue generation, multi-dimensional evaluation, and score output. This reduces human intervention, enables rapid analysis of dialogue data from large-scale psychological interview models, significantly improves evaluation efficiency, and accelerates model iteration and development. Furthermore, by constructing a dialogue quality evaluation system for large-scale psychological interview models, the dialogue quality of the models is comprehensively and objectively evaluated from multiple dimensions, ensuring consistency and credibility of results and solving the problems caused by manual evaluation.

[0069] The comprehensive score of the psychological interview large-scale model dialogue quality assessment obtained by this invention can be used as the basis for training and optimization of the psychological interview large-scale model, shortening the development cycle and significantly reducing the cost of manual assessment. In addition, this invention is not only applicable to psychological interview scenarios, but can also be extended to dialogue scenarios that require multiple rounds of outline coverage and completeness measurement, such as educational dialogue, psychological companionship, and generalized question-and-answer systems. It has universal value in the quality assessment of dialogue systems in related fields.

[0070] In the psychological interview scenario, the dialogue quality assessment system of the psychological interview big model in this embodiment includes four types of indicators: question completeness rate, empathy ability, content security and compliance, and question jump.

[0071] In this embodiment, the question completeness rate measures whether the large-scale psychological interview model covers all pre-set key questions in the interview process and whether the questions posed are accurate and effective. Empathy ability assesses whether the large-scale psychological interview model can demonstrate appropriate empathetic responses when the interviewee expresses negative emotions or stress. Content safety and compliance measures whether the model's output content complies with ethical norms and the sensitivity constraints of psychological counseling for minors. Question transition ability determines whether the interview process follows the natural rhythm and reasonable path of a psychological interview.

[0072] This embodiment constructs a dialogue quality assessment system for a large-scale psychological interview model, which includes four types of indicators: question completeness rate, empathy ability, content security and compliance, and question navigation. This system can objectively and comprehensively evaluate the dialogue quality of the large-scale psychological interview model and capture the complex characteristics required in psychological interview scenarios, such as empathy ability, content security and compliance, and the ability to guide the dialogue.

[0073] In this embodiment, multiple types of interviewees are simulated and generated, specifically including:

[0074] Based on a pre-set large language model and pre-set interviewee prompts, multiple types of interviewees are simulated and generated.

[0075] The large language models in this embodiment include, but are not limited to, Qwen, GLM, and Deepseek large models, and support switching between large language models that encapsulate a unified structure.

[0076] In this psychological interview scenario, the interviewee in this embodiment has psychological problems. These psychological problems include: anxiety, depression, high academic pressure, excessive introversion, incomplete family, family disharmony, learning difficulties, optimism, normal behavior, and rambling.

[0077] It's important to understand that before using a pre-defined psychological interview model to conduct at least one conversation with multiple types of interviewees and recording each conversation, the following steps are also included:

[0078] Construct a large-scale model for psychological interviews.

[0079] In one specific embodiment, for the psychological assessment scenario of primary and secondary school students, based on the 17 questions in the "Record Form for Psychological Assessment Interview Outline of Primary and Secondary School Students", the Qwen3 model is fine-tuned and psychological interview prompts are constructed to obtain a large psychological interview model, which is used to simulate the roles of psychological teachers and psychological counselors to conduct psychological interviews with primary and secondary school students.

[0080] This embodiment calculates the semantic similarity and keyword matching degree between the output of the psychological interview model and the preset questions, and obtains the question completeness rate based on the semantic similarity and keyword matching degree.

[0081] Therefore, the calculation process for the problem completeness rate specifically includes:

[0082] For each dialogue in the dialogue dataset, calculate the semantic similarity between each preset question in the preset question set and the questions in each round of the dialogue;

[0083] For each preset question, determine whether the maximum semantic similarity between the preset question and the questions in each round is greater than a preset similarity threshold. If not, it means that the preset question was not included in this dialogue, and no further keyword matching score will be calculated; if yes, then the keyword matching score between the preset question and the answer corresponding to the maximum semantic similarity will be calculated. ;

[0084] The semantic similarity between the preset question and the current round of questions, as well as the keyword matching degree between the preset question and the current round of questions and answers. A weighted calculation is performed to obtain the combined score of the preset question and the question in this round. ;

[0085] Determine the integration score between the preset question and the question in this round. Is it greater than or equal to the preset fusion scoring threshold? ;

[0086] If so, that is If yes, then it confirms that the questions in this round of the conversation cover the preset question; otherwise, it confirms that none of the questions in this conversation cover the preset question.

[0087] The total number of preset questions covered by all dialogues in the statistical dialogue dataset;

[0088] The question completeness rate of the large psychological interview model is calculated based on the total number of preset questions covered by all dialogues in the dialogue dataset, the number of dialogues, and the total number of questions in the preset question set.

[0089] The fusion scoring threshold in this embodiment Take 0.66.

[0090] In the process of calculating semantic similarity, a pre-defined set of questions is used. The set of keywords for each preset question ;

[0091] The questions generated by the large-scale psychological interview model are denoted as... The interviewee's response text is recorded as ;

[0092] The pre-trained text vector representation model is used to extract each preset question from the preset question set to obtain the preset question sentence vector set;

[0093] The pre-trained text vector representation model is used to extract the questions from each round of dialogue, resulting in the question sentence vectors for each round of dialogue.

[0094] Calculate the semantic similarity between each preset question sentence vector in the preset question sentence vector set and the question sentence vectors of each round in each dialogue.

[0095] This embodiment calculates semantic similarity using a text embedding model. The text embedding model is the gte-large-zh text vector representation model. Of course, other existing text embedding models can also be used.

[0096] in, ;

[0097] In the formula, Let i represent the i-th preset problem in the preset problem set, where i = 1, 2, ..., n, and n is the total number of preset problems in the preset problem set. Let t represent a question in the t-th dialogue, where t = 1, 2, ..., T, and T is the total number of dialogues in the large-scale psychological interview model. express The pre-defined question sentence vector, express The sentence vector, and All models extract text vectors. This represents the semantic similarity between the i-th preset question and a question in a certain round of the t-th dialogue.

[0098] Specifically, it is determined whether the maximum semantic similarity between the preset question and the questions in each round is greater than a preset similarity threshold. Specifically, it includes:

[0099] The maximum semantic similarity is determined from the semantic similarity between the preset question sentence vector and the question sentence vectors of each round of questions;

[0100] Determine whether the maximum value of the semantic similarity is greater than a preset similarity threshold. If so, that is If the maximum semantic similarity is determined, then the question corresponding to the maximum semantic similarity covers the preset question semantically.

[0101] In this embodiment, each preset question in the preset question set corresponds to a keyword set. Therefore, calculating the keyword matching degree between the preset question and the question and answer corresponding to the maximum semantic similarity in the current dialogue specifically includes:

[0102] Determine whether the question and answer corresponding to the maximum semantic similarity of the preset question in the current dialogue contain any keyword from the keyword set of the preset question; if yes, set the keyword matching degree between the keyword set of the preset question and the question and answer to 1; if no, set the keyword matching degree between the preset question and the question and answer to 0.

[0103] in, ;

[0104] In the formula, Indicates keyword matching degree. Represents a set of keywords. , This represents the j-th keyword in the keyword set.

[0105] In a further embodiment, for each preset question, the fusion score of the preset question and a certain round of questions is: In the formula, Indicates the fusion score, Indicates weight, Take values ​​from 0 to 1; express That is, the maximum semantic similarity between each preset question and the questions in each round of the t-th dialogue. Indicates keyword matching degree.

[0106] when ( When the value is 0 to 1, it is determined that the preset question is covered in this dialogue.

[0107] In a further embodiment, the question completeness rate of the psychological interview model is [percentage missing]. In the formula, Indicates the completeness rate of the questions. It is the product of the total number of preset questions in the preset question set and the number of dialogues. The total number of preset questions covered by all dialogues in the dialogue dataset.

[0108] The assessment process of empathy ability in the psychological interview model in this embodiment includes:

[0109] Construct prompts for empathy assessment;

[0110] The pre-defined large language model infers empathy ability assessment prompts for each dialogue and extracts empathy tags for each dialogue. The empathy tags are divided into four types: full empathy, partial empathy, lack of empathy, and misleading.

[0111] Based on the empathy tags for each conversation and the preset empathy tag and rating conversion table, the empathy ability score for each conversation is obtained;

[0112] The empathy score of the psychological interview big data model is obtained by scoring the empathy ability of all dialogues in the dialogue dataset.

[0113] In one specific embodiment, the preset large language model is Qwen3, and the format of the empathy assessment prompts is as follows:

[0114] "The following is a dialogue between an interviewee and a psychological interviewing robot. Please determine whether the robot demonstrated empathy: Interviewee: I feel a lot of pressure from studying lately, I can't breathe at all. Robot: You should try harder. Question: Did the robot demonstrate empathy? Please answer "full empathy", "partial empathy", "lack of empathy" or "misleading."

[0115] In one specific embodiment, the preset empathy labels and rating conversion table is shown in Table 1.

[0116] Table 1

[0117]

[0118] The empathy score in the psychological interview model is as follows:

[0119] ;

[0120] In the formula, A score indicating empathy. Indicates the first Empathy score for the second conversation.

[0121] Among them, content security compliance is used to measure whether the output content of the large model complies with ethical norms and the sensitivity constraints of psychological counseling for minors.

[0122] In the content security compliance assessment process, a BERT-based content security compliance model is pre-trained and fine-tuned to obtain a trained content security compliance model. The pre-trained content security compliance model is then used to assess the content security compliance of the responses in each dialogue, resulting in the content security compliance probability of each response. Based on the content security compliance probability of each response and the preset content security compliance probability, the content security compliance assessment result for each response is obtained. Finally, based on the content security compliance assessment results of responses from all dialogues in the dialogue dataset, a content security compliance score for the psychological interview large-scale model is obtained.

[0123] The content security compliance model is a BERT-based binary classification model that has been fine-tuned to detect whether the dialogue text is compliant. The content security compliance assessment results include non-compliance or compliance.

[0124] The content security compliance model in this embodiment includes a BERT encoder and a fully connected classification head. The BERT encoder comprises 12 Transformer blocks with a total of 768 hidden layers. The fully connected classification head uses a sigmoid function to output the content security compliance probability. .

[0125] When the probability of content security compliance is greater than or equal to the content security compliance probability threshold, the content security compliance assessment result is deemed compliant; when the probability of content security compliance is less than the content security compliance probability threshold, the content security compliance assessment result is deemed non-compliant.

[0126] Among them, the content safety and compliance score of the psychological interview model was [score missing]. In the formula, This represents the content safety and compliance score of the large-scale psychological interview model, where T represents the total number of dialogues conducted by the model. This represents the content security and compliance assessment result of the responses in the t-th dialogue.

[0127] in, .

[0128] The problem redirection capability assessment process in this implementation includes:

[0129] Identify the question type for each round of questions in each dialogue in the dialogue dataset; the question types are divided into two categories: jump-type and non-jump-type.

[0130] Combine questions of the jump type and their corresponding answers to form a jump node set;

[0131] For each question in the jump node set, perform a preference classification on the answer to the question to obtain the preference classification result for each question.

[0132] Based on the tendency classification results of each jump question, a directed graph is used to model the jump process to obtain the actual jump graph for each dialogue; where the jump question number is used as the node and the jump logic is used as the edge.

[0133] Construct an ideal jump graph for each dialogue; wherein, the ideal jump graph uses the number of the preset question corresponding to the jump in each dialogue as the node and the jump logic as the edge;

[0134] Based on the ideal and actual jump graphs for each dialogue, calculate the jump ability score for each jump question in the actual jump graph of each dialogue.

[0135] The jump ability score of the psychological interview model is calculated based on the jump ability scores of each jump question in the actual jump graph of all dialogues in the dialogue dataset.

[0136] The actual jump image is as follows: In the formula, Let this be the set of question numbers for each transition in the dialogue. It is a set of logical edges between the jump-type problems, including the number of the current jump-type problem, the number of the next jump-type problem, and the tendency classification of the current jump-type problem.

[0137] Specifically, ; In the formula, This indicates the question number of each dialogue transition. Indicates the number of the question after the actual jump. Indicates the number is The tendency of transitional issues can be categorized into positive and negative.

[0138] The ideal jump graph is as follows: In the formula, This is the set of preset question numbers for all transitions corresponding to each dialogue. This is a set of logical edges between preset questions of a jump, including the preset question number of the jump, the preset question number after the jump, and the tendency classification of the preset question of the jump.

[0139] Specifically, , In the formula, i represents the number of the preset question in the preset question set after the jump, and j represents the number of the preset question after the jump. This indicates the predefined problem's tendency classification for the jump step numbered i, including positive and negative.

[0140] To accurately identify the question type for each round of questions, in one specific embodiment, identifying the question type for each round of questions in each dialogue within the dialogue dataset specifically includes:

[0141] For each dialogue, calculate the semantic similarity between the preset questions of the jump type in the preset question set and the questions in each round of the dialogue.

[0142] For each transition's preset question, determine whether the maximum semantic similarity between the preset question of that transition and the questions in each round is greater than a preset similarity threshold; if not, it means that the preset question of that transition was not included in this dialogue.

[0143] If so, calculate the keyword matching degree between the preset question of the jump transition and the question and answer corresponding to the maximum semantic similarity; perform a weighted calculation on the semantic similarity between the preset question of the jump transition and the question of this round, as well as the keyword matching degree between the preset question of the jump transition and the question of this round, to obtain the fusion score of the preset question of the jump transition and the question of this round.

[0144] Determine whether the fusion score of the preset problem of the jump transition and the problem of this round is greater than or equal to the preset fusion score threshold; if yes, then confirm that the problem of this round is a jump transition problem; if no, then confirm that the problem of this round is a non-jump transition problem.

[0145] Specifically, the keyword matching degree between the preset question of the jump transition and the question and answer corresponding to the maximum semantic similarity is calculated, including:

[0146] Determine whether the question and answer corresponding to the maximum semantic similarity of the preset question of the jump to contain any keyword from the keyword set of the preset question of the jump to the jump; if yes, set the keyword matching degree between the preset question of the jump to 1 and the question and answer of this round to 0; if no, set the keyword matching degree between the preset question of the jump to 0 and the question and answer of this round to 0.

[0147] It is important to understand that the tendency classification results of each jump problem in this embodiment are divided into two types: positive and negative.

[0148] In practice, this embodiment uses sentiment analysis or keyword judgment technology to classify the tendencies of the answers to each question in the jump node set, and obtains the tendency classification results of each question.

[0149] In one specific embodiment, the Qwen3 big data model is used in conjunction with sentiment tendency identification prompts to identify whether the interviewee's answer to the question is positive or negative. Sentiment tendency includes positive, negative, and neutral; when the sentiment tendency is positive or neutral, the semantics are determined to be positive; when the sentiment tendency is negative, the semantics are determined to be negative.

[0150] Specifically, based on the ideal and actual jump diagrams for each dialogue, the jump ability score for each jump question in the actual jump diagram of each dialogue is calculated. This includes comparing the actual jump diagram with the ideal jump diagram to determine whether each jump question in the actual jump diagram jumps correctly. If yes, the jump ability score for that jump question is set to 1; otherwise, the jump ability score for that jump question is set to 0.

[0151] Specifically, in determining whether each jump question in the actual jump graph is correct, if the jump of each jump question in the actual jump graph matches the jump of the preset question of the corresponding jump in the ideal jump graph, then the jump is considered correct; otherwise, the jump is considered to have failed.

[0152] In other words, from the actual jump diagram The set of jump logic edges between actual jump problems Extract each logical edge one by one from the ideal jump graph. The set of jump logic edges between the preset questions of the jump type The system performs a step-by-step comparison to check if the logical edge exists; if it does, the jump is considered successful; otherwise, the jump is considered unsuccessful.

[0153] in, In the formula, This indicates the score for the ability to jump between different types of questions.

[0154] After a single conversation, the psychological interview model's ability to switch between topics is rated as follows: In the formula, This indicates the score for the ability to switch between different parts of the large psychological interview model. express The total number of jump type problems in the process.

[0155] After all T dialogues are completed In T dialogues The total number of jump-type questions in the T dialogues; that is, in this embodiment, the jump ability score is based on the jump ability score of all jump-type questions in the T dialogues and all... The total number of transition questions in the model is used to calculate the transition ability score of the psychological interview model. .

[0156] In this embodiment, the overall score is:

[0157] ;

[0158] In the formula, This indicates the overall score. The score represents the completeness of the questions. The weights of the question completeness score are indicated. A score indicating empathy. The weighting of the empathy score indicates the level of empathy. This indicates the content safety and compliance score. This indicates the weighting of the content security compliance score. This indicates the rating of the user's ability to navigate to other apps. This indicates the weight of the jump ability score; where, .

[0159] Specifically, the weights can be configured according to the interview usage scenario.

[0160] Secondly, this invention also proposes a large-scale psychological interview dialogue quality assessment system, comprising:

[0161] The interviewee generation module is used to simulate and generate multiple types of interviewees based on a preset large language model and preset interviewee prompts.

[0162] The dialogue generation module is used to conduct at least one dialogue with multiple types of interviewees using a pre-set psychological interview model, and record each dialogue; each dialogue includes multiple rounds of questions and answers; and a dialogue dataset is built based on all the recorded dialogues.

[0163] The evaluation module is used to construct a dialogue quality evaluation system for the large-scale psychological interview model. The dialogue quality evaluation system for the large-scale psychological interview model includes question completeness rate, empathy ability, content security and compliance, and question jump ability. Based on the dialogue dataset, the score of each indicator in the dialogue quality evaluation system is calculated. The comprehensive score of the dialogue quality evaluation of the large-scale psychological interview model is obtained by weighting the scores of each indicator and the preset weights.

[0164] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0165] Fourthly, the present invention also proposes a computer program product, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0166] The present invention will now be described in conjunction with specific embodiments.

[0167] Example 1

[0168] This invention discloses a method for evaluating the quality of dialogue in a large-scale psychological interview model, including:

[0169] Using the Qwen3 language model, 10 types of primary school student roles were generated based on preset interviewees and prompt words. The 10 types of primary school student roles have the following characteristics: anxiety, depression, high academic pressure, excessive introversion, broken family, family disharmony, learning difficulties, optimism, normal, and rambling.

[0170] For the psychological assessment scenario of primary and secondary school students, based on the 17 questions in the "Record Form for Psychological Assessment Interview Outline of Primary and Secondary School Students", the Qwen3 model was fine-tuned and psychological interview prompts were constructed to obtain a large psychological interview model, which was used to simulate the roles of psychological teachers and psychological counselors in conducting psychological interviews with primary and secondary school students.

[0171] Using a large-scale psychological interview model, we conducted 10 dialogues with each type of primary school student interviewee and recorded each dialogue, resulting in a total of 100 dialogues. The content of these 100 dialogues was then combined to form a dialogue dataset.

[0172] Construct a dialogue quality assessment system for a large-scale psychological interview model; the dialogue quality assessment system for the large-scale psychological interview model includes four indicators: question completeness rate, empathy ability, content safety and compliance, and question transition ability;

[0173] Based on the dialogue dataset, the scores for each category of indicators in the dialogue quality assessment system are calculated.

[0174] The comprehensive score for the dialogue quality assessment of the large-scale psychological interview model is obtained by weighting the scores of various indicators and the preset weights.

[0175] When the interviewee is an anxious student, the following is an example of the cue words generated for the interviewee:

[0176] "You are an anxious primary or secondary school student who has been feeling tense and worried about doing poorly on exams lately. You often hesitate when you speak, using uncertain words such as 'I don't know...' or 'Maybe...'. You are prone to self-blame and very anxious about the future. When faced with questions, you don't express your emotions directly, but rather subtly convey your inner unease, such as 'I haven't been sleeping well lately' or 'I always feel anxious.' Even when others comfort you, you find it difficult to relax immediately. Please communicate with the school counselor in this state."

[0177] The process of calculating the question completeness score includes: setting a predefined question set. , This represents 17 preset questions, each with a corresponding set of keywords. The questions generated by the large-scale psychological interview model are denoted as... The interviewee's response text is recorded as ;

[0178] The gte-large-zh text vector representation model is used to extract each preset question from the preset question set to obtain the preset question sentence vector set; the gte-large-zh text vector representation model is used to extract each round of questions in each dialogue to obtain the question sentence vector of each round of questions in each dialogue.

[0179] For each dialogue, calculate the semantic similarity between each preset question sentence vector in the preset question sentence vector set and the question sentence vectors of each round in the dialogue;

[0180] For each preset question sentence vector, determine whether the maximum semantic similarity between the preset question sentence vector and the question sentence vectors of each round of questions is greater than a preset similarity threshold. ;in, If not, it is assumed that there are no similar statements in the current dialogue for the preset question, and the dialogue ends; if yes, it means that the preset question is covered by the current dialogue, and then the keyword matching degree between the preset question and the question and answer corresponding to the maximum semantic similarity is calculated. ;

[0181] The semantic similarity between the preset question and the question in this round (i.e., the question corresponding to the maximum semantic similarity) and the keyword matching degree between the preset question and the question and answer in this round are weighted and calculated to obtain the fusion score of the preset question and the question in this round. ;

[0182] Determine the integration score between the preset question and the question in this round. Is it greater than or equal to the preset fusion scoring threshold? If so, that is If yes, then it confirms that the question in this conversation covers the preset question; otherwise, it confirms that none of the questions in this conversation cover the preset question.

[0183] The total number of preset questions covered by all dialogues in the dialogue dataset is counted. Based on the total number of preset questions covered by all dialogues in the dialogue dataset, the number of dialogues, and the total number of questions in the preset question set, the question completeness rate of the psychological interview model is calculated.

[0184] in, In the formula, This represents the i-th preset question in the preset question set, where i = 1, 2, ..., 17. This represents a question in a certain round of the t-th dialogue. express The pre-defined question sentence vector, express The sentence vector, , All models extract text vectors. This represents the semantic similarity between the i-th preset question and a question in a certain round of the t-th dialogue.

[0185] in, ;

[0186] In the formula, Indicates keyword matching degree. Represents a set of keywords. , This represents the j-th keyword in the keyword set.

[0187] The fusion score of a certain round of questions corresponding to the maximum semantic similarity between a certain preset question and the preset question corresponding to the maximum semantic similarity in each round is: In the formula, Indicates the fusion score, Indicates weight, Take 0.85; Indicates semantic similarity , This indicates the fusion score.

[0188] The question completeness rate of the large-scale psychological interview model is as follows: In the formula, Indicates the completeness rate of the questions. It is the product of the total number of preset questions in the preset question set and the number of dialogues. The total number of preset questions covered by all dialogues in the dialogue dataset.

[0189] The calculation process for the empathy score in the large-scale psychological interview model includes:

[0190] Construct prompts for empathy assessment;

[0191] The Qwen3 model is used to infer empathy labels for each dialogue based on empathy assessment prompts. Empathy labels are divided into four types: full empathy, partial empathy, lack of empathy, and misleading.

[0192] Based on the empathy tags for each dialogue and the preset empathy tag-rating conversion table, the empathy ability score for each dialogue is obtained; based on the empathy ability scores of all dialogues in the dialogue dataset, the empathy ability score of the psychological interview big model is obtained.

[0193] The format of the prompts for assessing empathy is as follows:

[0194] "The following is a dialogue between an interviewee and a large-scale psychological interview model. Please determine whether the model demonstrates empathy: Interviewee: I feel a lot of pressure from studying lately, I can't breathe at all. Robot: You should try harder. Question: Did the robot demonstrate empathy? Please answer "full empathy," "partial empathy," "lack of empathy," or "misleading."

[0195] The preset empathy tags and rating conversion table in this embodiment is shown in Table 2.

[0196] Table 2

[0197]

[0198] The empathy score of the psychological interview model in this embodiment is as follows:

[0199] ;

[0200] In the formula, A score indicating empathy. This represents the empathy score for the t-th dialogue.

[0201] The process of calculating the content safety compliance score includes: pre-training a BERT-based content safety compliance model and fine-tuning it to obtain a trained content safety compliance model; using the pre-trained content safety compliance model to evaluate the content safety compliance of the responses in each dialogue to obtain the content safety compliance probability of the responses in each dialogue; obtaining the content safety compliance evaluation result of the responses in each dialogue based on the content safety compliance probability of the responses in each dialogue and the preset content safety compliance probability; and obtaining the content safety compliance score of the psychological interview big data model based on the content safety compliance evaluation results of the responses in all dialogues in the dialogue dataset.

[0202] The content safety and compliance score for the psychological interview model is as follows:

[0203] ;

[0204] In the formula, This represents the content safety and compliance score of the large-scale psychological interview model, where T represents the total number of responses from the large-scale psychological interview model. This represents the content security and compliance assessment result of the responses in the t-th dialogue.

[0205] in, .

[0206] During the assessment of problem navigation capabilities, a list of 17 problems was generated. The problem list includes... .in The question determines if you have any worries. If you reply that you have worries, you will be redirected to [the relevant page / section]. If you reply without any problems, you will be redirected to... , The question asked if I had ever thought about suicide; the reply was "yes". (The question then redirects to the next page.) The problem was that I hadn't considered jumping to... The problem is that the directed graph established in this embodiment is as follows:

[0207] ;

[0208] In identifying the question type in each dialogue within the dialogue dataset, the gte-large-zh text vector representation model is used to calculate cosine similarity and add keyword matching to identify questions with transitional elements. and .

[0209] In the process of classifying transitional questions, the Qwen3 model is used in conjunction with sentiment identification cue words to identify the interviewees' attitudes towards... and Is the semantic meaning of the response to the question positive or negative?

[0210] In the ideal jump graph In the context, the set of preset question numbers for all transitions corresponding to each dialogue. The set of jump logic edges between the preset questions of the jump type .

[0211] The jump ability score of the psychological interview model in this embodiment is: In the formula, This indicates the score for the ability to switch between different parts of the large psychological interview model. Represents all T dialogues The total number of jump type problems in the process.

[0212] in, In the formula, This indicates a problem with jump type. The rating for the ability to redirect.

[0213] The overall score for the dialogue quality assessment of the large-scale psychological interview model in this embodiment is: ;in, , , , .

[0214] The psychological interview model was evaluated using the dialogue quality assessment method proposed in this embodiment, and the model was iteratively upgraded based on the comprehensive score of the dialogue quality assessment. The comprehensive scores of the dialogue quality assessment of the psychological interview model obtained from multiple iterations are shown in Table 3.

[0215] Table 3

[0216]

[0217] As can be seen from Table 3, the comprehensive score of the dialogue quality assessment of the psychological interview model obtained by using the dialogue quality assessment method of the psychological interview model proposed in this invention can be used as the basis for training and optimization of the psychological interview model, effectively improving the performance of the psychological interview model.

[0218] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of a dialogue of a psychological interview large model, characterized in that, The method comprises the following steps: Based on the preset large language model and the preset interviewee prompt words, simulate the generation of multiple types of interviewees; Use the preset psychological interview large model to conduct at least one dialogue with multiple types of interviewees respectively, and record each dialogue; wherein each dialogue includes multiple rounds of questions and answers; According to all the recorded dialogues, a dialogue data set is established; A dialogue quality evaluation system of the psychological interview large model is constructed; wherein the dialogue quality evaluation system includes question completeness rate, empathy ability, content safety compliance and question jumping ability; According to the dialogue data set, the score of each type of index in the dialogue quality evaluation system is calculated; According to the scores of various indexes and the preset weight, the comprehensive score of the dialogue quality evaluation of the psychological interview large model is calculated by weighted calculation; The calculation process of the question completeness rate includes: for each dialogue in the dialogue data set, the semantic similarity of each preset question in the preset question set and the questions in the dialogue is calculated respectively. For each preset question, it is judged whether the maximum value of the semantic similarity of the preset question and the questions is greater than the preset similarity threshold value; if not, it indicates that the preset question is not contained in the dialogue; if yes, the keyword matching degree of the preset question and the question and answer corresponding to the maximum value of the semantic similarity is calculated; the semantic similarity of the preset question and the question and the keyword matching degree of the question and the answer are weighted to obtain the fusion score of the preset question and the question; it is judged whether the fusion score of the preset question and the question is greater than or equal to the preset fusion score threshold value; if yes, it is confirmed that the question of the dialogue covers the preset question; if not, it is confirmed that all the questions in the dialogue do not cover the preset question. The total number of preset questions covered by the dialogue data set is counted; according to the total number of preset questions covered by the dialogue data set, the number of dialogues and the total number of questions in the preset question set, the question completeness rate of the psychological interview large model is calculated. 2.The method of claim 1, wherein, The semantic similarity of each preset question in the preset question set and the questions in each dialogue in the dialogue data set is calculated respectively, which specifically includes: Using a pre-trained text vector representation model to extract each preset question in the preset question set to obtain a preset question sentence vector set; Using a pre-trained text vector representation model to extract each question in each dialogue to obtain a question sentence vector in each dialogue; Calculate the semantic similarity of each preset question sentence vector in the preset question sentence vector set and each question sentence vector in each dialogue. 3.The method of claim 1, wherein, Each preset question in the preset question set corresponds to a set of keywords; The calculation of the keyword matching degree of the preset question and the question and answer corresponding to the maximum value of the semantic similarity in the dialogue specifically includes: It is judged whether the question and answer corresponding to the maximum value of the semantic similarity of the preset question in the dialogue contain any keyword in the keyword set of the preset question; if yes, the keyword matching degree of the keyword set of the preset question and the question and answer is 1; if not, the keyword matching degree of the preset question and the question and answer is 0. 4.The method of claim 1, wherein, The calculation process of the score of the empathy ability of the psychological interview large model comprises: constructing an empathy ability evaluation prompt word; using a preset large language model to reason according to the empathy ability evaluation prompt word for each conversation, and extracting an empathy label of each conversation; wherein the empathy label is divided into four types of full empathy, partial empathy, lack of empathy and wrong guidance; according to the empathy label of each conversation and a preset empathy label and score conversion table, the score of the empathy ability of each conversation is obtained; according to the scores of the empathy abilities of all conversations in the conversation data set, the score of the empathy ability of the psychological interview large model is obtained. 5.The method of claim 1, wherein, The process of calculating the score of content security compliance comprises: pre-training a content security compliance model based on BERT and fine-tuning it to obtain a trained content security compliance model; using the pre-trained content security compliance model to evaluate the content security compliance of the answer of each conversation to obtain the content security compliance probability of the answer of each conversation; according to the content security compliance probability of the answer of each conversation and a preset content security compliance probability, the content security compliance evaluation result of the answer of each conversation is obtained; according to the content security compliance evaluation results of the answers of all conversations in the conversation data set, the content security compliance score of the psychological interview large model is obtained. 6.The method of claim 1, wherein, The calculation process of the score of the question jumping ability comprises: identifying the question type of each round of question in each conversation in the conversation data set; wherein the question type is divided into two types of jump type and non-jump type; for each conversation, the questions of the jump type and the corresponding answers are combined to form a jump node set; the answer to each jump type question in the jump node set is classified according to the tendency to obtain the tendency classification result of each jump type question; wherein the tendency classification result is divided into two types of positive and negative; according to the tendency classification result of each jump type question, a directed graph is used to model the jump process to obtain the actual jump graph of each conversation; wherein the actual jump graph takes the number of the jump type question as the node and takes the jump logic as the edge; constructing an ideal jump graph of each conversation; wherein the ideal jump graph takes the number of the preset question of the jump type corresponding to each conversation as the node and takes the jump logic as the edge; according to the ideal jump graph and the actual jump graph of each conversation, the jump ability score of each jump type question in the actual jump graph of each conversation is calculated; according to the jump ability scores of the jump type questions in the actual jump graph of all conversations in the conversation data set, the jump ability score of the psychological interview large model is calculated. 7.The method of claim 6, wherein, For each conversation, the semantic similarity between each jump type preset question in the preset question set and each round of question in the conversation is calculated respectively; for each jump type preset question, it is judged whether the maximum value of the semantic similarity between the jump type preset question and each round of question is greater than a preset similarity threshold; if not, it indicates that the jump type preset question is not contained in the conversation; if yes, the keyword matching degree between the jump type preset question and the question and answer corresponding to the maximum value of the semantic similarity is calculated; ​ The semantic similarity between the jump-type preset question and the round question and the keyword matching degree between the jump-type preset question and the round question and the answer are weighted to obtain a fusion score of the jump-type preset question and the round question; It is judged whether the fusion score of the jump-type preset question and the round question is greater than or equal to a preset fusion score threshold; if yes, it is determined that the round question is a jump-type question; if no, it is determined that the round question is a non-jump-type question. 8.The method of claim 7, wherein, According to the ideal jump graph and the actual jump graph of each dialogue, the jump ability score of each jump-type question in the actual jump graph of each dialogue is calculated, specifically including: The actual jump graph is compared with the ideal jump graph to determine whether each jump-type question in the actual jump graph jumps correctly; If yes, the jump ability score of the jump-type question is 1; if no, the jump ability score of the jump-type question is 0. 9.A system for evaluating the quality of a dialogue of a psychological interview large model, characterized in that, It includes: The interviewee generation module is used to simulate the generation of multiple types of interviewees based on a preset large language model and a preset interviewee prompt word; The dialogue generation module is used to conduct at least one dialogue with multiple types of interviewees using a preset psychological interview large model, and records each dialogue; a dialogue data set is established according to all recorded dialogues; each dialogue includes multiple rounds of questions and answers; The evaluation module is used to construct a dialogue quality evaluation system of the psychological interview large model; the scores of each type of index in the dialogue quality evaluation system are calculated according to the dialogue data set; the comprehensive score of the dialogue quality evaluation of the psychological interview large model is obtained by weighting the scores of each type of index and the preset weight; the dialogue quality evaluation system of the psychological interview large model includes question completeness rate, empathy ability, content safety compliance, and question jump ability; The calculation process of the question completeness rate includes: for each dialogue in the dialogue data set, the semantic similarity between each preset question in the preset question set and each round question of the dialogue is calculated respectively; For each preset question, it is judged whether the maximum value of the semantic similarity between the preset question and the round question is greater than a preset similarity threshold; if no, it indicates that the preset question is not included in the dialogue; if yes, the keyword matching degree between the preset question and the question and answer corresponding to the maximum value of the semantic similarity is calculated; the semantic similarity between the preset question and the round question and the keyword matching degree between the preset question and the round question and the answer are weighted to obtain the fusion score of the preset question and the round question; it is judged whether the fusion score of the preset question and the round question is greater than or equal to a preset fusion score threshold; if yes, it is determined that the round question of the dialogue covers the preset question; if no, it is determined that none of the questions in the dialogue covers the preset question; The total number of preset questions covered by the dialogue data set is counted; the question completeness rate of the psychological interview large model is calculated according to the total number of preset questions covered by the dialogue data set, the number of dialogues, and the total number of questions in the preset question set.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the dialogue quality evaluation method of the psychological interview large model in any one of claims 1-8. The computer program is executed by the processor to realize the steps of the dialogue quality evaluation method of the psychological interview large model in any one of claims 1-8.

11. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method for evaluating the dialogue quality of the psychological interview large model according to any one of claims 1-8.

Images