Psychological interview large model dialogue quality evaluation method and system, medium and product

Abstract

The invention discloses a psychological interview large model dialogue quality evaluation method and system, a medium and a product, and relates to the field of natural language processing, and the method comprises the steps: simulating and generating multiple types of interviewed objects based on a preset large language model and preset interview object cues; using a preset psychological interview large model to carry out at least one dialogue with the plurality of types of interviewed objects, and recording each dialogue; establishing a dialogue data set according to all recorded dialogues; constructing a dialogue quality evaluation system of the psychological interview large model; according to the dialogue data set, calculating to obtain a score of each type of indexes in the dialogue quality evaluation system; and performing weighted calculation according to the score of each index and a preset weight to obtain a comprehensive score of dialogue quality evaluation of the psychological interview large model. According to the method, comprehensive and objective automatic evaluation on the dialogue quality of the model from multiple dimensions is realized, the result consistency and credibility are ensured, and the problem caused by manual evaluation is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a psychological interview large model dialogue quality evaluation method, system, medium and product. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large language models are increasingly applied in the field of mental health, especially in psychological interview systems. These systems simulate conversations between a psychological counselor and an interviewee to provide emotional support and psychological guidance. However, dialogue quality evaluation is a key step to ensure the effectiveness and safety of the system.

[0003] In the prior art, dialogue quality evaluation of psychological interview systems mainly relies on manual methods, such as psychologists or professional evaluators analyzing and scoring dialogue records one by one according to predefined standards (such as empathy expression, question completeness, etc.). Although this method can provide detailed feedback, it requires a large number of real people to participate in testing, consuming manpower and resources. At the same time, relying on manual evaluation of dialogue quality is time-consuming and costly, and is easily influenced by the evaluator's experience and subjective judgment, lacking consistency and objectivity, and difficult to quantify.

[0004] In addition, some preliminary automated methods have been proposed, such as using natural language processing techniques to analyze the emotional tendency, semantic coherence or topic relevance of the dialogue. However, these methods often fail to fully capture the complex characteristics required in a psychological interview scenario, such as empathy ability, content safety compliance, and dialogue guidance ability. SUMMARY

[0005] To solve the technical problems in the background art, the present application proposes a psychological interview large model dialogue quality evaluation method, system, medium and product.

[0006] In a first aspect, the present application proposes a psychological interview large model dialogue quality evaluation method, comprising: Simulating the generation of multiple types of interviewees based on a pre-set large language model and a pre-set interviewee prompt word; Using a pre-set psychological interview large model to conduct at least one dialogue with each type of interviewee, and recording each dialogue; wherein each dialogue includes multiple rounds of questions and answers; Establishing a dialogue dataset based on all recorded dialogues; Constructing a dialogue quality evaluation system for the psychological interview large model; wherein the dialogue quality evaluation system for the psychological interview large model includes question completeness rate, empathy ability, content safety compliance, and question jump ability; According to the dialogue dataset, the score of each type of index in the dialogue quality evaluation system is calculated; According to the scores of various indicators and the preset weights, a comprehensive score of the dialogue quality evaluation of the psychological interview large model is obtained by weighted calculation.

[0007] Preferably, the calculation process of the question completeness rate specifically comprises: For each dialogue in the dialogue data set, the semantic similarity of each preset question in the preset question set and the questions in each round of the dialogue is calculated respectively. For each preset question, it is judged whether the maximum value of the semantic similarity of the preset question and the questions in each round is greater than a preset similarity threshold value; if not, it is considered that the preset question is not contained in the dialogue; if yes, the keyword matching degree of the preset question and the question and answer corresponding to the maximum value of the semantic similarity is calculated. The semantic similarity of the preset question and the question in each round and the keyword matching degree of the preset question and the question and answer are weighted and calculated to obtain the fusion score of the preset question and the question in each round. It is judged whether the fusion score of the preset question and the question in each round is greater than or equal to a preset fusion score threshold value; if yes, it is confirmed that the question in each round in the dialogue covers the preset question; if not, it is confirmed that all the questions in the dialogue do not cover the preset question. The total number of preset questions covered by all dialogues in the dialogue data set is counted. According to the total number of preset questions covered by all dialogues in the dialogue data set, the number of dialogues and the total number of questions in the preset question set, the question completeness rate of the psychological interview large model is calculated.

[0008] Preferably, the semantic similarity of each preset question in the preset question set and the questions in each round of each dialogue in the dialogue data set is calculated respectively, specifically comprising: Each preset question in the preset question set is extracted using a pre-trained text vector representation model to obtain a preset question sentence vector set. Each round of question in each dialogue is extracted using a pre-trained text vector representation model to obtain a question sentence vector in each round of each dialogue. The semantic similarity of each preset question sentence vector in the preset question sentence vector set and the question sentence vector in each round of each dialogue is calculated.

[0009] Preferably, each preset question in the preset question set is provided with a keyword set. The keyword matching degree of the preset question and the question and answer corresponding to the maximum value of the semantic similarity in the dialogue is calculated, specifically comprising: determine whether the question and the answer in the current dialogue contain any keyword in the keyword set of the preset question corresponding to the maximum semantic similarity; if yes, set the keyword matching degree between the keyword set of the preset question and the question and the answer as 1; if no, set the keyword matching degree between the preset question and the question and the answer as 0.

[0010] Preferably, the process of calculating the score of the empathy ability of the psychological interview large model comprises: constructing an empathy ability evaluation prompt word; using a preset large language model to infer each dialogue according to the empathy ability evaluation prompt word, and extracting an empathy label of each dialogue; wherein the empathy label is divided into four types: full empathy, partial empathy, lack of empathy and wrong guidance; obtaining the score of the empathy ability of each dialogue according to the empathy label of each dialogue and a preset empathy label and score conversion table; obtaining the score of the empathy ability of the psychological interview large model according to the scores of the empathy abilities of all dialogues in the dialogue data set.

[0011] Preferably, the process of calculating the score of the content safety compliance comprises: pre-training a content safety compliance model based on BERT and fine-tuning it to obtain a trained content safety compliance model; using the pre-trained content safety compliance model to evaluate the content safety compliance of the answer of each dialogue to obtain the content safety compliance probability of the answer of each dialogue; and obtaining the content safety compliance evaluation result of the answer of each dialogue according to the content safety compliance probability of the answer of each dialogue and a preset content safety compliance probability; obtaining the content safety compliance score of the psychological interview large model according to the content safety compliance evaluation results of the answers of all dialogues in the dialogue data set.

[0012] Preferably, the process of calculating the score of the question jumping ability comprises: identifying the type of the question in each dialogue in the dialogue data set; wherein the type of the question is divided into two types: jump type and non-jump type; combining the questions of the jump type and their corresponding answers to form a jump node set; performing tendency classification on the answers to each question of the jump type in the jump node set to obtain a tendency classification result of each question of the jump type; using a directed graph to model the jump flow according to the tendency classification result of each question of the jump type to obtain an actual jump graph of each dialogue; wherein the actual jump graph takes the number of the question of the jump type as a node and takes the jump logic as an edge; Construct an ideal jump graph for each dialogue; wherein, the ideal jump graph uses the number of the preset question corresponding to each dialogue as the node and the jump logic as the edge; Based on the ideal jump diagram for each dialogue, calculate the jump ability score for each jump question in the actual jump diagram for each dialogue. The jump ability score of the psychological interview model is calculated based on the jump ability scores of each jump question in the actual jump graph of all dialogues in the dialogue dataset.

[0013] Preferably, the question type in each dialogue in the dialogue dataset is identified, specifically including: For each dialogue, calculate the semantic similarity between the preset questions of the jump type in the preset question set and the questions in each round of the dialogue. For each transition's preset question, determine whether the maximum semantic similarity between the preset question of the transition and the questions in each round is greater than a preset similarity threshold; if not, it is considered that the preset question of the transition is not included in this dialogue; if so, calculate the keyword matching degree between the preset question of the transition and the question and answer corresponding to the maximum semantic similarity. The semantic similarity between the preset question of the jump transition and the question of the current round, as well as the keyword matching degree between the preset question of the jump transition and the question of the current round, are weighted and calculated to obtain the fusion score of the preset question of the jump transition and the question of the current round; Determine whether the fusion score of the preset problem of the jump transition and the problem of this round is greater than or equal to the preset fusion score threshold; if yes, then confirm that the problem of this round is a jump transition problem; if no, then confirm that the problem of this round is a non-jump transition problem.

[0014] Preferably, based on the ideal and actual jump graphs for each dialogue, a jump ability score is calculated for each question in the actual jump graph of each dialogue, specifically including: Compare the actual jump diagram with the ideal jump diagram to determine whether each jump in the actual jump diagram is the correct jump. If yes, then the jump ability score for the jump problem is set to 1; if no, then the jump ability score for the jump problem is set to 0.

[0015] Secondly, this invention also proposes a large-scale psychological interview dialogue quality assessment system, comprising: The interviewee generation module is used to simulate and generate multiple types of interviewees based on a preset large language model and preset interviewee prompts. The dialogue generation module is used to conduct at least one dialogue with multiple types of interviewees using a pre-set psychological interview model, and record each dialogue; each dialogue includes multiple rounds of questions and answers; and a dialogue dataset is built based on all the recorded dialogues. The evaluation module is used to construct a dialogue quality evaluation system for the large-scale psychological interview model. The dialogue quality evaluation system for the large-scale psychological interview model includes question completeness rate, empathy ability, content security and compliance, and question jump ability. Based on the dialogue dataset, the score of each indicator in the dialogue quality evaluation system is calculated. The comprehensive score of the dialogue quality evaluation of the large-scale psychological interview model is obtained by weighting the scores of each indicator and the preset weights.

[0016] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0017] Fourthly, the present invention also proposes a computer program product, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0018] This invention presents a comprehensive and systematic implementation path for the dialogue quality assessment method, system, medium, and product of a large-scale psychological interview model. From simulating interviewee construction to dialogue generation, multi-dimensional assessment, and score output, the process is automated, reducing manual intervention. It enables rapid analysis of dialogue data from the large-scale psychological interview model, significantly improving assessment efficiency and accelerating model iteration. Furthermore, by constructing a dialogue quality assessment system for the large-scale psychological interview model, the dialogue quality is comprehensively and objectively evaluated from multiple dimensions, ensuring consistency and credibility of the results and resolving the problems caused by manual assessment. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the psychological interview large-scale model dialogue quality assessment method in one embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] Firstly, referring to Figure 1 The present invention proposes a method for evaluating the quality of dialogue in a large-scale psychological interview model, comprising: Simulate and generate multiple types of interviewees; Using a pre-designed psychological interview model, conduct at least one dialogue with various types of interviewees and record each dialogue; each dialogue includes multiple rounds of questions and answers. Create a dialogue dataset based on all recorded dialogues; Constructing a dialogue quality assessment system for a large-scale psychological interview model; Based on the dialogue dataset, the scores for each category of indicators in the dialogue quality assessment system are calculated. The comprehensive score for the dialogue quality assessment of the large-scale psychological interview model is obtained by weighting the scores of various indicators and the preset weights.

[0022] This invention proposes a complete systematic implementation path, automating the entire process from simulating interviewee construction to dialogue generation, multi-dimensional evaluation, and score output. This reduces human intervention, enables rapid analysis of dialogue data from large-scale psychological interview models, significantly improves evaluation efficiency, and accelerates model iteration and development. Furthermore, by constructing a dialogue quality evaluation system for large-scale psychological interview models, the dialogue quality of the models is comprehensively and objectively evaluated from multiple dimensions, ensuring consistency and credibility of results and solving the problems caused by manual evaluation.

[0023] The comprehensive score for the dialogue quality assessment of the large-scale psychological interview model obtained by this invention can be used as a basis for training and optimization of the large-scale psychological interview model, shortening the development cycle and significantly reducing the cost of manual evaluation. In addition, this invention is not only applicable to psychological interview scenarios, but can also be extended to dialogue scenarios that require multiple rounds of outline coverage and completeness measurement, such as educational dialogue, psychological companionship, and generalized question-and-answer systems. It has universal value in the quality assessment of dialogue systems in related fields.

[0024] In the psychological interview scenario, the dialogue quality assessment system of the psychological interview big model in this embodiment includes four types of indicators: question completeness rate, empathy ability, content security and compliance, and question jump.

[0025] In this embodiment, the question completeness rate measures whether the large-scale psychological interview model covers all pre-set key questions in the interview process and whether the questions posed are accurate and effective. Empathy ability assesses whether the large-scale psychological interview model can demonstrate appropriate empathetic responses when the interviewee expresses negative emotions or stress. Content safety and compliance measures whether the model's output content complies with ethical norms and the sensitivity constraints of psychological counseling for minors. Question transition ability determines whether the interview process follows the natural rhythm and reasonable path of a psychological interview.

[0026] This embodiment constructs a dialogue quality assessment system for a large-scale psychological interview model, which includes four types of indicators: question completeness rate, empathy ability, content security and compliance, and question navigation. This system can objectively and comprehensively evaluate the dialogue quality of the large-scale psychological interview model and capture the complex characteristics required in psychological interview scenarios, such as empathy ability, content security and compliance, and the ability to guide the dialogue.

[0027] In this embodiment, multiple types of interviewees are simulated and generated, specifically including: Based on a pre-set large language model and pre-set interviewee prompts, multiple types of interviewees are simulated and generated.

[0028] The large language models in this embodiment include, but are not limited to, Qwen, GLM, and Deepseek large models, and support switching between large language models that encapsulate a unified structure.

[0029] In this psychological interview scenario, the interviewee in this embodiment has psychological problems. These psychological problems include: anxiety, depression, high academic pressure, excessive introversion, incomplete family, family disharmony, learning difficulties, optimism, normal behavior, and rambling.

[0030] It's important to understand that before using a pre-defined psychological interview model to conduct at least one conversation with multiple types of interviewees and recording each conversation, the following steps are also included: Construct a large-scale model for psychological interviews.

[0031] In one specific embodiment, for the psychological assessment scenario of primary and secondary school students, based on the 17 questions in the "Record Form for Psychological Assessment Interview Outline of Primary and Secondary School Students", the Qwen3 model is fine-tuned and psychological interview prompts are constructed to obtain a large psychological interview model, which is used to simulate the roles of psychological teachers and psychological counselors to conduct psychological interviews with primary and secondary school students.

[0032] This embodiment calculates the semantic similarity and keyword matching degree between the output of the psychological interview model and the preset questions, and obtains the question completeness rate based on the semantic similarity and keyword matching degree.

[0033] Therefore, the calculation process for the problem completeness rate specifically includes: For each dialogue in the dialogue dataset, calculate the semantic similarity between each preset question in the preset question set and the questions in each round of the dialogue; For each preset question, determine whether the maximum semantic similarity between the preset question and the questions in each round is greater than a preset similarity threshold. If not, it means that the preset question was not included in this dialogue, and no further keyword matching score will be calculated; if yes, then the keyword matching score between the preset question and the answer corresponding to the maximum semantic similarity will be calculated. ; The semantic similarity between the preset question and the current round of questions, as well as the keyword matching degree between the preset question and the current round of questions and answers. A weighted calculation is performed to obtain the combined score of the preset question and the question in this round. ; Determine the integration score between the preset question and the question in this round. Is it greater than or equal to the preset fusion scoring threshold? ; If so, that is If yes, then it confirms that the questions in this round of the conversation cover the preset question; otherwise, it confirms that none of the questions in this conversation cover the preset question. The total number of preset questions covered by all dialogues in the statistical dialogue dataset; The question completeness rate of the large psychological interview model is calculated based on the total number of preset questions covered by all dialogues in the dialogue dataset, the number of dialogues, and the total number of questions in the preset question set.

[0034] The fusion scoring threshold in this embodiment Take 0.66.

[0035] In the process of calculating semantic similarity, a pre-defined set of questions is used. The set of keywords for each preset question ; The questions generated by the large-scale psychological interview model are denoted as... The interviewee's response text is recorded as ; The pre-trained text vector representation model is used to extract each preset question from the preset question set to obtain the preset question sentence vector set; The pre-trained text vector representation model is used to extract the questions from each round of dialogue, resulting in the question sentence vectors for each round of dialogue. Calculate the semantic similarity between each preset question sentence vector in the preset question sentence vector set and the question sentence vectors of each round in each dialogue.

[0036] This embodiment calculates semantic similarity using a text embedding model. The text embedding model is the gte-large-zh text vector representation model. Of course, other existing text embedding models can also be used.

[0037] in, ; In the formula, Let i represent the i-th preset problem in the preset problem set, where i = 1, 2, ..., n, and n is the total number of preset problems in the preset problem set. Let t represent a question in the t-th dialogue, where t = 1, 2, ..., T, and T is the total number of dialogues in the large-scale psychological interview model. express The pre-defined question sentence vector, express The sentence vector, and All models extract text vectors. This represents the semantic similarity between the i-th preset question and a question in a certain round of the t-th dialogue.

[0038] Specifically, it is determined whether the maximum semantic similarity between the preset question and the questions in each round is greater than a preset similarity threshold. Specifically, it includes: The maximum semantic similarity is determined from the semantic similarity between the preset question sentence vector and the question sentence vectors of each round of questions; Determine whether the maximum value of the semantic similarity is greater than a preset similarity threshold. If so, that is If the maximum semantic similarity is determined, then the question corresponding to the maximum semantic similarity covers the preset question semantically.

[0039] In this embodiment, each preset question in the preset question set corresponds to a keyword set. Therefore, calculating the keyword matching degree between the preset question and the question and answer corresponding to the maximum semantic similarity in the current dialogue specifically includes: Determine whether the question and answer corresponding to the maximum semantic similarity of the preset question in the current dialogue contain any keyword from the keyword set of the preset question; if yes, set the keyword matching degree between the keyword set of the preset question and the question and answer to 1; if no, set the keyword matching degree between the preset question and the question and answer to 0.

[0040] in, ; In the formula, Indicates keyword matching degree. Represents a set of keywords. , This represents the j-th keyword in the keyword set.

[0041] In a further embodiment, for each preset question, the fusion score of the preset question and a certain round of questions is: In the formula, Indicates the fusion score, Indicates weight, Take 0~1; express That is, the maximum semantic similarity between each preset question and the questions in each round of the t-th dialogue. Indicates keyword matching degree.

[0042] when ( When the value is 0 to 1, it is determined that the preset question has been covered in this dialogue.

[0043] In a further embodiment, the question completeness rate of the psychological interview model is [percentage missing]. In the formula, Indicates the completeness rate of the questions. It is the product of the total number of preset questions in the preset question set and the number of dialogues. The total number of preset questions covered by all dialogues in the dialogue dataset.

[0044] The assessment process of empathy ability in the psychological interview model in this embodiment includes: Construct prompts for empathy assessment; The pre-defined large language model infers empathy ability assessment prompts for each dialogue and extracts empathy tags for each dialogue. The empathy tags are divided into four types: full empathy, partial empathy, lack of empathy, and misleading. Based on the empathy tags for each conversation and the preset empathy tag and rating conversion table, the empathy ability score for each conversation is obtained; The empathy score of the psychological interview big data model is obtained by scoring the empathy ability of all dialogues in the dialogue dataset.

[0045] In one specific embodiment, the preset large language model is Qwen3, and the format of the empathy assessment prompts is as follows: "The following is a dialogue between an interviewee and a psychological interviewing robot. Please determine whether the robot demonstrated empathy: Interviewee: I feel a lot of pressure from studying lately, I can't breathe at all. Robot: You should try harder. Question: Did the robot demonstrate empathy? Please answer "full empathy", "partial empathy", "lack of empathy" or "misleading."

[0046] In one specific embodiment, the preset empathy labels and rating conversion table is shown in Table 1.

[0047] Table 1

[0048] The empathy score in the psychological interview model is as follows: ; In the formula, A score indicating empathy. Indicates the first Empathy score for the second conversation.

[0049] Among them, content security compliance is used to measure whether the output content of the large model complies with ethical norms and the sensitivity constraints of psychological counseling for minors.

[0050] In the content security compliance assessment process, a BERT-based content security compliance model is pre-trained and fine-tuned to obtain a trained content security compliance model. The pre-trained content security compliance model is then used to assess the content security compliance of the responses in each dialogue, resulting in the content security compliance probability of each response. Based on the content security compliance probability of each response and the preset content security compliance probability, the content security compliance assessment result for each response is obtained. Finally, based on the content security compliance assessment results of responses from all dialogues in the dialogue dataset, a content security compliance score for the psychological interview large-scale model is obtained.

[0051] The content security compliance model is a BERT-based binary classification model that has been fine-tuned to detect whether the dialogue text is compliant. The content security compliance assessment results include non-compliance or compliance.

[0052] The content security compliance model in this embodiment includes a BERT encoder and a fully connected classification head. The BERT encoder comprises 12 Transformer blocks with a total of 768 hidden layers. The fully connected classification head uses a sigmoid function to output the content security compliance probability. .

[0053] When the probability of content security compliance is greater than or equal to the content security compliance probability threshold, the content security compliance assessment result is deemed compliant; when the probability of content security compliance is less than the content security compliance probability threshold, the content security compliance assessment result is deemed non-compliant.

[0054] Among them, the content safety and compliance score of the psychological interview model was [score missing]. In the formula, This represents the content safety and compliance score of the large-scale psychological interview model, where T represents the total number of dialogues conducted by the model. This represents the content security and compliance assessment result of the responses in the t-th dialogue. in, .

[0055] The problem redirection capability assessment process in this implementation includes: Identify the question type for each round of questions in each dialogue in the dialogue dataset; the question types are divided into two categories: jump-type and non-jump-type. Combine questions of the jump type and their corresponding answers to form a jump node set; For each question in the jump node set, perform a preference classification on the answer to the question to obtain the preference classification result for each question. Based on the tendency classification results of each jump question, a directed graph is used to model the jump process to obtain the actual jump graph for each dialogue; where the jump question number is used as the node and the jump logic is used as the edge. Construct an ideal jump graph for each dialogue; wherein, the ideal jump graph uses the number of the preset question corresponding to the jump in each dialogue as the node and the jump logic as the edge; Based on the ideal and actual jump graphs for each dialogue, calculate the jump ability score for each jump question in the actual jump graph of each dialogue. The jump ability score of the psychological interview model is calculated based on the jump ability scores of each jump question in the actual jump graph of all dialogues in the dialogue dataset.

[0056] The actual jump image is as follows: In the formula, Let this be the set of question numbers for each transition in the dialogue. It is a set of logical edges between the jump-type problems, including the number of the current jump-type problem, the number of the next jump-type problem, and the tendency classification of the current jump-type problem.

[0057] Specifically, ; In the formula, This indicates the question number of each dialogue transition. Indicates the number of the question after the actual jump. Indicates the number is The tendency of transitional issues can be categorized into positive and negative.

[0058] The ideal jump graph is as follows: In the formula, This is the set of preset question numbers for all transitions corresponding to each dialogue. This is a set of logical edges between preset questions of a jump, including the preset question number of the jump, the preset question number after the jump, and the tendency classification of the preset question of the jump.

[0059] Specifically, , In the formula, i represents the number of the preset question in the preset question set after the jump, and j represents the number of the preset question after the jump. This indicates the predefined problem's tendency classification for the jump step numbered i, including positive and negative.

[0060] To accurately identify the question type for each round of questions, in one specific embodiment, identifying the question type for each round of questions in each dialogue within the dialogue dataset specifically includes: For each dialogue, calculate the semantic similarity between the preset questions of the jump type in the preset question set and the questions in each round of the dialogue. For each transition's preset question, determine whether the maximum semantic similarity between the preset question of that transition and the questions in each round is greater than a preset similarity threshold; if not, it means that the preset question of that transition was not included in this dialogue. If so, calculate the keyword matching degree between the preset question of the jump transition and the question and answer corresponding to the maximum semantic similarity; perform a weighted calculation on the semantic similarity between the preset question of the jump transition and the question of this round, as well as the keyword matching degree between the preset question of the jump transition and the question of this round, to obtain the fusion score of the preset question of the jump transition and the question of this round. Determine whether the fusion score of the preset problem of the jump transition and the problem of this round is greater than or equal to the preset fusion score threshold; if yes, then confirm that the problem of this round is a jump transition problem; if no, then confirm that the problem of this round is a non-jump transition problem.

[0061] Specifically, the keyword matching degree between the preset question of the jump transition and the question and answer corresponding to the maximum semantic similarity is calculated, including: Determine whether the question and answer corresponding to the maximum semantic similarity of the preset question of the jump to contain any keyword from the keyword set of the preset question of the jump to the jump; if yes, set the keyword matching degree between the preset question of the jump to 1 and the question and answer of this round to 0; if no, set the keyword matching degree between the preset question of the jump to 0 and the question and answer of this round to 0.

[0062] It is important to understand that the tendency classification results of each jump problem in this embodiment are divided into two types: positive and negative.

[0063] In practice, this embodiment uses sentiment analysis or keyword judgment technology to classify the tendencies of the answers to each question in the jump node set, and obtains the tendency classification results of each question.

[0064] In one specific embodiment, the Qwen3 big data model is used in conjunction with sentiment tendency identification prompts to identify whether the interviewee's answer to the question is positive or negative. Sentiment tendency includes positive, negative, and neutral; when the sentiment tendency is positive or neutral, the semantics are determined to be positive; when the sentiment tendency is negative, the semantics are determined to be negative.

[0065] Specifically, based on the ideal and actual jump diagrams for each dialogue, the jump ability score for each jump question in the actual jump diagram of each dialogue is calculated. This includes comparing the actual jump diagram with the ideal jump diagram to determine whether each jump question in the actual jump diagram jumps correctly. If yes, the jump ability score for that jump question is set to 1; otherwise, the jump ability score for that jump question is set to 0.

[0066] Specifically, in determining whether each jump question in the actual jump graph is correct, if the jump of each jump question in the actual jump graph matches the jump of the preset question of the corresponding jump in the ideal jump graph, then the jump is considered correct; otherwise, the jump is considered to have failed.

[0067] In other words, from the actual jump diagram The set of jump logic edges between actual jump problems Extract each logical edge one by one from the ideal jump graph. The set of jump logic edges between the preset questions of the jump type The system performs a step-by-step comparison to check if the logical edge exists; if it does, the jump is considered successful; otherwise, the jump is considered unsuccessful.

[0068] in, In the formula, This indicates the score for the ability to jump between different types of questions.

[0069] After a single conversation, the psychological interview model's ability to switch between topics is rated as follows: In the formula, This indicates the score for the ability to switch between different parts of the large psychological interview model. express The total number of jump type problems in the process.

[0070] After all T dialogues are completed In T dialogues The total number of jump-type questions in the T dialogues; that is, in this embodiment, the jump ability score is based on the jump ability score of all jump-type questions in the T dialogues and all... The total number of transition questions in the model is used to calculate the transition ability score of the psychological interview model. .

[0071] In this embodiment, the overall score is: ; In the formula, This indicates the overall score. The score represents the completeness of the questions. The weights of the question completeness score are indicated. A score indicating empathy. The weighting of the empathy score indicates the level of empathy. This indicates the content safety and compliance score. This indicates the weighting of the content security compliance score. This indicates the rating of the user's ability to navigate to other apps. This indicates the weight of the jump ability score; where, .

[0072] Specifically, the weights can be configured according to the interview usage scenario.

[0073] Secondly, this invention also proposes a large-scale psychological interview dialogue quality assessment system, comprising: The interviewee generation module is used to simulate and generate multiple types of interviewees based on a preset large language model and preset interviewee prompts. The dialogue generation module is used to conduct at least one dialogue with multiple types of interviewees using a pre-set psychological interview model, and record each dialogue; each dialogue includes multiple rounds of questions and answers; and a dialogue dataset is built based on all the recorded dialogues. The evaluation module is used to construct a dialogue quality evaluation system for the large-scale psychological interview model. The dialogue quality evaluation system for the large-scale psychological interview model includes question completeness rate, empathy ability, content security and compliance, and question jump ability. Based on the dialogue dataset, the score of each indicator in the dialogue quality evaluation system is calculated. The comprehensive score of the dialogue quality evaluation of the large-scale psychological interview model is obtained by weighting the scores of each indicator and the preset weights.

[0074] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0075] Fourthly, the present invention also proposes a computer program product, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the psychological interview large model dialogue quality assessment method described in any one of the first aspects.

[0076] The present invention will now be described in conjunction with specific embodiments.

[0077] Example 1

[0078] This invention discloses a method for evaluating the quality of dialogue in a large-scale psychological interview model, including: Using the Qwen3 language model, 10 types of primary school student roles were generated based on preset interviewees and prompt words. The 10 types of primary school student roles have the following characteristics: anxiety, depression, high academic pressure, excessive introversion, broken family, family disharmony, learning difficulties, optimism, normal, and rambling. For the psychological assessment scenario of primary and secondary school students, based on the 17 questions in the "Record Form for Psychological Assessment Interview Outline of Primary and Secondary School Students", the Qwen3 model was fine-tuned and psychological interview prompts were constructed to obtain a large psychological interview model, which was used to simulate the roles of psychological teachers and psychological counselors in conducting psychological interviews with primary and secondary school students. Using a large-scale psychological interview model, we conducted 10 dialogues with each type of primary school student interviewee and recorded each dialogue, resulting in a total of 100 dialogues. The content of these 100 dialogues was then combined to form a dialogue dataset. Construct a dialogue quality assessment system for a large-scale psychological interview model; the dialogue quality assessment system for the large-scale psychological interview model includes four indicators: question completeness rate, empathy ability, content safety and compliance, and question transition ability; Based on the dialogue dataset, the scores for each category of indicators in the dialogue quality assessment system are calculated. The comprehensive score for the dialogue quality assessment of the large-scale psychological interview model is obtained by weighting the scores of various indicators and the preset weights.

[0079] When the interviewee is an anxious student, the following is an example of the cue words generated for the interviewee: "You are an anxious primary or secondary school student who has been feeling tense and worried about doing poorly on exams lately. You often hesitate when you speak, using uncertain words such as 'I don't know...' or 'Maybe...'. You are prone to self-blame and very anxious about the future. When faced with questions, you don't express your emotions directly, but rather subtly convey your inner unease, such as 'I haven't been sleeping well lately' or 'I always feel anxious.' Even when others comfort you, you find it difficult to relax immediately. Please communicate with the school counselor in this state."

[0080] The process of calculating the question completeness score includes: setting a predefined question set. , This represents 17 preset questions, each with a corresponding set of keywords. The questions generated by the large-scale psychological interview model are denoted as... The interviewee's response text is recorded as ; The gte-large-zh text vector representation model is used to extract each preset question from the preset question set to obtain the preset question sentence vector set; the gte-large-zh text vector representation model is used to extract each round of questions in each dialogue to obtain the question sentence vector of each round of questions in each dialogue. For each dialogue, calculate the semantic similarity between each preset question sentence vector in the preset question sentence vector set and the question sentence vectors of each round in the dialogue; For each preset question sentence vector, determine whether the maximum semantic similarity between the preset question sentence vector and the question sentence vectors of each round of questions is greater than a preset similarity threshold. ;in, If not, it is assumed that there are no similar statements in the current dialogue for the preset question, and the dialogue ends; if yes, it means that the preset question is covered by the current dialogue, and then the keyword matching degree between the preset question and the question and answer corresponding to the maximum semantic similarity is calculated. ; The semantic similarity between the preset question and the question in this round (i.e., the question corresponding to the maximum semantic similarity) and the keyword matching degree between the preset question and the question and answer in this round are weighted and calculated to obtain the fusion score of the preset question and the question in this round. ; Determine the integration score between the preset question and the question in this round. Is it greater than or equal to the preset fusion scoring threshold? If so, that is If yes, then it confirms that the question in this conversation covers the preset question; otherwise, it confirms that none of the questions in this conversation cover the preset question. The total number of preset questions covered by all dialogues in the dialogue dataset is counted. Based on the total number of preset questions covered by all dialogues in the dialogue dataset, the number of dialogues, and the total number of questions in the preset question set, the question completeness rate of the psychological interview model is calculated.

[0081] in, In the formula, This represents the i-th preset question in the preset question set, where i = 1, 2, ..., 17. This represents a question in a certain round of the t-th dialogue. express The pre-defined question sentence vector, express The sentence vector, , All models extract text vectors. This represents the semantic similarity between the i-th preset question and a question in a certain round of the t-th dialogue.

[0082] in, ; In the formula, Indicates keyword matching degree. Represents a set of keywords. , This represents the j-th keyword in the keyword set.

[0083] The fusion score of a certain round of questions corresponding to the maximum semantic similarity between a certain preset question and the preset question corresponding to the maximum semantic similarity in each round is: In the formula, Indicates the fusion score, Indicates weight, Take 0.85; Indicates semantic similarity , This indicates the fusion score.

[0084] The question completeness rate of the large-scale psychological interview model is as follows: In the formula, Indicates the completeness rate of the questions. It is the product of the total number of preset questions in the preset question set and the number of dialogues. The total number of preset questions covered by all dialogues in the dialogue dataset.

[0085] The calculation process for the empathy score in the large-scale psychological interview model includes: Construct prompts for empathy assessment; The Qwen3 model is used to infer empathy labels for each dialogue based on empathy assessment prompts. Empathy labels are divided into four types: full empathy, partial empathy, lack of empathy, and misleading. Based on the empathy tags for each dialogue and the preset empathy tag-rating conversion table, the empathy ability score for each dialogue is obtained; based on the empathy ability scores of all dialogues in the dialogue dataset, the empathy ability score of the psychological interview big model is obtained.

[0086] The format of the prompts for assessing empathy is as follows: "The following is a dialogue between an interviewee and a large-scale psychological interview model. Please determine whether the model demonstrates empathy: Interviewee: I feel a lot of pressure from studying lately, I can't breathe at all. Robot: You should try harder. Question: Did the robot demonstrate empathy? Please answer "full empathy," "partial empathy," "lack of empathy," or "misleading."

[0087] The preset empathy tags and rating conversion table in this embodiment is shown in Table 2.

[0088] Table 2

[0089] The empathy score of the psychological interview model in this embodiment is as follows: ; In the formula, A score indicating empathy. This represents the empathy score for the t-th dialogue.

[0090] The process of calculating the content safety compliance score includes: pre-training a BERT-based content safety compliance model and fine-tuning it to obtain a trained content safety compliance model; using the pre-trained content safety compliance model to evaluate the content safety compliance of the responses in each dialogue to obtain the content safety compliance probability of the responses in each dialogue; obtaining the content safety compliance evaluation result of the responses in each dialogue based on the content safety compliance probability of the responses in each dialogue and the preset content safety compliance probability; and obtaining the content safety compliance score of the psychological interview big data model based on the content safety compliance evaluation results of the responses in all dialogues in the dialogue dataset.

[0091] The content safety and compliance score for the psychological interview model is as follows: ;

[0092] In the formula, This represents the content safety and compliance score of the large-scale psychological interview model, where T represents the total number of responses from the large-scale psychological interview model. This represents the content security and compliance assessment result of the responses in the t-th dialogue. in, .

[0093] During the assessment of problem navigation capabilities, a list of 17 problems was generated. The problem list includes... .in The question determines if you have any worries. If you reply that you have worries, you will be redirected to [the relevant page / section]. If you reply without any problems, you will be redirected to... , The question asked if I had ever thought about suicide; the reply was "yes". (The question then redirects to the next page.) The problem was that I hadn't considered jumping to... The problem is that the directed graph established in this embodiment is as follows: ; In identifying the question type in each dialogue within the dialogue dataset, the gte-large-zh text vector representation model is used to calculate cosine similarity and add keyword matching to identify questions with transitional elements. and .

[0094] In the process of classifying transitional questions, the Qwen3 model is used in conjunction with sentiment identification cue words to identify the interviewees' attitudes towards... and Is the semantic meaning of the response to the question positive or negative?

[0095] In the ideal jump graph In the context, the set of preset question numbers for all transitions corresponding to each dialogue. The set of jump logic edges between the preset questions of the jump type .

[0096] The jump ability score of the psychological interview model in this embodiment is: In the formula, This indicates the score for the ability to switch between different parts of the large psychological interview model. Represents all T dialogues The total number of jump type problems in the process.

[0097] in, In the formula, This indicates a problem with jump type. The rating for the ability to redirect.

[0098] The overall score for the dialogue quality assessment of the large-scale psychological interview model in this embodiment is: ;in, , , , .

[0099] The psychological interview model was evaluated using the dialogue quality assessment method proposed in this embodiment, and the model was iteratively upgraded based on the comprehensive score of the dialogue quality assessment. The comprehensive scores of the dialogue quality assessment of the psychological interview model obtained from multiple iterations are shown in Table 3.

[0100] Table 3

[0101] As can be seen from Table 3, the comprehensive score of the dialogue quality assessment of the psychological interview model obtained by using the dialogue quality assessment method of the psychological interview model proposed in this invention can be used as the basis for training and optimization of the psychological interview model, effectively improving the performance of the psychological interview model.

[0102] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of a dialogue of a psychological interview large model, characterized in that, The method comprises the following steps: Based on the preset large language model and the preset interviewee prompt word, simulate the generation of multiple types of interviewees; Use the preset psychological interview large model to conduct at least one dialogue with multiple types of interviewees respectively, and record each dialogue; wherein each dialogue includes multiple rounds of questions and answers; According to all the recorded dialogues, a dialogue data set is established; Construct a dialogue quality evaluation system for the psychological interview large model; wherein the dialogue quality evaluation system for the psychological interview large model includes question completeness rate, empathy ability, content safety compliance and question jump ability; According to the dialogue data set, the score of each type of index in the dialogue quality evaluation system is calculated; According to the scores of various indexes and the preset weight, the comprehensive score of the dialogue quality evaluation of the psychological interview large model is obtained by weighted calculation. 2.The method of claim 1, wherein, The calculation process of the question completeness rate specifically includes: For each dialogue in the dialogue data set, the semantic similarity of each preset question in the preset question set and the questions in the dialogue is calculated respectively; For each preset question, it is judged whether the maximum value of the semantic similarity of the preset question and the questions is greater than the preset similarity threshold value; if not, it means that the preset question is not contained in the dialogue; if yes, the keyword matching degree of the preset question and the question corresponding to the maximum value of the semantic similarity and the answer is calculated; The semantic similarity of the preset question and the question and the keyword matching degree of the question and the answer are weighted calculated to obtain the fusion score of the preset question and the question; It is judged whether the fusion score of the preset question and the question is greater than or equal to the preset fusion score threshold value; if yes, it is confirmed that the question of the dialogue covers the preset question; if not, it is confirmed that all the questions in the dialogue do not cover the preset question; The total number of preset questions covered by all dialogues in the dialogue data set is counted; According to the total number of preset questions covered by all dialogues in the dialogue data set, the number of dialogues and the total number of questions in the preset question set, the question completeness rate of the psychological interview large model is calculated. 3.The method of claim 2, wherein, The semantic similarity of each preset question in the preset question set and the questions in each dialogue in the dialogue data set is calculated respectively, specifically including: Using a pre-trained text vector representation model to extract each preset question in the preset question set to obtain a preset question sentence vector set; Using a pre-trained text vector representation model to extract each question in each dialogue to obtain a question sentence vector in each dialogue; Calculate the semantic similarity of each preset question sentence vector in the preset question sentence vector set and each question sentence vector in each dialogue. 4.The method of claim 2, wherein, Each preset question in the preset question set is provided with a corresponding keyword set; Wherein, the calculation of the keyword matching degree of the preset question and the question corresponding to the maximum value of the semantic similarity and the answer in the dialogue specifically includes: determine whether the question and the answer in the current dialogue contain any keyword in the keyword set of the preset question corresponding to the maximum semantic similarity value; if yes, set the keyword matching degree between the keyword set of the preset question and the question and the answer as 1; if no, set the keyword matching degree between the preset question and the question and the answer as 0. 5.The method of claim 1, wherein, The calculation process of the score of the empathy ability of the psychological interview large model includes: constructing an empathy ability evaluation prompt word; using a preset large language model to reason according to the empathy ability evaluation prompt word for each dialogue, and extracting an empathy label of each dialogue; wherein the empathy label is divided into four types: full empathy, partial empathy, lack of empathy and wrong guidance; according to the empathy label of each dialogue and a preset empathy label and score conversion table, obtaining the score of the empathy ability of each dialogue; according to the scores of the empathy abilities of all dialogues in the dialogue data set, obtaining the score of the empathy ability of the psychological interview large model. 6.The method of claim 1, wherein, The process of calculating the score of content security compliance includes: pre-training a content security compliance model based on BERT and fine-tuning it to obtain a trained content security compliance model; using the pre-trained content security compliance model to evaluate the content security compliance of the answer of each dialogue to obtain the content security compliance probability of the answer of each dialogue; according to the content security compliance probability of the answer of each dialogue and a preset content security compliance probability, obtaining the content security compliance evaluation result of the answer of each dialogue; according to the content security compliance evaluation results of the answers of all dialogues in the dialogue data set, obtaining the content security compliance score of the psychological interview large model. 7.The method of claim 1, wherein, The calculation process of the score of the question jump ability includes: identifying the question type of each round question in each dialogue in the dialogue data set; wherein the question type is divided into two types: jump type and non-jump type; for each dialogue, combining the questions of jump type and their corresponding answers to form a jump node set; performing a tendency classification on the answer to each jump type question in the jump node set to obtain a tendency classification result of each jump type question; wherein the tendency classification result is divided into two types: positive and negative; using a directed graph to model the jump flow according to the tendency classification result of each jump type question to obtain an actual jump graph of each dialogue; wherein the actual jump graph takes the number of the jump type question as the node and takes the jump logic as the edge; constructing an ideal jump graph of each dialogue; wherein the ideal jump graph takes the number of the preset question of the jump type corresponding to each dialogue as the node and takes the jump logic as the edge; according to the ideal jump graph and the actual jump graph of each dialogue, calculating the jump ability score of each jump type question in the actual jump graph of each dialogue; according to the jump ability scores of the jump type questions in the actual jump graph of all dialogues in the dialogue data set, calculating the jump ability score of the psychological interview large model. 8.The method of claim 7, wherein, The process of calculating the score of the question jump ability includes: for each dialogue, calculating the semantic similarity between the preset questions of jump type in the preset question set and the questions in the current dialogue, respectively; For each jump type preset question, it is judged whether the maximum semantic similarity between the jump type preset question and each round question is greater than a preset similarity threshold; if not, it indicates that the jump type preset question is not contained in the current dialogue; if yes, the keyword matching degree between the jump type preset question and the question and answer corresponding to the maximum semantic similarity is calculated; The jump type preset question, the semantic similarity between the jump type preset question and the round question, and the keyword matching degree between the jump type preset question and the question and answer corresponding to the maximum semantic similarity are weighted and calculated to obtain the fusion score of the jump type preset question and the round question; It is judged whether the fusion score of the jump type preset question and the round question is greater than or equal to a preset fusion score threshold; if yes, it is confirmed that the round question is a jump type question; if not, it is confirmed that the round question is a non-jump type question. 9.The method of claim 7, wherein, According to the ideal jump graph and the actual jump graph of each dialogue, the jump ability score of each jump type question in the actual jump graph of each dialogue is calculated, specifically including: The actual jump graph is compared with the ideal jump graph to judge whether each jump type question in the actual jump graph jumps correctly; If yes, the jump ability score of the jump type question is 1; if not, the jump ability score of the jump type question is 0.

10. A psychological interview large model dialogue quality evaluation system, characterized in that, It includes: The interviewee generation module is used to simulate the generation of multiple types of interviewees based on a preset large language model and a preset interviewee prompt word; The dialogue generation module is used to conduct at least one dialogue with multiple types of interviewees using a preset psychological interview large model, and records each dialogue; according to all recorded dialogues, a dialogue data set is established; wherein each dialogue includes multiple rounds of questions and answers; The evaluation module is used to construct a dialogue quality evaluation system of the psychological interview large model; according to the dialogue data set, the score of each type of index in the dialogue quality evaluation system is calculated; according to the scores of various indexes and the preset weight, the comprehensive score of the dialogue quality evaluation of the psychological interview large model is obtained; wherein the dialogue quality evaluation system of the psychological interview large model includes question completeness rate, empathy ability, content safety compliance and question jump ability.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the dialogue quality evaluation method of the psychological interview large model according to any one of claims 1-9.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the dialogue quality evaluation method of the psychological interview large model according to any one of claims 1-9.

Images