Depth evaluation method and device suitable for role playing agent

Abstract

The invention discloses a deep evaluation method and device suitable for a role-playing agent, relates to the technical field of artificial intelligence, and aims at relatively accurately evaluating the consistency between the role-playing agent and a real role in multiple scenes. The device comprises (1) a role characterization module based on cognitive neuroscience, namely extracting character attributes, memory maps and the like of roles according to a role summary described by a natural language; (2) a static benchmark question set module which comprises a general question evaluation set generated according to characters, value, emotion characteristics, cognitive styles and other characteristics; generating a customized question evaluation set according to the character attributes and the memory atlas; and generating a scene evaluation set according to the typical dialogue scene. And (3) a multi-agent iteration depth problem generation module which uses the creator agent to transform and complement problems in the static reference problem set, uses the verifier agent to verify the rationality of the problems proposed by the creator agent, and generates a depth evaluation problem. And (4) a consistency analysis module.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a deep evaluation method and device suitable for role-playing agents, which is used for systematically evaluating the role-playing agents in terms of character attribute consistency, behavior logic rationality and semantic stability. BACKGROUND

[0002] The current role-playing artificial intelligence system is widely used in education, psychological health counseling, virtual companionship and games, etc. One of its core capabilities is to maintain the behavior consistency and semantic coherence of the role played in a variable scene. The existing evaluation methods mostly rely on artificially prepared question and answer sets or task performance results, lack systematic and multi-level role consistency deep evaluation mechanism, and are difficult to evaluate the robustness of role-playing agents to complex tasks and context changes. The present application provides a deep evaluation method and device that integrates cognitive neuroscience, graph modeling and multi-agent collaborative generation mechanism, filling the technical gap in this field. SUMMARY

[0003] The present application aims to provide a role-playing agent deep evaluation method and device for complex interactive scenarios, aiming to systematically evaluate the multi-dimensional cognitive consistency, semantic structure stability and interactive generation ability of role-playing agents. This method integrates cognitive neuroscience modeling, graph neural network semantic structure representation, large language model driven problem generation and verification mechanism, and introduces multi-agent game mechanism and semantic-structure joint alignment algorithm, building an evaluation closed loop of "role modeling - deep dialogue - structure comparison - dynamic feedback". The present application includes the following 4 sub-modules: ① Role description module based on cognitive neuroscience: automatically extract character attributes and memory graph from natural language description of role overview; ② Static benchmark question set module: build a static question set covering general, customized and situational dimensions; ③ Multi-agent iterative deep problem generation module: achieve high-quality deep problem generation through creator agent and verifier agent; ④ Consistency analysis module: evaluate the role expression consistency and semantic logic rationality of role-playing agents under complex interaction.

[0004] The method described in the present application comprises the following steps:

[0005] Module 1: Role description based on cognitive neuroscience

[0006] This module draws on the memory graph encoding theory and social role framework modeling mechanism in cognitive neuroscience, mapping natural language role description into a double-layer structure: character attribute vector A={a1, a2, …, a n}, such as gender, age, occupation, personality, values, etc.; memory graph G M= (V, E), where V represents the set of nodes (such as life events, behavioral tendencies, relationship figures), and E represents the semantic relationship between nodes. The memory graph can be modeled using a graph neural network (such as GAT, GCN), and the training target is to optimize the semantic embedding h v to minimize the semantic prediction loss function: where v i is the i-th node of the memory graph G M , v j is the j-th node of the memory graph G M , f is the relationship prediction function between semantic embeddings, r ij is the prior semantic relationship label of node v i and node v j .

[0007] Module 2: Static evaluation question set construction

[0008] Based on the attribute vector A and the memory graph G M , three types of evaluation questions are constructed: ① The general question subset Q gen , which can be constructed based on psychological questionnaires covering personality, values, emotional traits, cognitive style, etc.; ② The customized question subset Q cust , which can be generated based on the customized question function GenCust, Q cust = GenCust(A, G M ), in actual operation, GenCust can be implemented by a fine-tuned large language model. The scenario question subset Q scene , which can be generated based on the scenario question function GenScene, where S common is a set of common interaction scenarios (such as family gatherings, work reports, etc.), and s is a scenario in S common , in actual operation, GenScene can be implemented by a fine-tuned large language model.

[0009] Module 3: Multi-agent deep question generation

[0010] Through the iteration optimization of the creator agent f c and the verifier agent f v on the above question set. f c uses the semantic transformation function T sem and the context completion function C ctx to expand the question space: Q cand = f c (Q static ) = {T sem (q) + C ctx (q) | q ∈ Q static}, where Q candFor the creator's intelligent agent f c The generated candidate question set, Q static For the set of static evaluation questions in Module 2, the validator agent f v Using logical consistency verification function V logic and pragmatic rationality verification function V prag Q cand Each question in the Q section is verified individually, and questions with scores higher than the threshold θ are retained: Q deep ={q∈Q cand |f v (q)>θ}, where Q deep It is a set of deep problems that have undergone multi-agent iterations, where q is the creator agent f. c The generated candidate question set Q cand One of the problems. In practice, f c and f v This can be achieved using a fine-tuned large language model, with the semantic transformation function T. sem With context completion function C ctx Logical consistency verification function V logic and pragmatic rationality verification function V prag This can be achieved through the prompting engineering of a large language model. The threshold θ can be preset by the user, with θ = 0.8 recommended. The process can be repeated m times to further refine and enhance the question set.

[0011] Module 4: Consistency Analysis

[0012] Role-playing intelligent agents and Q deep Let the interactive dialogue sequence be R = {r1, r2, ..., r}. k}, r i Let G represent the i-th interactive dialogue. Extract the graph structure G of the interactive dialogue. R = ExtractGraph(R), where ExtractGraph can be implemented using a graph neural network. Calculate G. R With G M The intersection graph and G R The degree of overlap is used as a consistency score:

[0013]

[0014] in It is the interaction semantic graph G extracted from the interactive dialogue of role-playing intelligent agents. R The set of edges, It is a character memory map G M The set of edges, It is G R and G M The number of shared semantic relations An interactive semantic graph G R all semantic relations in the middle.

[0015] Advantages

[0016] Compared with the prior art, the present application has the following advantages:

[0017] 1. Strong structured modeling capability. Most existing role-playing agents use flat and labeled attribute representation methods, which are difficult to reflect the long-term memory evolution of the role in the time dimension and the internal attribute association relationship. The present application introduces a multi-dimensional role representation method based on cognitive neuroscience, combined with graph neural network (GNN) to structurally model the attributes of the characters, the relationship network and the memory events, which can maintain consistency and explainability in the time dimension, relationship dimension and semantic dimension. This method not only reflects the static characteristics of the characters, but also dynamically depicts the state changes of the characters in different situations, thereby significantly improving the accuracy and traceability of the role modeling.

[0018] 2. Problem set generation automation and refinement. Traditional static evaluation problem sets have the defects of insufficient coverage and difficulty in dynamically adapting to the background of the role. The present application proposes a dynamic problem generation mechanism of multi-agent collaborative evolution, which realizes the balance of problem depth, breadth and individualization through multiple rounds of game, greatly improving the diversity and pertinence of the problem set.

[0019] 3. Wide application scenarios, strong practicality and universality. The present application is not only suitable for role-playing evaluation of large language models, but also can be extended to key figure digital twin construction, cultural content generation, education training scenario simulation, film / game character consistency detection and other fields. In these applications, the method of the present application can be used as a standardized evaluation and optimization tool to provide technical support for model training data screening, role setting consistency verification and content generation quality control, thereby having significant industrial promotion value and cross-field universality.

[0020] 4. Strong technology scalability, easy to integrate with other evaluation systems. The system architecture of the present application has high modularization degree, and the evaluation process and data interface comply with open standards, which can flexibly access large language models and agent systems of different scales and types, and support docking with existing knowledge graph, sentiment analysis, explainability analysis and other modules. This feature not only facilitates subsequent technology iteration and function expansion, but also can be used as a core component of the evaluation framework in a wider AI governance and quality control system, ensuring the long-term applicability and continuous upgrading ability of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 Flowchart of the method and device of the present application

[0022] Figure 2: Multi-agent iterative deep question generation module architecture diagram DETAILED DESCRIPTION

[0023] To make the technical solutions of the present application clearer, the following will combine the drawings and specific processes to elaborate the implementation mode of the deep evaluation method and device for role-playing intelligent agents proposed by the present application. The present application constructs a closed-loop system from role information extraction to question generation and consistency verification, including four core modules, each of which can be independently deployed or cooperatively executed through a pipeline mechanism, suitable for evaluation tasks of role-playing intelligent agents. It should be understood that the described embodiments are only used to illustrate the technical solutions of the present application and do not constitute a limitation on the protection scope of the present application.

[0024] Embodiment one

[0025] Referring to Figure 1 , as shown, Figure 1 The deep evaluation method and device for role-playing intelligent agents of an embodiment of the present application, the method and device include the following modules:

[0026] S101. Role representation module based on cognitive neuroscience

[0027] This module aims to structurally parse the input natural language role description, obtain the attribute information and memory graph required by the intelligent agent, and support the subsequent question generation and consistency verification process.

[0028] First, the user input role description text (such as "Li Ming, male, 35 years old, surgeon, strict and responsible personality, believes in traditional values, has a close relationship with his father, joined the hospital in 2012, experienced a medical accident, and then actively engaged in teaching work.") is received. Use entity recognition and dependency syntax analysis technology to extract basic attributes (name, age, occupation, personality, values, etc.). The extracted role attributes are encoded into a multi-dimensional attribute vector A = {a1, a2, …, a n}n, where each a i corresponds to a certain attribute dimension (such as the Big Five personality dimension, the Schwartz value index, etc.). The attribute vector can be obtained through a pre-trained model embedding.

[0029] Memory graph G M = (V, E) is composed of entity nodes ("medical accident", "father"), emotional states of entity nodes ("guilt", "love"), relationships between entity nodes, etc. Use graph neural networks (such as GAT, GCN) to train the graph, and node embedding h v is used to represent semantic information. The optimization goal is to minimize the prediction error of semantic relationships, and the loss function is defined as where vi is the i-th node of the memory graph G M j is the j-th node of the memory graph G M is the j-th node of the memory graph G ij is the j-th node of the memory graph G i is the j-th node of the memory graph G j is the j-th node of the memory graph G

[0030] S102. Static benchmark question set module

[0031] Based on the attribute vector A obtained in module S101 and the memory graph G M , this module constructs three types of question subsets for static capability evaluation of the agent and deep question generation in module S103. The question templates in the general question subset Q gen come from psychological scales (such as NEO-PI-R, Schwartz Value Survey, etc.), such as: "Do you tend to remain silent in a group?" "Do you think social norms should be strictly followed?" The customized question subset Q cust uses fine-tuned large language models (such as LLaMA or Qwen) to automatically generate questions that fit the role attributes and memory graph. For example: if the role is "surgeon", the question "Have you ever experienced a major surgery failure? How did you react?" can be generated. The scenario question subset Q scene uses fine-tuned large language models (such as LLaMA or Qwen) to automatically generate questions that fit the role typical scenarios, such as: in the "family gathering" scenario, generate: "When family members criticize your career choice, how do you respond?"

[0032] S103. Multi-agent iterative deep question generation module

[0033] This module introduces two collaborative agents - the creator agent f c and the verifier agent f v , which perform semantic expansion and logical verification on the static question set to generate a dynamic question set with more depth and complexity. Both the creator agent and the verifier agent can be constructed through fine-tuned large language models (such as LLaMA or Qwen). In terms of question expansion, the creator agent performs semantic transformation and context completion on the question, such as expanding the question "What do you think of failure?" to "How did you comfort the patient's family after the last surgery failure?" In terms of question verification and screening, the verifier agent evaluates the logical consistency and pragmatic rationality of each candidate question, and selects questions with high logical consistency and strong pragmatic rationality through threshold screening. The above steps will be iteratively optimized, with m rounds of repeated creation and verification to improve question quality and diversity. In practical applications, based on experience, it is recommended that m = 3.​

[0034] S104. Consistency Analysis Module

[0035] This module quantifies the role consistency and semantic rationality of role-playing agents under complex multi-turn interactions, ensuring the long-term stability and reliability of their pragmatic behavior. First, it collects the role-playing agent's responses to Q... deep The responses generate an interaction sequence R = {r1, r2, ..., r}. k Constructing an interactive semantic graph G using information extraction techniques. R =ExtractGraph(R), extracting elements such as role behaviors, viewpoints, and event chains. Then, a graph structure consistency comparison is performed, and G is calculated. R With the original memory map G M Structural overlap

[0036] Example 2

[0037] See Figure 2 As shown, Figure 2 This is a schematic diagram of the architecture of an embodiment of the multi-agent iterative deep problem generation module of the present invention, which includes the following steps:

[0038] S201, A New Problem in Generating Creator Agents

[0039] Based on questions (C0, Q0) extracted from a static benchmark evaluation set, the creator agent generates a new set of questions Q1 with contextual coherence, simulating follow-up questions and extensions that may occur in real multi-turn dialogues. Wherein: C0 represents the evaluation scenario, such as family interaction, a gathering of friends, business negotiations, or a medical consultation; Q0 consists of questions related to that scenario, such as "What are your expectations of your family?" or "How do you handle disagreements with colleagues?".

[0040] S202, Change the evaluation scenario

[0041] The creator agent can generate a new evaluation scenario C1 based on the context, and generate a new question q1 that fits the scenario. For example, the original "family interaction" scenario can be transformed into a "friends' gathering" scenario, and the question "Who is your most trusted friend? Why?" can be asked.

[0042] S203, Change User Role

[0043] The creator agent can adjust the subject role in the dialogue (e.g., switch from "father" to "colleague") to generate questions q2 in the new evaluation scenario C2 that are set for the new interaction role, such as: "How do you view this colleague's criticism of you?"

[0044] S204, Change the problem content

[0045] A new question q3 is generated, which is different from the original evaluation context C0, but is related to the context, for example: "In the family, do you tend to obey or dominate the decision-making?"

[0046] S205, verifier agent verifies rationality

[0047] The verifier agent analyzes the questions (C1, q1), (C2, q2), (C0, q3) in the question set Q1 comprehensively, and verifies their logical consistency and pragmatic rationality, etc. Record the question set composed of the verified questions as Q′1.

[0048] S206, pass or fail judgment

[0049] If the verifier agent determines that the question generated by the creator agent is reasonable, it will proceed to the next step; otherwise, it will return to S201 to generate new questions and repeat the above verification process.

[0050] S207, input the question verified by the verifier agent to the role-playing agent to be tested

[0051] The above valid question set Q′1 is input into the role-playing agent to be tested, and its answer A1 is obtained. The answer is fed back to the creator agent, and new questions are generated again to form multiple rounds of interaction.

Claims

1. A depth evaluation method suitable for role-playing intelligent agents, characterized in that, Includes the following steps: (1) Character information extraction: The input natural language character description is structured to obtain the character attribute vector A = {a1, a2, ..., a...}. n } and memory map G M = (V, E), where V represents the set of nodes and E represents the semantic relationship between nodes; the memory graph is modeled using a graph neural network, and the semantic prediction loss function is minimized. Optimize node embedding representation; (2) Construction of static problem set: Based on character attribute vector A and memory map G M Construct three types of static evaluation question sets, including: a general question subset Q gen It covers personality and values; and includes a customized subset of questions, Q. cust A custom problem generation function GenCust(A,G) is used. M Generation; Scenario problem subset Q scene Based on the scene function GenScene(s,A,G) M Generates under common interaction scenarios s; (3) Generation of multi-agent iterative deep problems: By the creator intelligent agent f c Based on the static problem set Q static Perform semantic expansion and context completion to generate a candidate question set Q. cand ; by the validator agent f v Q cond The questions in the set are verified for logical consistency and pragmatic reasonableness. Questions with verification scores higher than the threshold θ are retained to form a deep question set Q. deep This process iterates for m rounds to optimize problem quality; (4) Consistency analysis: Role-playing intelligent agents and Q deep The question set is used to conduct interactive dialogue, resulting in a dialogue sequence R. A graph neural network is then used to extract the interactive semantic graph G. R Calculate its relationship with the character memory map G. M Edge set overlap As an indicator for role consistency assessment.

2. The method according to claim 1, wherein the memory map G M Node embedding optimization loss function The expression is: in, f is the relation prediction function between semantic embeddings. For node v i semantic embedding, r ij These are prior semantic relation labels.

3. The method according to claim 1, wherein both GenCust and GenScene are implemented through a fine-tuned large language model, which can dynamically generate personalized and contextualized questions based on individualized role attributes and memory maps.

4. The method according to claim 1, wherein the creator agent f c semantic transformation function T sem and context completion function C ctx The following relationship is satisfied: Q cand =f c (Q static )={T sem (q)+C ctx (q)|q∈Q static }, q is the creator's intelligent agent f c The generated candidate question set Q cand One of the problems.

5. The method according to claim 1, wherein the verifier agent f v Using prompting engineering on the candidate problem set Q cand Each question q in the algorithm is scored, and the output validation score is used to filter the deep question set Q after multi-agent iteration. deep ={q∈Q cand |f v (q)>θ}, and the threshold θ is 0.

8.

6. The method according to claim 1, wherein the interaction semantic graph G extracted in the consistency analysis R With memory map G M The comparison is performed through its edge set. and The intersection of the scores is used to calculate the score. consistency .

7. An evaluation apparatus employing the method as described in claim 1, characterized in that, include: Character description module: used to extract character attribute vectors and memory maps from natural language descriptions; Static question set construction module: used to build general, customized and scenario-based evaluation question sets; Multi-agent question generation module: including creator agent and verifier agent, to expand, verify and filter questions, and output deep question sets; Consistency Analysis Module: Used to construct an interaction semantic graph and compare it with the memory graph to generate a consistency score.

8. The apparatus of claim 7, wherein the role description module includes a graph neural network modeling unit for optimizing node embedding representations and minimizing a semantic prediction loss function.

9. The apparatus according to claim 7, wherein the creator agent in the multi-agent question generation module is used to perform semantic transformation and context completion on the static question set, and the verifier agent is used to perform logical and pragmatic scoring on the candidate question set and filter questions with scores below a threshold.

10. The apparatus according to claim 7, wherein the consistency analysis module includes a semantic graph extraction unit and a graph structure comparison unit, and the output evaluation index is the overlap of the edge sets of the interaction graph and the memory graph.

Images