Intelligent agent evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides an agent evaluation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of agents, large language models and the like. The intelligent evaluation method comprises the steps that a first agent and a second agent are called to execute a simulation task to obtain dialogue data, the simulation task is used for at least one simulation dialogue between multiple first objects and a second object generated by the second agent, and the multiple first objects are generated by the first agent and have different dialogue styles; and inputting the dialogue data into at least one evaluation large model of the third agent to obtain a target dialogue feature, and outputting an evaluation result of the second agent based on the target dialogue feature.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as intelligent agents and large language models, and specifically to intelligent agent evaluation methods, devices, electronic devices, intelligent agents, and storage media. Background Art

[0002] In the business of merchant intelligent agents, the evaluation effect of the intelligent agents is limited by the scarcity of real dialogue samples, which leads to limited confidence in the evaluation conclusions, high evaluation difficulty and high evaluation cost. The small number of real dialogue samples has become a bottleneck restricting the further development of the intelligent agent business. Summary of the invention

[0003] The present disclosure provides a method, an apparatus, an electronic device, an agent, and a storage medium for agent evaluation.

[0004] According to one aspect of the present disclosure, there is provided an agent evaluation method, comprising: calling a first agent and a second agent to perform a simulation task to obtain dialogue data, the simulation task being used for a plurality of first objects to conduct at least one simulated dialogue with a second object generated by a second agent, the plurality of first objects being generated by the first agent and having different dialogue styles; inputting the dialogue data into at least one evaluation macro model of a third agent to obtain target dialogue features, and outputting an evaluation result of the second agent based on the target dialogue features.

[0005] According to another aspect of the present disclosure, there is provided an agent evaluation device, comprising: a calling module, used to call a first agent and a second agent to perform a simulation task to obtain dialogue data, the simulation task is used for a plurality of first objects to conduct at least one simulated dialogue with a second object generated by a second agent, the plurality of first objects are generated by the first agent and have different dialogue styles; an evaluation module, used to input the dialogue data into at least one evaluation model of a third agent to obtain target dialogue features, and output an evaluation result of the second agent based on the target dialogue features.

[0006] According to another aspect of the present disclosure, an artificial intelligence agent is provided, which is configured to execute the method provided by the embodiment of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 A schematic diagram schematically shows an application scenario in which the agent evaluation method and device according to an embodiment of the present disclosure can be applied;

[0013] Figure 2 A flowchart of an intelligent agent evaluation method according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 3 A schematic diagram schematically shows generation of conversation data according to an embodiment of the present disclosure;

[0015] Figure 4 The schematic diagram schematically shows the principle of the second agent evaluation according to the embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram of an interactive interface according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 6 The schematic diagram schematically shows the principle of adjusting the second agent according to the embodiment of the present disclosure;

[0018] Figure 7 A schematic diagram of task allocation of a second agent in practical application according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 8 The structure block diagram of the intelligent agent evaluation device according to the embodiment of the present disclosure is schematically shown;

[0020] Fig. 9 The structure diagram of the intelligent agent according to the embodiment of the present disclosure is schematically shown;

[0021] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] Figure 1 A schematic diagram of an application scenario to which the intelligent agent evaluation method and device according to an embodiment of the present disclosure can be applied is schematically shown.

[0024] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the agent evaluation method and apparatus can be applied may include a terminal device of a client, but the terminal device may implement the agent evaluation method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0025] like Figure 1 As shown, in the application scenario 100, the terminal device 102 of the client can communicate with the server 103 of the service end via the network. The user 101 can send a dialogue request through the interactive interface provided by the terminal device 102, and the terminal device 102 can send the dialogue request to the server 103, so that the server 103 calls the intelligent agent to have a dialogue with the user 101 and outputs a dialogue reply. The server 103 sends the dialogue reply to the terminal device 102, so that the terminal device 102 displays the dialogue reply to the user 101 through the interactive interface.

[0026] The terminal device 102 may be configured with various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc. Various communication client applications may be installed on the terminal device 102, such as instant messaging tools, smart assistant applications, search applications, knowledge reading applications, web browser applications, email clients, and / or social platform software, etc. (only as examples). The user 101 may input a dialogue request in the interactive interface of these client applications, and these client applications will display the generated dialogue response to the user 101.

[0027] Server 103 may be a server that provides various services, such as a background management server that provides support for the content browsed by user 101 using the interactive interface of terminal device 102 (for example only). The background management server may call an intelligent agent to generate an object for dialogue with user 101 in response to the dialogue request received from user 101, and make a dialogue reply according to the dialogue content of the dialogue request, feed back the dialogue reply to terminal device 102, and display it through the interactive interface. Server 103 may also be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). Server 103 may also be a server of a distributed system, or a server combined with blockchain.

[0028] It should be noted that the agent evaluation method provided in the embodiment of the present disclosure can generally be executed by the server 103. Accordingly, the agent evaluation device provided in the embodiment of the present disclosure can also be set in the server 103. The agent evaluation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 103 and can communicate with the terminal device 102 and / or the server 103. Accordingly, the agent evaluation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 103 and can communicate with the terminal device 102 and / or the server 103.

[0029] Alternatively, when the terminal device 102 also has computing power and computing resources that meet the requirements, the agent evaluation method provided in the embodiment of the present disclosure can generally be executed by the terminal device 102. Accordingly, the agent evaluation device provided in the embodiment of the present disclosure can generally be set in the terminal device 102.

[0030] It should be understood that Figure 1 The number of terminal devices and servers in the embodiment is only for illustration. Any number of terminal devices and servers may be provided according to implementation requirements.

[0031] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0032] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0033] In view of this, an embodiment of the present disclosure provides an agent evaluation method, which utilizes a first agent to simulate multiple different users and conducts a large number of simulated conversations with a second agent to simulate a business object, so as to generate a large amount of simulated conversation data, and thereby evaluate the simulation ability of the second agent.

[0034] The following will be combined Figure 2~Figure 7 The agent evaluation method provided by the present disclosure is described in detail.

[0035] Figure 2 The flowchart of the agent evaluation method according to an embodiment of the present disclosure is schematically shown.

[0036] like Figure 2 As shown, the method 200 may include operations S210 to S220.

[0037] In operation S210, the first agent and the second agent are called to perform a simulation task to obtain dialogue data. The simulation task is used for multiple first objects to conduct at least one simulated dialogue with a second object generated by the second agent. The multiple first objects are generated by the first agent and have different dialogue styles.

[0038] The first object may represent an individual user, a corporate user, an industry professional, or a group organization, etc. The first object may initiate a dialogue request to the second object according to its own needs to obtain information of interest to the second object. The first object has multiple personality characteristics such as identity attributes, living habits, behavioral habits, language habits, personal preferences, etc. Personality characteristics will affect the conversation style of the first object in the conversation. The personality characteristics of different first objects are more or less different, resulting in different conversation styles of different first objects.

[0039] The second object can be an intelligent customer service, a virtual personal assistant, a robot, an intelligent device, an intelligent professional assistant, and an intelligent system, etc. The second object has characteristic information such as industry category, business scope, business model, marketing characteristics, and market positioning. The second object can understand the intention and request of the first object by parsing the conversation content of the first object through the natural language processing technology it relies on, and obtain information related to the intention and request of the first object by searching the knowledge base, information retrieval, etc., integrate the acquired information, and integrate the characteristic information of the second object with the acquired knowledge based on the natural language processing technology to generate a reply dialogue to the first object.

[0040] For example, the first object is an individual user, and the second object is an intelligent customer service. When a user wants to learn about a product, he can ask the intelligent customer service "Which smart homes have a calling function"; after the intelligent customer service receives the user's request for "Which smart homes have a calling function", it understands the information the user wants to know based on the language model, obtains relevant information about the smart home by combining the product information of the brand to which the intelligent customer service belongs, organizes it into colloquial reply content through the language model, and feeds it back to the user through a dialogue, such as "This brand's smart refrigerators, smart TVs, smart speakers and smart headphones have a calling function, and the specific product models involved include...", thereby completing a dialogue.

[0041] Intelligent agents include systems or entities that can autonomously perceive the environment, make decisions, and perform actions to complete specific tasks. Large models can provide decision support for intelligent agents and provide them with the ability of reasoning analysis and task planning. Intelligent agents can use the analysis results of large models to execute or optimize their decision-making process. Intelligent agents can integrate multiple large models to handle different types of tasks.

[0042] The first agent can simulate different first objects according to different conversation styles to generate conversation content. The first agent can be constructed based on a large amount of historical conversation data and conversation style of the first object. During the conversation process, the first agent simulates the conversation style of the first object and generates personalized conversation content in combination with the current conversation context, thereby simulating the conversation behavior of real users and being able to have a natural and realistic conversation with the second agent.

[0043] The second agent can imitate the second object's speech according to the second object's configuration information, and output the dialogue content in response to the first object within the scope of the answer and the scope of the business. During the dialogue process, the second agent can understand the first object's intention and request through the language model, and obtain knowledge information related to the first object's intention and request by searching the knowledge base, information retrieval, etc., and then cater to the user's preferred dialogue style to generate a reply dialogue to the first object.

[0044] For example, after the second object simulated by the second agent obtains knowledge information related to the user's request, for users who prefer concise answers, the first agent will give priority to providing concise replies, which only include relevant knowledge information without excessive rhetoric; and for users who like detailed explanations, the first agent will provide more detailed answers.

[0045] The simulation task is a dialogue task created by the R&D personnel when the dialogue ability of the second agent needs to be evaluated. The simulation task can guide the first agent to generate multiple first objects with different dialogue styles, and guide the second agent to generate a second object that can dialogue with the first object.

[0046] According to the embodiments of the present disclosure, by calling the first agent and the second agent to perform a simulation task, the first agent imitates the conversation style of multiple first objects and conducts at least one conversation with the second object, thereby simulating the conversation process between the first object and the second object in the actual scene, and can efficiently generate a sufficient amount of simulated conversation data, providing an evaluation basis for evaluating the conversation ability of the second agent.

[0047] In operation S220, the dialogue data is input into at least one evaluation model of the third agent to obtain target dialogue features, and an evaluation result of the second agent is output based on the target dialogue features.

[0048] The third agent can evaluate the second agent's ability to understand, answer, manage, and perform dialogue based on the dialogue data. Each evaluation model can include a general model, and can also include a general model that is fine-tuned according to the evaluation requirements of specific indicators. Each evaluation model can output the evaluation results of the second agent in one or more dimensions based on the input dialogue data. Exemplarily, the evaluation results of the second agent can be obtained based on the outputs of all evaluation models.

[0049] Currently, the evaluation of the second agent is performed manually based on the dialogue data in the actual scene. However, the dialogue data in the actual scene is scarce, and the manual evaluation depends on the experience and time of the evaluator, so there is a lack of efficient and accurate evaluation methods.

[0050] According to an embodiment of the present disclosure, an automated evaluation method for the second agent is provided based on a third agent, and the second agent's feedback of the first object's dialogue content is evaluated based on the dialogue data generated by the simulated dialogue. The third agent can evaluate the second agent's multi-dimensional dialogue ability based on at least one evaluation macro model to timely discover the second agent's problems.

[0051] According to the agent evaluation method provided by the embodiment of the present disclosure, the first agent imitates multiple first objects with different conversation styles, and the second agent imitates the second object, so as to simulate different users having conversations with the second object, thereby efficiently generating a sufficient amount of simulated conversation data, and providing an evaluation basis for evaluating the conversation ability of the second agent; the method evaluates the ability of the second agent through the third agent, provides an automated evaluation method, and improves the evaluation efficiency.

[0052] It should be noted that the second agent in the simulation evaluation environment should be consistent with the second agent in the actual application environment to ensure the accuracy, reliability, predictability, verification, debugging and optimization efficiency, and cross-platform compatibility of the simulation results. This helps to better understand and evaluate the performance of the agent in the actual application environment and provide strong support for the development and optimization of the system.

[0053] Among them, the simulation evaluation environment can reuse some stateless modules and large model modules in the real application environment. Stateless modules refer to modules in the software architecture or system that do not save or rely on any state information related to a specific user session (Session) or request, such as database access modules, cache services, account authentication modules, etc.; large model modules are the core components of the intelligent agent technology architecture, which integrates deep learning, natural language processing, multimodal processing, decision reasoning, knowledge base and memory system, tool and interface integration, and continuous optimization and updating. The iteration hotspot modules and stateful modules of the intelligent agent in the simulation evaluation environment are isolated from the iteration hotspot modules and stateful modules of the intelligent agent in the actual application environment to ensure that the simulation evaluation environment is isolated from the traffic data in the actual application environment, so as to minimize the impact of the simulation evaluation environment on the actual application environment. The hotspot iteration module includes the base model, the policy management module, the background management module and the session management module. The base model is actually the intelligent agent. The policy management module provides strategies for generating conversation content, the background management module is used to process conversation data, and the session management module is used to manage conversation processes. The stateful module includes Redis state storage, which is used to store state data generated in the conversation.

[0054] By reusing and isolating the agent's simulation evaluation environment and the real application environment, the integrated deployment of the agent in the simulation evaluation environment and the real application environment can be achieved. The simulation configuration in the simulation environment is updated synchronously according to the configuration update of the online environment, so that the agent in the simulation environment can play the same function and role as the agent in the actual application.

[0055] Figure 3 The diagram schematically shows a schematic diagram of generating conversation data according to an embodiment of the present disclosure.

[0056] like Figure 3As shown, in the application scenario 300 of calling the first agent 331 and the second agent 332 to perform the simulation task 333 and generate the dialogue data 336, the client 302 can provide an interactive interface for the R&D personnel and can receive the simulation request submitted by the R&D personnel through the interactive interface. The simulation request can be generated according to the operation instructions, action operations, voice instructions and other instruction forms of the R&D personnel, which are not limited here. The simulation request is used to submit the simulation task 333. The first agent 331 and the second agent 332 can be deployed in the server 303, and the simulation task 333 is executed based on the computing power of the server 303.

[0057] In some embodiments, the simulation task 333 includes multiple task data, each task data includes first portrait information for configuring the first agent 331 to generate a first object 334 and second portrait information for configuring the second agent 332 to generate a second object 335; wherein the first portrait information of multiple task data for the same simulation task 333 are different from each other and the second portrait information of multiple task data are the same as each other.

[0058] The first portrait information may include the identity attributes, social attributes, living habits, preferred behaviors, historical behaviors and other information of the first object 334. For example, when the object is a user, the object portrait can be obtained based on information such as age, gender, occupation, identity, tone, personality, conversation method, historical information and preferences. For example, the source of the user's first portrait information includes the user's historical conversation records, purchase behavior, browsing history, interest tags, feedback information, etc. Through steps such as data cleaning, feature extraction and modeling, the portrait features of each user are constructed. These features may include the user's common questions, preferred answer methods, purchase tendencies, common communication languages, etc.

[0059] The second portrait information may include information such as entity attributes, social activities, business scope, user group characteristics, etc. based on the second object 335. For example, when the second object 335 is an intelligent customer service that provides services, its second portrait information can be obtained based on its industry category, brand, business scope, marketing concept, marketing model, service content, customer group, etc.

[0060] A simulation task 333 includes a plurality of task data, each of which corresponds to first portrait information of a first object 334, so that the first agent 331 can generate a plurality of different first objects 334. In the same simulation task 333, the second object 335 remains unchanged, and the second portrait information of the second object 335 included in different task data is the same. In the task data, the first portrait information can be portrait information of various customer groups such as real individual users, enterprises, industry professionals or dealers, and the second portrait information can be feature information of the second object 335 such as real enterprises, merchants, brands and products about industry categories, business scopes, business models, marketing characteristics, market positioning, etc.

[0061] In some embodiments, calling the first agent 331 and the second agent 332 to perform the simulation task 333 and obtaining the dialogue data 336 includes: obtaining multiple task data of the simulation task 333; sending the multiple task data to the first agent 331 and the second agent 332, the first agent 331 generating multiple first objects 334 according to the first portrait information, and the second agent 332 generating the second object 335 according to the second portrait information; and conducting a dialogue between the multiple first objects 334 and the second objects 335 based on the first agent 331 and the second agent 332 to obtain the dialogue data 336.

[0062] As Figure 3 Taking the embodiment shown as an example, the first agent 331 can generate n first objects 334, i.e., simulated user 1 to simulated user n, based on the first portrait information; the second agent 332 can generate a second object 335, i.e., intelligent customer service, based on the second portrait information. The first agent 331 and the second agent 332 perform the simulation task 333 to realize the dialogue between the n simulated users and the intelligent customer service, wherein each simulated user has at least one dialogue with the intelligent customer service, thereby generating a large amount of dialogue data 336.

[0063] For example, in a simulation task 333, the first agent 331 generates simulated user 1, simulated user 2 and simulated user 3. Simulated user 1 is an individual user who wants to know the functions of a smart refrigerator, simulated user 2 is an enterprise that wants to purchase a batch of computers, and simulated user 3 is a dealer who wants to know the marketing model of lighting products; the intelligent customer service generated by the second agent 332 is an online robot customer service of a certain brand, which can provide users with information about the brand introduction, related products, product distribution model and other information of the brand; through the first agent 331 and the second agent 332, the simulated user 1, the simulated user 2 and the simulated user 3 respectively have conversations with the intelligent customer service, so that the intelligent customer service provides simulated user 1 with an introduction to the functions of smart refrigerators of different models of the brand, provides simulated user 2 with relevant information such as model information, quotation information and performance parameters of the brand's computer products, and provides simulated user 3 with various marketing models such as distribution methods, marketing concepts, marketing channels and other marketing models of the brand's lighting products.

[0064] In some embodiments, the first agent 331 generates a first object 334 based on the first portrait information, and the second agent 332 generates a second object 335 based on the second portrait information, including: extracting language features in the first portrait information based on the first agent 331, and constructing a first object 334 with language features in the dialogue behavior; extracting business features in the second portrait information based on the second agent 332, and constructing a second object 335 with business features in the dialogue behavior; wherein the language features are related to the business features.

[0065] The language features in the first portrait information may include the language habits of the first object 334 (e.g., vocabulary, grammar, pauses, idioms, etc.), historical conversation content features (e.g., historical conversation focus, historical preferences, historical responses to a certain conversation topic, etc.), etc.; the business features in the second portrait information may include the business operation information, product marketing information, marketing methods, reply skills, blocked words, etc. of the second object 335. Among them, the language features of the first object 334 may reflect the business-related content such as the business category, business product or business information that the first object 334 is interested in, that is, the language features are related to the business features of the second object 335.

[0066] Taking the first object 334 as an individual user as an example, the first portrait information of the individual user may include his recent historical conversation records (for example, communication records of inquiring about the video playback effect of a certain category of electronic products from online customer service) and language habits (for example, preference for concise conversations). Based on the first portrait information, the chat topic preferred by the individual user (about "video playback effect of a certain category of electronic products") and the language characteristics of conversation efficiency (efficient conversation) can be extracted, wherein the chat topic reflects the product feature of the electronic product about the video playback effect that the individual user wants to know (i.e., a business feature); taking the second object 335 as intelligent customer service as an example, the first object 334 of its service covers a variety of different customer groups such as individual users, enterprises, industry professionals or dealers. In the process of daily service, it may have conversations with the above-mentioned different customer groups, and provide brand introduction, product introduction, technical descriptions of the product and its marketing methods and other business services to different customer groups. The brand concept, product characteristics, service characteristics, technical means, market positioning, marketing model, etc. of the above business services can serve as its business characteristics. In view of the language characteristics of individual users with specific chat topics, the first agent 331 generates a first object 334 with a chat topic of "video playback effect of electronic products", and the second agent 332 generates a second object 335 that can provide video playback effect information of this type of electronic products. A conversation is conducted between the first object 334 and the second object 335 to generate conversation data 336 about the product feature of "video playback effect of electronic products".

[0067] In some embodiments, multiple conversations between the first object 334 and the second object 335 are conducted based on the first agent 331 and the second agent 332 to obtain conversation data 336 including: generating a first speech of the first object 334 based on the first agent 331, the first speech having language features and including query information related to business features; generating a second speech of the second object 335 based on the second agent 332, the second speech being feedback information of the query information; and conducting multiple rounds of conversations between the first object 334 and the second object 335 to obtain conversation data 336.

[0068] For example, in a simulation task 333, the first agent 331 generates simulated user 1, simulated user 2 and simulated user 3. Each user wants to know the information of a commodity (i.e., business characteristics). The intelligent customer service generated by the second agent 332 is an intelligent assistant of a certain brand, which can provide users with information about the brand introduction, related products, product distribution model, etc. of the brand; simulated user 1, simulated user 2, and simulated user 3 can gradually put forward the information they need through multiple conversations. The intelligent customer service can gradually provide users with complete feedback information according to the needs of each conversation. The feedback information conforms to the language characteristics of the user's habits or preferences, thereby forming multiple natural and smooth conversations with the user and generating conversation data 336.

[0069] Adding language features and business features to the first speech of the first object 334 can enhance the understanding and response capabilities of the second agent 332, and guide the second agent 332 to generate a more accurate second speech for the second object 335, which not only improves the user experience, but also enhances the adaptability and intelligence level of the agent.

[0070] In some embodiments, generating a second speech of a second object 335 based on the second agent 332 includes: retrieving knowledge data matching the query information from a business knowledge base; inputting the knowledge data into the second agent 332 , and generating a second speech in combination with the business characteristics of the second object 335 .

[0071] The business knowledge base stores knowledge data that supports the agent's ability to understand, make decisions, and execute in a specific business field. Knowledge data may include business information, business data, document materials, professional knowledge (such as domain knowledge, technical knowledge, and technical cases), relationship data (such as customer relationships, supply chain relationships, and internal relationships), dynamic data (such as real-time data and forecast data), and knowledge graphs (such as the internal organizational structure, business processes, product relationships, and other complex information of the enterprise. These data together constitute the "brain" of the agent in the business field, enabling it to more accurately understand business scenarios, make decisions, and execute tasks.

[0072] For example, in response to the first object 334 raising a query about the marketing plan for electronic products, first, the second agent 332 can retrieve and obtain knowledge data such as marketing information, marketing methods and marketing cases of electronic products from the business knowledge base; then, based on the second object 335 generated by the second agent 332 as a smart device, such as a smart phone, the second agent 332 combines the acquired knowledge data to generate a set of product marketing plans for the smart phone, and generates a second speech introducing the product marketing plan in the identity of the smart phone and feeds it back to the first object 334.

[0073] Figure 4The schematic diagram schematically shows the principle of the second agent evaluation according to the embodiment of the present disclosure.

[0074] like Figure 4 As shown, in the application environment of the server 403, the dialogue data 436 generated by the first agent and the second agent performing the simulation task is sent to the third agent 437 for evaluation; the third agent 437 may include multiple evaluation large models 438, which can evaluate the second agent's various dialogue capabilities based on the dialogue data 436.

[0075] In some embodiments, the evaluation big model 438 includes a topic evaluation big model 4381, inputting the conversation data 436 into at least one evaluation big model 438 of the third agent 437 to extract target conversation features, and outputting an evaluation result 439 for the second agent based on the target conversation features, including: inputting the conversation data 436 into the topic evaluation big model 4381, extracting a first descriptive feature of the first object and a second descriptive feature of the second object in the conversation data 436, and outputting an evaluation result 439 of the correlation between the first descriptive feature and the second descriptive feature, wherein the first descriptive feature and the second descriptive feature are related to the conversation topic.

[0076] The first description feature and the second description feature respectively include feature information related to the conversation topic of the first object and the second object in the conversation, and the feature information related to the conversation topic includes the conversation content, conversation context and language style. In the conversation, the conversation content of the two conversation objects has the same or similar description features, which help to maintain the coherence, consistency and effectiveness of the conversation.

[0077] The topic evaluation model 4381 can be a general model or a model obtained by fine-tuning the general model according to the evaluation requirements of specific indicators. The topic evaluation model 4381 can automatically mine and extract key topic features from the dialogue text by using advanced natural language processing technology and deep learning algorithms. It has a strong unsupervised learning ability and can automatically discover potential and meaningful topic structures in the dialogue without manually annotated data. In addition, the topic evaluation model 4381 can also use the method of probabilistic generation model to reveal the internal logic and topic distribution of the dialogue content by modeling the probability distribution relationship between vocabulary and topics. When dealing with complex dialogues, the topic evaluation model 4381 can show the advantages of its multi-level structure, not only identifying broad topics, but also going deep into more specific sub-topic levels, thereby achieving a comprehensive and in-depth understanding of the dialogue content. At the same time, it has the ability to efficiently process large-scale data sets, can cope with the challenges of long dialogues and multi-round dialogues, and ensure that the extraction of dialogue topic features is completed within a reasonable time.

[0078] In addition, the topic evaluation model 4381 can also have good scalability and adaptability, and can continuously update and optimize its topic feature extraction capabilities as new conversation data 436 is added to meet the needs of conversation analysis in different fields and scenarios. In terms of semantic understanding, the topic evaluation model 4381 can fully consider the contextual information of the conversation, accurately understand the meaning of words and sentences in a specific context, and thus more accurately identify the topic and key information in the conversation.

[0079] The topic evaluation model 4381 extracts the first descriptive feature of the first object and the second descriptive feature of the second object in the conversation data 436, and calculates the correlation between the first descriptive feature and the second descriptive feature. The correlation can be used as an evaluation result 439 for evaluating the coherence, consistency and effectiveness of the second intelligent agent in the conversation, thereby expressing the second intelligent agent's understanding of the conversation content of the first object in the conversation.

[0080] The method for calculating the correlation between the first description feature and the second description feature may include semantic similarity calculation, topic model calculation, machine learning model or deep learning surface model calculation and knowledge base matching calculation, etc., which are not specifically limited here.

[0081] For example, assuming that the first descriptive feature and the second descriptive feature extracted by the topic assessment model 4381 are sentences about the topic of the conversation, the sentences corresponding to the first descriptive feature and the second descriptive feature are converted into vector representations (for example, through a sentence embedding model), and then the similarity between these vectors is calculated to evaluate the relevance of the conversation content.

[0082] In some embodiments, the evaluation big model 438 includes a matching evaluation big model 4382, inputs the dialogue data 436 into at least one evaluation big model 438 of the third agent 437 to extract target dialogue features, and outputs an evaluation result 439 for the second agent based on the target dialogue features, including: obtaining a set of standard answers to a set of questions in the dialogue data 436; inputs the dialogue data 436 and the set of standard answers into the matching evaluation big model 4382, and obtains an evaluation result 439 of the degree of matching between the question set and the set of standard answers.

[0083] The standard dialogue set can be a series of pre-designed dialogue instances with clear topics and dialogue flows. These dialogue instances should cover various scenarios and situations that the agent may encounter, including but not limited to: dialogues in professional fields, complex situational dialogues, and daily dialogues. Among them, dialogues in professional fields include common questions and answers in specific fields designed according to the agent, such as medical, legal, and educational fields; complex situational dialogues include dialogues involving complex situations such as multiple topic transitions, emotional exchanges, and logical reasoning; daily dialogues include conversational content such as greetings, self-introductions, and inquiries about the time and weather. Evaluating dialogue accuracy: The standard dialogue set provides an objective basis for evaluating the accuracy of agent dialogues.

[0084] The matching evaluation model 4382 extracts the set of questions and answers of the second subject in the dialogue in the dialogue data 436, and compares the set of questions and answers with the standard answers, so as to quantitatively evaluate the ability of the agent in understanding the dialogue content and generating reasonable responses, thereby evaluating the accuracy of the answer content generated by the second agent. The evaluation result output by the matching evaluation model 4382 reflects the answering ability of the second agent.

[0085] For example, the dialogue data 436 includes the following dialogue content: "User: What is the weather like tomorrow? Voice assistant: Tomorrow's weather is expected to be sunny and suitable for outdoor activities.", wherein the user is the first object simulated by the first agent, and the voice assistant is the second object generated by the second agent. The standard answer to the dialogue data 436 is "There will be showers tomorrow, please be prepared for rain."; Based on the dialogue data 436, the matching evaluation model 4382 performs the following workflow: First, extract the second object's question answer set "Tomorrow's weather is expected to be sunny and suitable for outdoor activities."; Then, compare the question answer set with the standard answer, and the matching evaluation model 4382 will compare the similarity and difference between the voice assistant's answer and the standard answer, which usually involves text similarity calculation, keyword matching, semantic analysis and other technologies, and quantitatively evaluate the similarity and difference between the question answer set and the standard answer, and output an evaluation result 439. In this embodiment, when answering the inquiry about tomorrow's weather, the answer given by the voice assistant does not match the standard answer, indicating that its accuracy in understanding and responding to the dialogue content is low. Based on the above evaluation results, the third agent 437 can believe that the voice assistant’s ability to answer this question is weak because it does not accurately understand the question and give the correct answer.

[0086] The matching evaluation model 4382 needs to be based on a massive amount of training data, which should cover a wide range of topics and fields to ensure that the model can understand and generate conversation content related to various topics. The matching model needs to have a broad knowledge base, accurate language understanding, flexible reasoning, efficient comparison algorithms, adaptability and robustness, explainability, and continuous learning capabilities. These features work together to enable the model to maintain a high degree of accuracy and reliability in complex and changing conversation environments.

[0087] In some embodiments, the evaluation big model 438 includes an evaluation evaluation model 4383, inputting the dialogue data into at least one evaluation big model 438 of the third agent 437 to extract the target dialogue features, and outputting the evaluation result 439 of the second agent based on the target dialogue features, including: inputting the dialogue data 436 into the evaluation evaluation big model 4383, extracting the evaluation description of the first object on the simulated dialogue in the dialogue data 436, and outputting the evaluation result 439 of the second agent on the efficiency of understanding the simulated dialogue based on the evaluation description.

[0088] The evaluation description characterizes the first subject's satisfaction with the conversation service provided by the second subject. Based on the evaluation description of the first subject, the second agent's ability and efficiency in interacting with the first subject in the conversation can be reflected, which is an important part of the operating power. Therefore, by evaluating the evaluation description of the second agent in the simulated conversation, we can indirectly evaluate its operating power in terms of improving customer satisfaction, enhancing customer interaction, and improving operational efficiency.

[0089] When the conversation data 436 is input into the evaluation model 4383, the evaluation model 4383 will automatically analyze and extract the evaluation description of the first subject on the performance of the second agent in the simulated conversation. Based on these evaluation descriptions, the evaluation model 4383 will comprehensively consider the second agent's understanding accuracy, response speed, and ability to solve user problems in the conversation, and evaluate its operating ability in improving user satisfaction, optimizing customer experience, and enhancing business efficiency.

[0090] For example, suppose there is a simulated conversation data in which the first object (user) communicates with the second agent (such as intelligent customer service), and the conversation data is as follows: "User: "Hello, I would like to check the status of my recent order." Intelligent customer service: "OK, please tell me your order number." User: "The order number is XYZ123." Second agent: "Thank you for your information. I found that your order has been shipped and is expected to arrive tomorrow." User: "Wow, it was shipped so quickly! You understand it very quickly and your answers are very accurate. Thank you very much! ””; Input this conversation data into the evaluation and assessment model 4383, and the workflow of the evaluation and assessment model 4383 is as follows: the above conversation is fully input into the evaluation and assessment model 4383; the evaluation and assessment model 4383 identifies and extracts the evaluation description of the first object (user) on the second agent (intelligent customer service) from the conversation: "Wow, the goods were shipped so quickly! You understand very quickly, and your answers are also very accurate. Thank you very much!"; The evaluation and assessment model 4383 analyzes this evaluation description and identifies the keywords "quick understanding speed" and "accurate answers". These keywords are directly related to the efficiency of the second agent's understanding of the simulated conversation; based on the above evaluation description, the evaluation and assessment model 4383 outputs an evaluation result 439 of the second agent's understanding efficiency. For example, it can be a high score under a scoring system, indicating that the second agent performs well in understanding user intentions, responding quickly, and providing accurate information. This evaluation result 439 can be used as a positive indicator for evaluating the operating ability of the second agent (intelligent customer service). In the evaluation of business capability, quickly and accurately understanding user needs and giving satisfactory responses is one of the key factors in improving customer satisfaction, enhancing customer interaction and improving business efficiency. Therefore, the excellent performance of the second agent in this regard reflects its good business capability.

[0091] Figure 5 A schematic diagram of an interactive interface according to an embodiment of the present disclosure is schematically shown.

[0092] like Figure 5 As shown, in some embodiments, during the process of the first agent and the second agent performing the simulation task, the dialogue data 501 can be output on the interactive interface 500; and in response to the at least one evaluation model outputting the evaluation result 502 of the second agent, the evaluation result 502 is output on the interactive interface. The dialogue data may include data in various forms such as text data, image data, chart data, audio data, and video data.

[0093] For example, the conversation data of user M simulated by the first agent is "searching for the career of athlete A", and the intelligent assistant simulated by the second agent replies to the conversation about describing the personal profile of athlete A, and the conversation content may include relevant photos of athlete A, charts describing the types of events in which athlete A participates, tables describing the awards won by athlete A, interview audio of athlete A, and video of athlete A's competition, etc. By clicking on the corresponding conversation content, the user can further understand the detailed information of the relevant photos, charts, tables, audio and video. The intelligent assistant can also convert the relevant content into text form and output it in the interactive interface by understanding the content of the relevant photos, charts, tables, audio and video. After the conversation data 501 is generated, the evaluation result 502 obtained according to the conversation data can be output below the interactive interface in the form of a numerical value.

[0094] According to the embodiments of the present disclosure, the process of generating dialogue data can be displayed to the R&D personnel while the simulation task is being executed, so that the R&D personnel can monitor the generation of dialogue data in real time and judge the dialogue ability of the second agent by intuitive feeling at the first time. The evaluation result 502 output by the third agent can be diverse, and can reflect the evaluation quantitative values ​​of the second agent in the three major dialogue abilities of understanding, answering, and management, so that the R&D personnel can have a concrete understanding of the dialogue ability of the second agent.

[0095] It should be noted that the sources of the data displayed on the interactive interface 500 are all in compliance with relevant regulations and are legal, and the display method and display content can comply with relevant regulations.

[0096] Figure 6 The schematic diagram schematically shows the principle of adjusting the second intelligent agent according to an embodiment of the present disclosure.

[0097] like Figure 6 As shown, in some embodiments, in the server 603 equipped with the second agent 623, the following operations can be repeated to adjust the second agent 632 until the evaluation result 639 of the second agent 632 meets the preset conditions: adjust the model parameters and business knowledge base of the second agent 632 based on the evaluation result 639 to obtain the adjusted second agent 632; call the first agent 631 and the adjusted second agent 632 to perform the simulation task to obtain new dialogue data; and input the new dialogue data into the third agent 637 to obtain a new evaluation result 639 of the second agent 632, wherein the preset condition represents that the evaluation is higher than the preset threshold. The experience value provided by the second agent 632 to the first object during the dialogue process can be a comprehensive evaluation value of multiple evaluation results 639 obtained by evaluating the second agent 632 by multiple evaluation models of the third intelligence.

[0098] For example, assuming that the experience value given to the user by the second agent 632 is higher than 85 points (out of 100 points), it means that the second agent 632 meets the expected requirements. The experience value can be obtained by the R&D personnel based on expert knowledge or a series of indicators. According to the dialogue data obtained from the simulation task, if according to the evaluation result 639 of the second agent 632, the experience value of the second agent 632 is only 75 points, then the parameters of the second agent 632 are adjusted, and its business data is updated, adding more latest information and FAQs in related fields; then, simulating a real user scenario, let the first agent 631 simulate the user's dialogue with the second agent 632 to generate new dialogue data; based on the new dialogue data, the third agent 637 re-evaluates the second agent 632 to obtain a new evaluation result 639. If according to the new evaluation result 639, the experience value of the second agent 632 reaches 88 points, it means that the conversation accuracy of the second agent 632 has been significantly improved and has met the expected requirements, and the adjustment of the second agent 632 can be stopped; if according to the new evaluation result 639, the experience value of the second agent 632 is only 80 points, repeat the above steps and continue to adjust the second agent 632 until its experience value is higher than 85 points.

[0099] According to the embodiments of the present disclosure, by repeatedly optimizing the parameters and business knowledge base of the second agent 632, the adaptability and generalization ability of the second agent 632 can be continuously improved, so that the target second agent 632 after training can still maintain high-quality dialogue output in the scenario where there is a lack of sufficient real dialogue data.

[0100] In some embodiments, after the evaluation result of the second agent meets preset conditions, the second agent can be put into actual application scenarios, including: in response to a dialogue request, calling N second agents to respectively perform multiple dialogue tasks, the dialogue tasks are used to generate a second object based on the second agent, and conduct at least one real-time dialogue with multiple real first objects based on the second object, where N is an integer greater than 1; in response to at least a portion of the multiple dialogue tasks not being completed, selecting at least one target second agent with an added value higher than a preset threshold from the N second agents, the added value being generated based on the task status of each of the multiple dialogue tasks; and allocating at least a portion of the multiple dialogue tasks that have not been completed to at least one target second agent for processing.

[0101] Figure 7 The following schematically shows a task allocation diagram of a second agent in actual application according to an embodiment of the present disclosure.

[0102] like Figure 7As described, in the actual application scenario 700, there are I users 701 who simultaneously send out dialogue requests 702 to an intelligent customer service 704. The server equipped with a second agent generates M dialogue tasks 703 based on the dialogue requests 702 of the I users, where M is less than or equal to I, and one dialogue task 703 may include dialogue requests 702 of multiple users; the server distributes the M dialogue tasks 703 to the intelligent customer services 704 generated by N second agents, and the N intelligent customer services 704 are used to process the M dialogue tasks 703, where N is less than or equal to M; in the process of processing the dialogue tasks 704, affected by factors such as the amount of user dialogues, the number of users, the content of user dialogues, and the network status, there may be different processing efficiencies of different second agents. Some intelligent customer services 704 have high processing efficiency and may even be idle after processing the dialogue tasks 703. Therefore, they can be selected as target intelligent customer services 706 to help other intelligent customer services 704 that have not completed the dialogue tasks 705. Since the intelligent customer service 704 (ie, the second object) generated by a second agent is fixed, the target intelligent customer service 706 can only handle dialogue tasks with the same intelligent customer service 704 generated by it to ensure the efficiency and accuracy of dialogue processing.

[0103] According to the embodiments of the present disclosure, after the second agent is evaluated, the second agent that meets the preset evaluation conditions can be put into practical application; when there are multiple dialogue tasks, multiple second agents can be enabled to process the dialogue tasks simultaneously to improve the dialogue processing efficiency; when the number of dialogue tasks is large, the unfinished dialogue tasks can be allocated to the second agent with high processing efficiency to achieve load balancing and improve the overall processing efficiency of the second agent.

[0104] Through the intelligent agent evaluation method provided by the present invention, a large number of dialogue data samples are obtained by means of simulated dialogue, thereby solving the problem of lack of intelligent agent evaluation samples; the multi-dimensional dialogue ability of the second intelligent agent is evaluated based on at least one large evaluation model, thereby realizing the automation of intelligent agent evaluation; according to the intelligent agent evaluation results, R&D personnel can be guided to continuously optimize the second intelligent agent and improve the dialogue efficiency and dialogue accuracy of the second intelligent agent in practical applications.

[0105] Based on the agent evaluation method provided by the present disclosure, the present disclosure also provides an agent evaluation device. Figure 7 The device is described in detail.

[0106] The figure schematically shows a structural block diagram of an intelligent agent evaluation device according to an embodiment of the present disclosure.

[0107] like Figure 8 As shown, the intelligent agent evaluation device 800 may include: a calling module 810 and an evaluation module 820.

[0108] The calling module 810 is used to call the first agent and the second agent to perform a simulation task to obtain dialogue data. The simulation task is used for multiple first objects to conduct at least one simulated dialogue with the second object generated by the second agent. The multiple first objects are generated by the first agent and have different dialogue styles.

[0109] The evaluation module 820 is used to input the dialogue data into at least one evaluation model of the third agent, obtain the target dialogue features, and output the evaluation results of the second agent based on the target dialogue features.

[0110] According to an embodiment of the present disclosure, a simulation task includes multiple task data, each task data includes first portrait information for configuring a first agent to generate a first object and second portrait information for configuring a second agent to generate a second object; wherein the first portrait information of multiple task data for the same simulation task are different from each other and the second portrait information of multiple task data are the same as each other.

[0111] According to an embodiment of the present disclosure, the calling module 810 includes: a data acquisition submodule, an object generation submodule and a first data generation submodule. The data acquisition submodule is used to acquire multiple task data of the simulation task. The object generation submodule is used to send the multiple task data to the first agent and the second agent, the first agent generates multiple first objects according to the first portrait information, and the second agent generates the second object according to the second portrait information. The first data generation submodule is used to conduct a dialogue between multiple first objects and second objects based on the first agent and the second agent to obtain dialogue data.

[0112] According to an embodiment of the present disclosure, the object generation submodule includes: a first object generation unit, which is used to extract language features in the first portrait information based on the first intelligent agent, and construct a first object with language features in the dialogue behavior; a second object generation unit, which is used to extract business features in the second portrait information based on the second intelligent agent, and construct a second object with business features in the dialogue behavior; wherein the language features are related to the business features.

[0113] According to an embodiment of the present disclosure, the first data generation submodule includes: a first data generation unit, which is used to generate a first speech of a first object based on a first agent, wherein the first speech has language features and includes query information related to business features; a second data generation unit, which is used to generate a second speech of a second object based on a second agent, wherein the second speech is feedback information of the query information. The first data generation submodule is used to conduct multiple rounds of dialogue between the first object and the second object to obtain dialogue data.

[0114] According to an embodiment of the present disclosure, the second data generation unit includes: a knowledge acquisition subunit, used to retrieve knowledge data matching the query information from a business knowledge base; and a speech generation subunit, used to input the knowledge data into the second intelligent agent and generate a second speech in combination with the business characteristics of the second object.

[0115] According to an embodiment of the present disclosure, the evaluation big model includes a topic evaluation big model, and the evaluation module 820 includes: a first evaluation sub-module, used to input the conversation data into the topic evaluation big model, extract the first descriptive feature of the first object and the second descriptive feature of the second object in the conversation data, and output an evaluation result of the correlation between the first descriptive feature and the second descriptive feature, wherein the first descriptive feature and the second descriptive feature are related to the conversation topic.

[0116] According to an embodiment of the present disclosure, the evaluation big model includes a matching evaluation big model, and the evaluation module 820 includes: an answer acquisition submodule, used to obtain a set of standard answers to a set of questions in the dialogue data; and a second evaluation submodule, used to input the dialogue data and the set of standard answers into the matching evaluation big model to obtain an evaluation result of the degree of matching between the question set and the set of standard answers.

[0117] According to an embodiment of the present disclosure, the evaluation big model includes an evaluation evaluation model, and the evaluation module 820 includes: a third evaluation sub-module, used to input the dialogue data into the evaluation evaluation big model, extract the evaluation description of the first object on the simulated dialogue in the dialogue data, and output the evaluation result of the second intelligent agent on the efficiency of understanding the simulated dialogue based on the evaluation description.

[0118] According to an embodiment of the present disclosure, the agent evaluation device 800 also includes an adjustment module for repeatedly adjusting the second agent until the evaluation result meets the preset condition. The adjustment module includes: a parameter adjustment submodule for adjusting the model parameters and business knowledge base of the second agent based on the evaluation result to obtain the adjusted second agent; a second data generation submodule for calling the first agent and the adjusted second agent to perform the simulation task to obtain new dialogue data; and a fourth evaluation submodule for inputting the new dialogue data into the third agent to obtain a new evaluation result of the second agent, wherein the preset condition represents that the experience value provided by the second agent to the first object during the dialogue process is higher than the preset threshold.

[0119] According to an embodiment of the present disclosure, the agent evaluation device 800 also includes an application module, which is used to perform a dialogue task based on the second agent when the evaluation result of the second agent meets the preset conditions. The dialogue task is used to generate a second object based on the second agent, and conduct at least one real-time dialogue with multiple real first objects based on the second object. The application module includes: a dialogue response submodule, which is used to call N second agents to perform multiple dialogue tasks respectively in response to a dialogue request, where N is an integer greater than 1; an agent selection submodule, which is used to select at least one target second agent with an added value higher than a preset threshold from the N second agents in response to at least a part of the multiple dialogue tasks not being completed, and the added value is generated based on the task status of each of the multiple dialogue tasks; a task allocation submodule, which is used to allocate at least a part of the multiple dialogue tasks that have not been completed to at least one target second agent for processing.

[0120] According to an embodiment of the present disclosure, the agent evaluation device 800 also includes an interaction module for outputting dialogue data on an interaction interface; and in response to at least one evaluation large model outputting an evaluation result of the second agent, outputting the evaluation result on the interaction interface.

[0121] Fig. 9 The structural block diagram of an intelligent agent according to an embodiment of the present disclosure is schematically shown.

[0122] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig. 9 As shown, the intelligent agent 900 may include five core modules: an input module 910 , a control module 920 , a storage module 930 , a calculation module 940 and an output module 950 .

[0123] The input module 910 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the intelligent agent 900. The input module 910 is the primary link for the intelligent agent 900 to interact with the outside world. It enables the intelligent agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0124] In an example, the input module 910 may input the evaluation request described above.

[0125] In the example, the control module 920 is the core support for the ability of the agent 900 to handle complex tasks. The control module 920 can execute the agent evaluation method described above.

[0126] In the example, the control module 920 will continuously interact with the storage module 930, the computing module 940 and / or the output module 950 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 920 acts as a single initiator to initiate communication with the storage module 930, the computing module 940 and / or the output module 950, and there is no communication coupling between the storage module 930, the computing module 940 and the output module 950.

[0127] In the example, the performance of the control module 920 may be closely related to the large model based on the agent 900. In order to give full play to the capabilities of the large language model, the internal structure of the control module 920 may be designed to be highly configurable and extensible to cope with various types of tasks and requirements in real scenarios.

[0128] The storage module 930 may be responsible for memorizing information such as historical conversations, event flows, etc. The conversation data and evaluation results as described above may be included in the storage module 930 .

[0129] In the example, after the agent 900 obtains the evaluation request, the agent 900 can use the first agent and the second agent to perform the simulation task and obtain the dialogue data. The dialogue data can be stored in the storage module 930. The agent 900 can obtain the dialogue data from the storage module 930 and feed it back to the control module 920. Then, the control module 920 can input the dialogue data into the third agent to obtain the evaluation result of the second agent, and pass the evaluation result to the output module 950.

[0130] The operation module 940 can be regarded as a predefined tool library. At least one evaluation model as described above can be included in the operation module 940 .

[0131] In the example, when the agent 900 needs to evaluate the dialogue data, it can call the relevant evaluation model from the operation module 940 and feed it back to the control module 920. Then, the control module 920 can use the feedback evaluation model to evaluate the dialogue data, obtain the evaluation result, and pass the evaluation result to the output module 950. It can be understood that although the large language model has excellent language understanding and generation capabilities, it is the same as a person, without the help of any tools, the tasks that can be solved are very limited. When the agent 900 is given the ability to call tools, it can achieve tasks such as completing mathematical operations with the help of a calculator, completing data analysis with the help of Python, and completing weather forecasts with the help of a search engine.

[0132] In an example, the output module 950 may output the conversation data and evaluation results described above.

[0133] According to the embodiment of the present disclosure, the intelligent agent 900 can simply and effectively improve the intelligence level, and improve flexibility and versatility.

[0134] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0135] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0136] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0137] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0138] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0139] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0140] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the agent evaluation method. For example, in some embodiments, the agent evaluation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the agent evaluation method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the agent evaluation method in any other suitable manner (eg, by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An agent evaluation method, comprising: Invoking a first agent and a second agent to perform a simulation task to obtain dialogue data, wherein the simulation task is used for a plurality of first objects to conduct at least one simulation dialogue with a second object generated by the second agent, wherein the plurality of first objects are generated by the first agent and have different dialogue styles; The dialogue data is input into at least one evaluation model of a third agent to obtain target dialogue features, and an evaluation result of the second agent is output based on the target dialogue features.

2. The method according to claim 1, wherein: The simulation task includes a plurality of task data, each of which includes first portrait information for configuring the first agent to generate a first object and second portrait information for configuring the second agent to generate a second object; Among them, the first portrait information of multiple task data for the same simulation task are different from each other and the second portrait information of the multiple task data are the same as each other.

3. The method according to claim 2, wherein: The calling of the first agent and the second agent to perform the simulation task and obtain the dialogue data comprises: Acquire multiple task data of the simulation task; Sending the plurality of task data to the first agent and the second agent, wherein the first agent generates a plurality of first objects according to the first portrait information, and the second agent generates a second object according to the second portrait information; The conversation data is obtained based on the first agent and the second agent conducting a conversation between the plurality of first objects and the second object.

4. The method according to claim 3, wherein: The first agent generates a first object according to the first portrait information, and the second agent generates a second object according to the second portrait information, comprising: Extracting language features in the first portrait information based on the first agent, and constructing a first object with the language features in the dialogue behavior; Extracting business features from the second portrait information based on the second agent, and constructing a second object with the business features in the dialogue behavior; The language feature is related to the service feature.

5. The method according to claim 4, wherein: The obtaining of the dialogue data based on the dialogue between the plurality of first objects and the second object by the first agent and the second agent comprises: generating a first speech of a first object based on the first agent, wherein the first speech has the language feature and includes query information related to the business feature; generating a second speech of a second object based on the second agent, where the second speech is feedback information of the query information; Conduct multiple rounds of dialogues between the first object and the second object to obtain the dialogue data.

6. The method according to claim 5, wherein: The generating of the second speech of the second object based on the second agent comprises: Retrieving knowledge data matching the query information from a business knowledge base; The knowledge data is input into the second agent, and the second speech is generated in combination with the business characteristics of the second object.

7. The method according to claim 1, wherein: The evaluation big model includes a topic evaluation big model, the inputting the dialogue data into at least one evaluation big model of the third agent to extract target dialogue features, and outputting the evaluation result of the second agent based on the target dialogue features includes: The conversation data is input into the topic evaluation model, a first descriptive feature of a first object and a second descriptive feature of a second object in the conversation data are extracted, and an evaluation result of the relevance of the first descriptive feature and the second descriptive feature is output, wherein the first descriptive feature and the second descriptive feature are related to the conversation topic.

8. The method according to claim 1, wherein: The evaluation big model includes a matching evaluation big model, the inputting the dialogue data into at least one evaluation big model of the third agent to extract target dialogue features, and outputting the evaluation result of the second agent based on the target dialogue features includes: Obtaining the standard answer set for the question set in the dialogue data; The dialogue data and the standard answer set are input into the matching evaluation model to obtain an evaluation result of the matching degree between the question set and the standard answer set.

9. The method according to claim 1, wherein: The evaluation big model includes an evaluation evaluation model, the inputting the dialogue data into at least one evaluation big model of the third agent to extract target dialogue features, and outputting the evaluation result of the second agent based on the target dialogue features includes: The dialogue data is input into the evaluation model, the evaluation description of the simulated dialogue by the first object in the dialogue data is extracted, and the evaluation result of the second agent on the efficiency of understanding the simulated dialogue is output based on the evaluation description.

10. The method according to any one of claims 1 to 9, further comprising repeatedly performing the following operations until the evaluation result meets a preset condition: Adjust the model parameters and business knowledge base of the second agent based on the evaluation result to obtain an adjusted second agent; Calling the first agent and the adjusted second agent to perform the simulation task to obtain new dialogue data; as well as The new dialogue data is input into the third agent to obtain a new evaluation result of the second agent, wherein the preset condition represents that the experience value provided by the second agent to the first object during the dialogue is higher than a preset threshold.

11. The method according to claim 10, when the evaluation result of the second agent meets the preset condition, the method further comprises: In response to the dialogue request, calling N of the second agents to respectively perform a plurality of dialogue tasks, wherein the dialogue tasks are used to generate a second object based on the second agent, and to conduct at least one real-time dialogue with a plurality of real first objects based on the second object, where N is an integer greater than 1; In response to at least a portion of the plurality of dialogue tasks not being completed, selecting at least one target second agent having an additional value higher than a preset threshold from the N second agents, wherein the additional value is generated based on the task status of each of the plurality of dialogue tasks; At least a portion of the plurality of dialogue tasks that have not been completed is assigned to the at least one target second agent for processing.

12. The method according to claim 1, further comprising: Outputting the conversation data on an interactive interface; In response to the at least one evaluation model outputting an evaluation result of the second agent, the evaluation result is output on the interactive interface.

13. An intelligent agent evaluation device, comprising: A calling module, configured to call a first agent and a second agent to perform a simulation task to obtain dialogue data, wherein the simulation task is used for a plurality of first objects to conduct at least one simulation dialogue with a second object generated by the second agent, wherein the plurality of first objects are generated by the first agent and have different dialogue styles; An evaluation module is used to input the dialogue data into at least one evaluation model of a third agent to obtain target dialogue features, and output an evaluation result of the second agent based on the target dialogue features.

14. The device according to claim 13, wherein: The simulation task includes a plurality of task data, each of which includes first portrait information for configuring the first agent to generate a first object and second portrait information for configuring the second agent to generate a second object; Among them, the first portrait information of multiple task data for the same simulation task are different from each other and the second portrait information of the multiple task data are the same as each other.

15. The device according to claim 14, wherein: The calling module includes: A data acquisition submodule, used for acquiring a plurality of task data of the simulation task; An object generation submodule, configured to send the plurality of task data to the first agent and the second agent, wherein the first agent generates a plurality of first objects according to the first portrait information, and the second agent generates a second object according to the second portrait information; The first data generating submodule is used for obtaining the dialogue data by conducting a dialogue between the plurality of first objects and the second object based on the first agent and the second agent.

16. The device according to claim 15, wherein: The object generation submodule comprises: A first object generation unit, configured to extract language features in the first portrait information based on the first agent, and construct a first object having the language features in the dialogue behavior; A second object generation unit, configured to extract the business features in the second portrait information based on the second agent, and construct a second object with the business features in the dialogue behavior; The language feature is related to the service feature.

17. The device according to claim 16, wherein: The first data generation submodule includes: A first data generating unit, configured to generate a first speech of a first object based on the first agent, wherein the first speech has the language feature and includes query information related to the business feature; A second data generating unit, configured to generate a second speech of a second object based on the second agent, wherein the second speech is feedback information of the query information; The first data generating submodule is used for conducting multiple rounds of dialogues between the first object and the second object to obtain the dialogue data.

18. The device according to claim 17, wherein: The second data generating unit includes: A knowledge acquisition subunit, used to retrieve knowledge data matching the query information from a business knowledge base; The speech generation subunit is used to input the knowledge data into the second agent and generate the second speech in combination with the business characteristics of the second object.

19. The device according to claim 13, wherein: The evaluation model includes a subject evaluation model, and the evaluation module includes: The first evaluation submodule is used to input the conversation data into the topic evaluation model, extract the first descriptive feature of the first object and the second descriptive feature of the second object in the conversation data, and output an evaluation result of the correlation between the first descriptive feature and the second descriptive feature, wherein the first descriptive feature and the second descriptive feature are related to the conversation topic.

20. The device according to claim 13, wherein: The evaluation model includes a matching evaluation model, and the evaluation modules include: An answer acquisition submodule, used to acquire the standard answer set for the question set in the dialogue data; The second evaluation submodule is used to input the dialogue data and the standard answer set into the matching evaluation model to obtain an evaluation result of the matching degree between the question set and the standard answer set.

21. The device according to claim 13, wherein: The evaluation model includes an evaluation model, and the evaluation module includes: The third evaluation submodule is used to input the dialogue data into the evaluation model, extract the evaluation description of the simulated dialogue by the first object in the dialogue data, and output the evaluation result of the second intelligent agent on the efficiency of understanding the simulated dialogue based on the evaluation description.

22. The apparatus according to claims 13-21, further comprising an adjustment module, configured to repeatedly adjust the second agent until the evaluation result meets a preset condition, the adjustment module comprising: A parameter adjustment submodule, used to adjust the model parameters and business knowledge base of the second agent based on the evaluation result to obtain an adjusted second agent; A second data generation submodule, used for calling the first agent and the adjusted second agent to perform the simulation task to obtain new dialogue data; as well as The fourth evaluation submodule is used to input the new dialogue data into the third agent to obtain a new evaluation result of the second agent, wherein the preset condition represents that the experience value provided by the second agent to the first object during the dialogue is higher than a preset threshold.

23. The device according to claim 13, wherein: The application module is further included, which is used to execute a dialogue task based on the second agent when the evaluation result of the second agent meets the preset condition, wherein the dialogue task is used to generate a second object based on the second agent, and to conduct at least one real-time dialogue with multiple real first objects based on the second object, and the application module includes: A dialogue response submodule, configured to call N of the second agents to respectively perform a plurality of dialogue tasks in response to a dialogue request, where N is an integer greater than 1; an agent selection submodule, configured to select, in response to at least a portion of the plurality of dialogue tasks not being completed, at least one target second agent having an additional value higher than a preset threshold from the N second agents, wherein the additional value is generated based on the task status of each of the plurality of dialogue tasks; The task allocation submodule is used to allocate at least a portion of the multiple dialogue tasks that have not been completed to the at least one target second intelligent agent for processing.

24. The device according to claim 13 further includes an interactive module for outputting the conversation data on an interactive interface; and in response to the at least one evaluation model outputting an evaluation result of the second agent, outputting the evaluation result on the interactive interface.

25. An agent configured to perform the method according to any one of claims 1-12.

26. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

27. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

28. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images