Intelligent agent evaluation method and device, equipment, storage medium and program product

Abstract

The invention provides an agent evaluation method and device, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of computers, in particular to the technical field of artificial intelligence such as data processing, deep learning and large models. The method comprises the following steps: acquiring description information of a to-be-evaluated target agent; determining a target industry to which the target agent belongs according to the description information; and generating multiple rounds of evaluation questions based on the description information and the target industry, and evaluating the target agent based on an answer result given by the target agent to the multiple rounds of evaluation questions. According to the method, accurate distribution of the agents is realized by evaluating the agents, so that the conversion rate of the agents is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as data processing, deep learning, and large models, and especially to an intelligent agent evaluation method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] As the number of agents created by developers continues to increase, accurately distributing agents to users becomes increasingly important. To ensure accurate distribution of agents to users and guarantee the quality of distributed agents, it is necessary to evaluate the quality and industry classification of agents and then provide agent distribution information. Summary of the Invention

[0003] The embodiments of the present disclosure propose an intelligent agent evaluation method, device, electronic device, computer-readable storage medium and computer program product, which achieve accurate distribution of intelligent agents by evaluating intelligent agents, thereby improving the conversion rate of intelligent agents.

[0004] In the first aspect, an embodiment of the present disclosure proposes an intelligent agent evaluation method, including: obtaining descriptive information of a target intelligent agent to be evaluated; determining the target industry to which the target intelligent agent belongs based on the descriptive information; generating multiple rounds of evaluation questions based on the descriptive information and the target industry, and evaluating the target intelligent agent based on the answers given by the target intelligent agent to the multiple rounds of evaluation questions.

[0005] In the second aspect, an embodiment of the present disclosure proposes an intelligent agent evaluation device, including: an information acquisition module, configured to obtain descriptive information of a target intelligent agent to be evaluated; an industry determination module, configured to determine the target industry to which the target intelligent agent belongs based on the descriptive information; an evaluation module, configured to generate multiple rounds of evaluation questions based on the descriptive information and the target industry, and evaluate the target intelligent agent based on the answers given by the target intelligent agent to the multiple rounds of evaluation questions.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the intelligent agent evaluation method described in any implementation method in the first aspect when executing the instructions.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the intelligent agent evaluation method described in any implementation method of the first aspect when executed.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the agent evaluation method described in the first aspect.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 is an exemplary system architecture in which the present disclosure may be applied; Figure 2 A flowchart of an agent evaluation method provided in an embodiment of the present disclosure; Figure 3 A flowchart of another agent evaluation method provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of the structure flow of an intelligent agent evaluation method in an application scenario provided by an embodiment of the present disclosure; Figure 5 The embodiment of the present disclosure provides Figure 4 Flowchart of agent evaluation in the illustrated scenario; Figure 6 A structural block diagram of an intelligent agent evaluation device provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of the structure of an electronic device suitable for executing an intelligent agent evaluation method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other unless there is a conflict.

[0012] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0013] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the agent evaluation method, apparatus, electronic device, and computer-readable storage medium disclosed herein can be applied.

[0014] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0015] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed, such as web browser applications, search applications, and instant messaging applications.

[0016] Terminal devices 101, 102, 103 and server 105 can be either hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here.

[0017] The server 105 can provide various services through various built-in applications. Taking the search application that can provide evaluation and distribution of intelligent entities as an example, the server 105 can achieve the following effects when running the search application: first, obtain the description information of the target intelligent entity to be evaluated; then determine the target industry to which the target intelligent entity belongs based on the description information; finally, generate multiple rounds of evaluation questions based on the description information and the target industry, and evaluate the target intelligent entity based on the answers given by the target intelligent entity to the multiple rounds of evaluation questions.

[0018] It should be noted that, in addition to being obtained from terminal devices 101, 102, and 103 via network 104, the target agent's description information can also be pre-stored locally on server 105 in various ways. Therefore, when server 105 detects that this data is already stored locally (for example, when it begins processing a previously saved agent evaluation task), it can choose to directly obtain this data locally. In this case, exemplary system architecture 100 may also not include terminal devices 101, 102, 103 and network 104.

[0019] Because evaluating a target agent based on its descriptive information requires significant computational resources and computing power, the agent evaluation methods provided in the subsequent embodiments of this disclosure are generally performed by a server 105 with significant computational power and resources. Accordingly, the agent evaluation apparatus is generally located within the server 105. However, it should also be noted that, if terminal devices 101, 102, and 103 also possess sufficient computational power and resources, the terminal devices 101, 102, and 103 can also utilize search applications installed thereon to perform the aforementioned computations delegated to the server 105, thereby outputting the same results as the server 105. In particular, in the presence of multiple terminal devices with varying computational capabilities, if the search application determines that the terminal device it is assigned possesses significant computational power and resources, the terminal device can be made to perform the aforementioned computations, thereby appropriately alleviating the computational burden on the server 105. Accordingly, the agent evaluation apparatus can also be located within the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0020] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0021] Please refer to Figure 2 , Figure 2 This is a flow chart of an agent evaluation method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps: Step 201: Obtain description information of the target agent to be evaluated.

[0022] In this implementation, the execution entity (e.g. Figure 1The server 105 shown in the figure can obtain the descriptive information of the target intelligent agent to be evaluated through various methods such as user interaction, web search, large models, etc., wherein the descriptive information is used to describe the overall information of the intelligent agent, which may include the name, introduction, task, goal, perception, input, knowledge base, background information, decision-making and reasoning ability, learning and adaptability, action and output, feedback mechanism, etc. of the intelligent agent.

[0023] Step 202: Determine the target industry to which the target agent belongs based on the description information.

[0024] In this embodiment, the execution entity receives the description information obtained in step 201 and determines the target industry to which the target agent belongs based on the description information. The target industry refers to different business scenarios corresponding to the agent, such as finance, education, customer service, etc.

[0025] In this embodiment, the execution entity can obtain keywords from the description information of the target intelligent entity, match the keywords with keywords corresponding to the target industry, and determine the target industry corresponding to the successfully matched keywords as the target industry of the target intelligent entity.

[0026] Step 203: Generate multiple rounds of evaluation questions based on the description information and the target industry, and evaluate the target agent based on the answers given by the target agent to the multiple rounds of evaluation questions.

[0027] In this embodiment, the execution entity generates multiple rounds of evaluation questions based on the descriptive information and the target industry, and evaluates the target agent based on the target agent's responses to these multiple rounds. Specifically, based on the descriptive information, the execution entity can select multiple rounds of evaluation questions from a pre-set question library corresponding to the target industry, invoke the target agent to answer these multiple rounds of evaluation questions, and then evaluate the target agent based on the responses. The number of rounds in the multiple rounds of evaluation questions can be a pre-set fixed number.

[0028] In this embodiment, after the execution subject calls the target agent to answer multiple rounds of evaluation questions, the execution subject can use human participation to score the answer results and evaluate the target agent based on the scoring results.

[0029] The intelligent agent evaluation method provided by the embodiment of the present disclosure determines the target industry to which the target intelligent agent belongs based on the obtained descriptive information of the target intelligent agent, generates multiple rounds of evaluation questions based on the descriptive information and the target industry, and evaluates the target intelligent agent based on the answers given by the target intelligent agent to the multiple rounds of evaluation questions. Through the above method, the intelligent agents are classified by industry and subjected to multiple rounds of assessment, effectively identifying and filtering low-quality, untrustworthy or unavailable intelligent agents, retaining high-quality, trustworthy and available intelligent agents, and ensuring the overall quality of the intelligent agents.

[0030] Please refer to Figure 3 , Figure 3 This is a flowchart of another agent evaluation method provided by an embodiment of the present disclosure, wherein process 300 includes the following steps: Step 301: Obtain agent metadata of the target agent to be evaluated.

[0031] In this embodiment, the executing entity obtains the agent metadata of the target agent to be evaluated provided by the developer through various methods such as text input, voice input, and image input. The agent metadata is used to describe the basic information of the agent's attributes, behaviors, and environment, such as the agent's name, introduction, type, task, behavior, perception, decision, etc.

[0032] Step 302: Convert the intelligent body metadata into unified structure data.

[0033] In this embodiment, after receiving the intelligent body metadata provided by the developer, the execution entity converts the intelligent body metadata into unified structure data, where the unified structure data is fixed-structure data including intelligent body metadata such as intelligent body profile, personality, dialogue capability plug-in, knowledge base, workflow, RAG (Relational Attention Graph), contextual memory, multimedia image, etc. When the intelligent body metadata provided by the developer is missing, the execution entity can send a notification to the developer to request the developer to complete the intelligent body metadata, or it can use Internet data capture to automatically complete the intelligent body metadata.

[0034] In this embodiment, after generating the unified structure data, the execution subject preliminarily stores the unified structure data.

[0035] Step 303: Detect the unified structure data according to a preset evaluation protocol to generate a target detection result.

[0036] In this embodiment, after generating the unified structure data, the execution subject detects the unified structure data according to a preset evaluation protocol to generate a target detection result. Specifically, the execution subject can call a preset data detection model to perform multiple rounds of iterative detection on the unified structure data to generate multiple detection results. Each round of the multiple rounds of iterative detection performs the following detection operation: first, the unified structure data is detected according to the preset evaluation protocol to generate a detection result. When the detection result does not meet the preset conditions, the iterative detection is continued. When the detection result meets the preset conditions, the iterative detection is stopped; then, the multiple detection results are merged to generate a target detection result, wherein the preset condition is a judgment of whether the detection result given by the data detection model is qualified; when the detection result is judged to be qualified, the iterative detection is stopped, and the multiple detection results obtained in the iterative detection process are merged to generate a target detection result. When the detection result is judged to be unqualified, the iterative detection is continued.

[0037] In this embodiment, the profile detection model's prompts are composed of a preset agent target user profile and target user behavior. The agent target user profile refers to the basic characteristics and needs of the user group using the agent, and the target user behavior refers to the target user's behavioral characteristics, such as the frequency, method, and decision-making patterns of interaction with the target agent. During multiple rounds of iterative testing, the profile detection model tests the agent's name, introduction, prompts, opening remarks, tool-calling capabilities, and follow-up questioning capabilities. The results of the previous round of testing serve as reference information for the next round of testing, continuously refining the user profile.

[0038] Step 304: Generate description information based on the target detection result.

[0039] In this embodiment, the execution entity generates description information of the target agent based on the target detection results. Specifically, the execution entity obtains the agent metadata under the qualified detection results from the target detection results, combines the agent metadata under the qualified detection results with the user portrait, and generates description information.

[0040] Step 305: Input the description information into the preset industry classification model.

[0041] In this embodiment, the execution entity uses the description information to pre-set the industry classification model. The industry classification model is a large model that takes the agent's description information as input and outputs the agent's industry. Its prompt words are generated by the mechanism that builds the "agent understanding expert" role. This mechanism comprehensively judges the agent's role setting, industry field, skills, and creation tools, and is used to classify the agent according to enterprise, industry, field, and application scenario.

[0042] Step 306: Determine the target industry to which the target agent belongs based on the output information of the industry classification model.

[0043] In this embodiment, the execution entity determines the target industry to which the target agent belongs based on the output information of the industry classification model. Specifically, the industry classification model determines the target agent's multi-dimensional industry characteristics and attributes based on the descriptive information, generates output information based on the judgment results, and determines the target industry to which the target agent belongs based on the output information. Multi-dimensional industry characteristics refer to a set of characteristics that analyze and describe an industry from multiple perspectives and dimensions, including market characteristics (such as market size, market growth rate, market structure, etc.), technological characteristics (such as technological innovation and technological maturity, etc.), product characteristics (such as product diversity and product life cycle, etc.), competitive characteristics (such as competitive landscape and market concentration, etc.), economic characteristics (such as profitability and cost structure, etc.), regulatory characteristics (such as industry regulation and compliance requirements, etc.), and social and environmental characteristics (such as social and environmental impact). The attribute judgment of the target agent can be divided into perception, behavior, goals, decision-making, autonomy, adaptability, learning ability, cooperation, reasoning ability, and situational awareness. The output information includes the target agent's corresponding industry label and pan-industry category.

[0044] Step 307: Input the description information and target industry into a preset evaluation model to obtain output multiple rounds of evaluation questions.

[0045] In this embodiment, the execution entity inputs the target agent's description and target industry into a preset evaluation model, which then outputs multiple rounds of evaluation questions for the target agent. The evaluation model's prompts are generated by a mechanism that establishes an "agent evaluation expert" role, which evaluates the agent's quality and performance. The number of rounds of multi-round evaluation questions can be pre-set. These questions are generated based on the target agent's description and target industry, using user-role models and simulated user behavior, to evaluate the target agent's quality and performance.

[0046] Step 308: Call the target agent to answer the multi-round evaluation questions and generate multi-round answer results corresponding to the multi-round evaluation questions.

[0047] In this embodiment, the execution subject calls the target agent dialogue interface, answers multiple rounds of evaluation questions, and generates multiple rounds of answer results corresponding to the multiple rounds of evaluation questions.

[0048] Step 309: Input the results of multiple rounds of answers into the evaluation model, and evaluate the target agent based on the output information of the evaluation model.

[0049] In this embodiment, the execution entity inputs the multiple rounds of answer results generated in step 308 into the evaluation model. The evaluation model scores the multiple rounds of answer results corresponding to the multiple rounds of evaluation questions and generates evaluation information for the target agent based on the scoring results. The evaluation information for the target agent includes the evaluation result and performance indicators of the target agent. The evaluation result refers to the overall pass / fail judgment of the target agent. The performance indicators include the agent's accuracy, response speed, stability, resource usage, and packet loss.

[0050] In this embodiment, the execution subject determines the evaluation result of the target intelligent agent by judging whether the target intelligent agent is qualified in terms of perception, behavior, goals, decision-making, autonomy, adaptability, learning ability, cooperation, reasoning ability, and situational awareness. Specifically, the execution subject may determine the evaluation result of the target intelligent agent as qualified if the target intelligent agent is qualified in terms of perception, behavior, goals, decision-making, autonomy, adaptability, learning ability, cooperation, reasoning ability, and situational awareness, and otherwise determine the evaluation result of the target intelligent agent as unqualified. The execution subject may also pre-set that the evaluation result of the target intelligent agent is qualified if only certain aspects of the target intelligent agent's perception, behavior, goals, decision-making, autonomy, adaptability, learning ability, cooperation, reasoning ability, and situational awareness are qualified, and otherwise determine the evaluation result of the target intelligent agent as unqualified.

[0051] In this embodiment, the evaluation model scores the results of multiple rounds of answers through a constructed scoring standard, wherein the scoring standard is based on the general quality standard of the intelligent body and is constructed by accumulating the quality standards of four dimensions corresponding to role setting, industry field, skill ability and creation tools.

[0052] Step 310: Determine target distribution information of the target agent based on the evaluation result of the target agent.

[0053] In this embodiment, after the execution subject obtains the evaluation information generated in step 309, it determines the target distribution information of the target intelligent agent. Specifically, when the evaluation result in the evaluation information of the target intelligent agent is qualified, a target recall word is generated based on the evaluation result, the target distribution information of the intelligent agent is composed according to the target recall word, and the target distribution information is entered into the distribution recall channel. Among them, when the evaluation result in the evaluation information of the target intelligent agent is qualified, it means that the quality of the target intelligent agent is high and can be distributed to users. The target recall word refers to a keyword for search or matching generated based on the evaluation result, which is used for search or recommendation in the intelligent agent distribution scenario, and can help the target user search or match the corresponding intelligent agent.

[0054] In this embodiment, different from Figure 2In the embodiment shown, this embodiment specifically provides an implementation method for determining the description information of the target intelligent agent through steps 301-304, specifically provides an implementation method for obtaining the target industry of the target intelligent agent through steps 305-306, specifically provides an implementation method for evaluating the target intelligent agent through steps 307-309, and specifically provides an implementation method for determining the distribution information of the target intelligent agent through step 310. There is no causal or dependent relationship between the four implementation methods, and they do not necessarily need to be applied simultaneously in one embodiment. They can be applied separately in different embodiments as needed to determine the description information of the target intelligent agent, or to obtain the target industry of the target intelligent agent, or to evaluate the target intelligent agent, or to determine the distribution information of the target intelligent agent. This embodiment exists only as a preferred embodiment that simultaneously includes four specific implementation methods.

[0055] After evaluating the target agent and obtaining an evaluation result, the target agent can be determined to be distributed based on the evaluation result. Specifically, if the target agent's evaluation result is unqualified, the target agent's target distribution information is set to not be distributed. Then, based on the target distribution information set to not be distributed, a corresponding rectification notice is issued to the target agent, and the target agent is rectified to improve its quality, thereby avoiding the distribution of low-quality agents to users.

[0056] In this embodiment, the execution entity can also evaluate the target agent in the following manner: First, the execution entity inputs the description information and target industry into a preset question-setter model, which outputs a question book containing multiple questions. The target agent is then called upon to answer each question in the question book, outputting multiple rounds of answers. Each question in the question book is updated by the question-setter model based on the target agent's previous answer. Finally, the multiple rounds of answers and the question book are input into a preset interviewer model, and evaluation information for the target agent is generated based on the interviewer model's scoring of the multiple rounds of answers. The question-setter model, based on the description information and target industry, plays the user and simulates user behavior to generate a set of question books. Each question set in the question book has the first round of conversation content, a preset number of conversation rounds, and a conversation scope. After receiving the question book and the multiple rounds of answers, the interviewer model assigns a pass or fail score to the comprehensive question and answer content of each question set in the question book, and outputs the assessment items and thought process. After scoring all question sets, the interviewer model comprehensively scores and presents an overall quality result for the agent, determining whether the agent is qualified or unqualified. The interviewer model scores the comprehensive Q&A content using a constructed scoring criterion. This scoring criterion is based on the general quality standards for agents and is constructed by accumulating quality standards across four dimensions: role setting, industry, skills, and authoring tools. This implementation method allows for multiple rounds of assessment of the target agent, effectively identifying and filtering out low-quality, untrustworthy, or unavailable agents, ensuring the quality of the distributed agents.

[0057] The intelligent agent evaluation method provided in this embodiment effectively identifies and filters low-quality intelligent agents by classifying intelligent agents by industry and conducting multiple rounds of assessments, retains high-quality intelligent agents, and distributes high-quality intelligent agents, so that high-quality intelligent agents can accurately reach target users in various traffic scenarios, significantly improve conversion rates and user experience, and help developers maximize profits.

[0058] To deepen understanding, this disclosure combines a specific application scenario and provides a specific implementation solution, please refer to Figure 4 Flow 400 is shown.

[0059] In this implementation, the developer builds an agent on the agent platform and sends the agent metadata of the target agent to the agent platform; after the agent platform collects the agent metadata, it sends the agent metadata to the collection Lambda operator; after the collection Lambda operator receives the agent metadata from different source platforms, it converts the agent metadata into unified structure data. After the conversion is completed, the system preliminarily stores the unified structure data of the target agent and sends the unified structure data to the data mart Lambda operator; after the data mart Lambda operator receives the unified structure data, it puts it into the scheduling queue, and the scheduling queue sends the unified structure data to the understanding Lambda operator; the understanding Lambda operator calls the data detection model after SFT distillation for multiple rounds to detect the target agent's name, introduction, prompt words, opening remarks, tool calling ability, and questioning ability, and generates the target detection result of the target agent. The data detection model generates the description information of the target agent based on the target detection result; the description information of the target agent is sent to the evaluation Lambda operator, and the evaluation Lambda operator calls the data detection model after SFT distillation. The distilled industry classification model makes multi-dimensional industry feature and attribute judgments on the target agent, and outputs the target industry of the target agent; the description information and target industry of the target agent are input into the question-setter model called by the evaluation Lambda operator to generate a question book, and then the evaluation Lambda operator calls the target agent dialogue interface to conduct an initial round of question-answering test on the question book, and then the question-setter model generates questions for the next round based on the initial round of questions and answers and the dialogue paradigm and content of the question set, until the preset round limit is reached; the evaluation Lambda operator calls the interviewer model to score the comprehensive question-answer content and performance indicator data of multiple rounds of questions and answers for each question set in the question book as qualified or unqualified, and outputs the examination items and thinking process; the interviewer model generates agent evaluation information based on the score, determines whether the agent is qualified or unqualified, and generates target distribution information of the target agent based on the question book, and returns it to the understanding Lambda operator; the data mart Lambda operator receives the information from the understanding evaluation Lambda After receiving the target distribution information, the operator determines whether to distribute the target agent based on the agent distribution criteria. If the target agent is determined to be distributed, the target distribution information is sent to the Distribution Lambda operator. The Distribution Lambda operator then builds a database of the target distribution information and enters it into distribution and recall channels such as search and information flow. The Lambda operator, also known as a Lambda expression, is a way to represent anonymous functions (i.e., functions without names). Its core concept is to pass computational processes or functions as values, rather than binding functions to names.

[0060] In this implementation, the scheduling queue is a priority queue built based on the Lambda framework instance operator scheduling capability. It regulates, understands, and evaluates the resource pool composed of operator instances, and has functions such as priority management, elastic scaling, traffic peak shaving, and error retry.

[0061] In this implementation, a complete closed loop of operations such as construction, testing, industry classification, quality inspection, and distribution of high-quality intelligent agents built through multiple channels is carried out to effectively identify and filter low-quality intelligent agents, retain high-quality intelligent agents, and distribute high-quality intelligent agents, so that high-quality intelligent agents can accurately reach target users in various traffic scenarios, significantly improve conversion rates and user experience, and help developers maximize their profits.

[0062] This disclosure also combines a specific application scenario and provides another specific implementation solution, see Figure 5 Flow 500 is shown.

[0063] In this implementation, the intelligent agent platform sends the unified structure data of the target intelligent agent to be evaluated to the data detection model. The data detection model detects the unified structure data according to the preset evaluation protocol to generate a detection result, generates descriptive information based on the detection result, and then sends the descriptive information to the industry classification model to determine the target industry to which the target intelligent agent belongs. Next, the target industry and descriptive information are sent to the question setter model. The question setter model outputs a set of questions, calls the intelligent agent to answer the question set, obtains the answer result, and inputs the answer result into the interviewer model. The interviewer model scores the question and answer process and generates target distribution information including evaluation results and performance evaluation results.

[0064] In this implementation, by calling the data detection model, industry classification model, question setter model and interviewer model, the intelligent agents are classified by industry and subjected to multiple rounds of assessment, effectively identifying and filtering low-quality intelligent agents, retaining high-quality intelligent agents, and distributing high-quality intelligent agents, so that high-quality intelligent agents can accurately reach target users in various traffic scenarios, significantly improving conversion rates and user experience, and helping developers maximize their profits.

[0065] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an intelligent agent evaluation device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0066] like Figure 6As shown, the agent evaluation device 600 of this embodiment may include: an information acquisition module 601, an industry determination module 602, and an evaluation module 603. The information acquisition module 601 is configured to acquire descriptive information of a target agent to be evaluated; the industry determination module 602 is configured to determine the target industry to which the target agent belongs based on the descriptive information; and the evaluation module 603 is configured to generate multiple rounds of evaluation questions based on the descriptive information and the target industry, and evaluate the target agent based on the target agent's answers to the multiple rounds of evaluation questions.

[0067] The specific processing of the volume information acquisition module 601, the industry determination module 602, and the evaluation module 603 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiment are not repeated here.

[0068] In some optional implementations of this embodiment, the information acquisition module 601 includes: a data acquisition unit configured to acquire agent metadata of a target agent to be evaluated; a data conversion unit configured to convert the agent metadata into unified structure data; a data detection unit configured to detect the unified structure data according to a preset evaluation protocol and generate a target detection result; and an information generation unit configured to generate descriptive information based on the target detection result. The data detection unit in the information acquisition module 601 includes: a multi-round detection unit configured to invoke a preset data detection model to perform multiple rounds of iterative detection on the unified structure data and generate multiple detection results. Each round of the multiple iterative detection performs the following detection operations: detecting the unified structure data according to the preset evaluation protocol and generating a detection result; in response to the detection result not meeting the preset condition, continuing the iterative detection; in response to the detection result meeting the preset condition, stopping the iterative detection; and a result merging unit configured to merge the multiple detection results to generate a target detection result. During the multiple rounds of iterative detection, the detection results of the previous round serve as reference information for the next round of detection.

[0069] In some optional implementations of this embodiment, the industry determination module 602 includes: a first information input unit, configured to input the descriptive information into a preset industry classification model; an information judgment unit, configured to determine the target industry to which the target intelligent entity belongs based on the output information of the industry classification model; wherein the industry classification model performs multi-dimensional industry characteristics and attribute judgment on the target intelligent entity based on the descriptive information, and generates output information based on the judgment result.

[0070] In some optional implementations of this embodiment, the evaluation module 603 includes: a second information input unit, configured to input the descriptive information and the target industry into a preset evaluation model to obtain output multiple rounds of evaluation questions; a first answer unit, configured to call the target agent to answer the multiple rounds of evaluation questions, and generate multiple rounds of answer results corresponding to the multiple rounds of evaluation questions; a first evaluation unit, configured to input the multiple rounds of answer results into the evaluation model, and evaluate the target agent based on the output information of the evaluation model; wherein the evaluation model scores the multiple rounds of answer results corresponding to the multiple rounds of evaluation questions, and generates evaluation information of the target agent based on the scoring results. The evaluation module 603 also includes: a third information input unit, which is configured to input the descriptive information and the target industry into the preset question-setting officer model, and obtain an output question book containing multiple questions; a second answer unit, which is configured to call the target intelligent agent to answer each question in the question book, and obtain output multiple rounds of answer results; wherein, each question in the question book will be updated by the question-setting officer model according to the target intelligent agent's answer result to the previous question; a second evaluation unit, which is configured to input multiple rounds of answer results and the question book into the preset interviewer model, and generate evaluation information for the target intelligent agent based on the scoring results of the multiple rounds of answer results output by the interviewer model.

[0071] In some optional implementations of this embodiment, the agent evaluation device 600 further includes: an agent distribution module 604, which is configured to determine target distribution information of the target agent based on the evaluation result obtained by evaluating the target agent.

[0072] In some optional implementations of this embodiment, the agent distribution module 604 includes: a recall word generation unit, configured to generate a target recall word based on the evaluation result in response to the target agent being qualified; a distribution information processing unit, configured to compose the target distribution information of the agent according to the target recall word, and enter the target distribution information into the distribution recall channel. The evaluation result includes a performance evaluation result, and the performance evaluation result includes at least one of the response speed, throughput, resource occupancy, and packet loss. The agent distribution module 604 also includes: a distribution information setting unit, configured to set the target distribution information of the target agent to not be distributed in response to the target agent being unqualified; a notification issuing unit, configured to issue a corresponding rectification notice to the target agent according to the target distribution information set to not be distributed.

[0073] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The intelligent agent evaluation device provided by this embodiment effectively identifies and filters low-quality intelligent agents by classifying intelligent agents by industry and conducting multiple rounds of assessments, retains high-quality intelligent agents, and distributes high-quality intelligent agents, so that high-quality intelligent agents can accurately reach target users in various traffic scenarios, significantly improve conversion rates and user experience, and help developers maximize profits.

[0074] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the intelligent agent evaluation method described in any of the above embodiments when executing.

[0075] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the intelligent agent evaluation method described in any of the above embodiments when executed.

[0076] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which, when executed by a processor, can implement the intelligent agent evaluation method described in any of the above embodiments.

[0077] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0078] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. Computing unit 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0079] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0080] The computing unit 701 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the agent evaluation method. For example, in some embodiments, the agent evaluation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the agent evaluation method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the agent evaluation method in any other suitable manner (e.g., via firmware).

[0081] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0083] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0084] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0085] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0086] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host. This is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and virtual private server (VPS) services.

[0087] According to the technical solution of the embodiment of the present disclosure, by classifying intelligent agents by industry and conducting multiple rounds of assessments, low-quality intelligent agents can be effectively identified and filtered, high-quality intelligent agents can be retained, and high-quality intelligent agents can be distributed, so that high-quality intelligent agents can accurately reach target users in various traffic scenarios, significantly improve conversion rates and user experience, and help developers maximize their profits.

[0088] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0089] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for evaluating an intelligent agent, comprising: Get the description information of the target agent to be evaluated; Determine the target industry to which the target agent belongs according to the description information; A plurality of rounds of evaluation questions are generated based on the description information and the target industry, and the target agent is evaluated based on the answers given by the target agent to the plurality of rounds of evaluation questions.

2. The method according to claim 1, wherein The step of obtaining the description information of the target agent to be evaluated includes: Obtaining agent metadata of the target agent to be evaluated; Converting the intelligent body metadata into unified structure data; Detecting the unified structure data according to the preset evaluation protocol to generate a target detection result; Generate description information based on the target detection result.

3. The method according to claim 2, wherein: The detecting the unified structure data according to the preset evaluation protocol to generate the detection result includes: Calling a preset data detection model to perform multiple rounds of iterative detection on the unified structure data to generate multiple detection results; wherein each round of the multiple rounds of iterative detection performs the following detection operations: detecting the unified structure data according to the preset evaluation protocol to generate a detection result; in response to the detection result not meeting the preset condition, continuing the iterative detection; in response to the detection result meeting the preset condition, stopping the iterative detection; The multiple detection results are combined to generate a target detection result.

4. The method according to claim 3, wherein: During multiple rounds of iterative testing, the results of the previous round of testing will be used as reference information for the next round of testing.

5. The method according to claim 1, wherein The step of determining the target industry to which the target agent belongs according to the description information includes: Inputting the descriptive information into a preset industry classification model; The target industry to which the target intelligent agent belongs is determined based on the output information of the industry classification model; wherein the industry classification model performs multi-dimensional industry characteristics and attribute judgment on the target intelligent agent according to the descriptive information, and generates the output information based on the judgment result.

6. The method according to claim 1, wherein Generating multiple rounds of evaluation questions based on the description information and the target industry, and evaluating the target agent based on answers given by the target agent to the multiple rounds of evaluation questions, includes: Input the description information and the target industry into a preset evaluation model to obtain multiple rounds of evaluation questions; Invoking the target agent to answer the multiple rounds of evaluation questions and generating multiple rounds of answer results corresponding to the multiple rounds of evaluation questions; The multi-round answer results are input into the evaluation model, and the target intelligent agent is evaluated based on the output information of the evaluation model; wherein, the evaluation model scores the multi-round answer results corresponding to the multi-round evaluation questions, and generates evaluation information of the target intelligent agent based on the scoring results.

7. The method according to claim 1, wherein Generating multiple rounds of evaluation questions based on the description information and the target industry, and evaluating the target agent based on answers given by the target agent to the multiple rounds of evaluation questions, includes: Inputting the description information and the target industry into a preset question setter model to output a question booklet containing multiple questions; Invoking the target agent to answer each question in the question booklet, and obtaining multiple rounds of answer results; wherein each question in the question booklet is updated by the question setter model according to the target agent's answer to the previous question; The multiple rounds of answer results and the question book are input into a preset interviewer model, and evaluation information of the target intelligent agent is generated based on the scoring results of the multiple rounds of answer results output by the interviewer model.

8. The method according to any one of claims 1 to 7, further comprising: The target distribution information of the target agent is determined according to an evaluation result obtained by evaluating the target agent.

9. The method according to claim 8, wherein The determining the target distribution information of the target agent according to the evaluation result of the target agent includes: In response to an evaluation result of the target agent being qualified, generating a target recall word based on the evaluation result; The target distribution information of the intelligent agent is composed according to the target recall words, and the target distribution information is entered into the distribution recall channel.

10. The method according to claim 9, further comprising: In response to an evaluation result of the target agent being unqualified, setting the target distribution information of the target agent to not be distributed; A corresponding rectification notice is issued to the target agent according to the target distribution information that is set to not be distributed.

11. The method according to claim 9, wherein The evaluation result includes a performance evaluation result, and the performance evaluation result includes at least one of response speed, throughput, resource occupancy, and packet loss.

12. An intelligent agent evaluation device, comprising: an information acquisition module, configured to obtain description information of a target agent to be evaluated; An industry determination module is configured to determine a target industry to which the target agent belongs based on the description information; The evaluation module is configured to generate multiple rounds of evaluation questions based on the description information and the target industry, and evaluate the target agent based on the answers given by the target agent to the multiple rounds of evaluation questions.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the agent evaluation method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the agent evaluation method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, which implements the steps of the agent evaluation method according to any one of claims 1 to 11 when executed by a processor.

Images