Intelligent Allocation Method and System for Subjective Problem Evaluation of Artificial Intelligence Agents

Abstract

This application relates to an intelligent allocation method and system for evaluating subjective questions of an artificial intelligence agent. Among them, the method includes: obtaining the data to be evaluated uploaded by the developer; the data to be evaluated is the question-and-answer pair data and reference documents of the artificial intelligence agent; determining the matching object based on the domain label of the data to be evaluated; and allocating the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object. Through this application, after determining the object corresponding to the domain label of the data to be evaluated, the data to be evaluated can be allocated to the target object according to the developer points and the object points, and the target object evaluates the data to be evaluated, realizing an intelligent allocation mechanism for the data to be evaluated and solving the problem of being unable to effectively and efficiently allocate evaluation tasks.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly to an intelligent allocation method and system for evaluating subjective questions of artificial intelligence agents. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the breakthroughs in the field of large language models, AI Agents (Artificial Intelligence Agents) rely on their powerful natural language processing capabilities and have excellent performances in information retrieval, professional Q&A, content generation, etc. However, the processing of subjective questions by AI Agents involves complex semantic understanding and professional knowledge in multiple fields, which is full of uncertainties and complexities. Accordingly, the evaluation of subjective questions of AI Agents also faces many challenges.

[0003] Traditional evaluation methods usually rely on objective indicators such as accuracy rate and response time, but these indicators are difficult to comprehensively measure the performance of AI Agents on subjective questions. Therefore, the evaluation tasks of subjective questions can be assigned to intelligent analysis tools or domain experts for evaluation. Especially in complex scenarios involving multiple fields and multi-cultural backgrounds, more accurate evaluation conclusions can be provided. Currently, although there are some frameworks and tools for AI Agent evaluation, they generally lack an intelligent allocation mechanism for evaluation tasks and cannot effectively and efficiently allocate evaluation tasks.

[0004] Regarding the problem in the related technology that evaluation tasks cannot be effectively and efficiently allocated, no effective solution has been proposed yet. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide an intelligent allocation method and system for evaluating subjective questions of artificial intelligence agents that can improve the quality and efficiency of AI Agents in evaluating subjective questions.

[0006] In a first aspect, in the present embodiment, an intelligent allocation method for evaluating subjective questions of artificial intelligence agents is provided, including:

[0007] Obtain the to-be-evaluated data uploaded by the developer; the to-be-evaluated data is the Q&A pair data and reference documents of the artificial intelligence agent;

[0008] Based on the domain tags of the to-be-evaluated data, determine the matching objects;

[0009] According to the developer points of the developer and the object points of the object, allocate the to-be-evaluated data to the corresponding target object.

[0010] In some of these embodiments, allocating the data to be evaluated to corresponding target objects according to the developer points of the developer and the object points of the object includes:

[0011] Determining the allocation weight of the developer among all developers according to the developer points;

[0012] Allocating the data to be evaluated according to the allocation weight and the object points, in combination with the current workload of the object.

[0013] In some of these embodiments, allocating the data to be evaluated according to the allocation weight and the object points, in combination with the current workload of the object, includes:

[0014] Determining the current remaining quota of the object according to the daily evaluation limit of the object and the current workload;

[0015] Filtering to obtain several objects with non-zero current remaining quotas as the target objects according to the preset quantity and the descending order of the object points;

[0016] Allocating the data to be evaluated to several target objects in turn according to the allocation weight.

[0017] In some of these embodiments, obtaining the data to be evaluated uploaded by the developer further includes:

[0018] Performing format standardization processing on the data to be evaluated; and determining the domain label of the data to be evaluated.

[0019] In some of these embodiments, it further includes:

[0020] Obtaining the multi-dimensional evaluation criteria corresponding to the data to be evaluated;

[0021] Evaluating the data to be evaluated by several target objects according to the multi-dimensional evaluation criteria to obtain the final evaluation result.

[0022] In some of these embodiments, it further includes:

[0023] Determining the quality score of the target object participating in the evaluation according to the final evaluation result, and correspondingly updating the object points; where the object points S e are calculated as:

[0024] ;

[0025] where k e is a coefficient related to the point level of the object, R i is the quality score of the object participating in the evaluation, Ne is the number of evaluations participated by the object, and α and β are weight coefficients.

[0026] In some of these embodiments, it further includes:

[0027] Determine the quality score of the data to be evaluated according to the final evaluation result, and correspondingly update the developer points; where the developer points S d are calculated as:

[0028] ;

[0029] where k d is a coefficient related to the point level of the developer, U d is the number of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

[0030] In a second aspect, an intelligent allocation system for evaluating subjective questions of an artificial intelligence agent is provided in this embodiment, including:

[0031] A data management module for obtaining the data to be evaluated uploaded by the developer; the data to be evaluated is the question-and-answer pair data and reference documents of the artificial intelligence agent.

[0032] A data distribution module for determining a matching object based on the domain label of the data to be evaluated; and allocating the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object.

[0033] In a third aspect, a computer device is provided in this embodiment, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent allocation method for evaluating subjective questions of the artificial intelligence agent described in the first aspect above.

[0034] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by the processor, it implements the intelligent allocation method for evaluating subjective questions of the artificial intelligence agent described in the first aspect above.

[0035] Compared with the related art, in the intelligent allocation method and system for subjective question evaluation of an artificial intelligence agent provided in this embodiment, the method includes obtaining the data to be evaluated uploaded by a developer; the data to be evaluated is the question-and-answer pair data and reference documents of the artificial intelligence agent; determining a matching object based on the domain label of the data to be evaluated; and allocating the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object. Through this embodiment, after determining the object corresponding to the domain label of the data to be evaluated, the data to be evaluated can be allocated to the target object according to the developer points and the object points, and the target object evaluates the data to be evaluated, realizing an intelligent allocation mechanism for the data to be evaluated and solving the problem of unable to effectively and efficiently allocate evaluation tasks.

[0036] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects, and advantages of this application more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of this application and constitute a part of this application. The illustrative embodiments and descriptions thereof are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0038] Figure 1 is a block diagram of the hardware structure of a terminal of an intelligent allocation method for subjective question evaluation of an artificial intelligence agent in an embodiment;

[0039] Figure 2 is a flowchart of an intelligent allocation method for subjective question evaluation of an artificial intelligence agent in an embodiment;

[0040] Figure 3 is a flowchart of an intelligent allocation method for subjective question evaluation of an artificial intelligence agent in another embodiment;

[0041] Figure 4 is a block diagram of the structure of an intelligent allocation system for subjective question evaluation of an artificial intelligence agent in an embodiment;

[0042] Figure 5 is a schematic diagram of the architecture of an intelligent allocation system for subjective question evaluation of an artificial intelligence agent in an embodiment.

[0043] In the figure: 102, a processor; 104, a memory; 106, a transmission device; 108, an input / output device; 10, a data management module; 20, a data distribution module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] To understand the purpose, technical solution, and advantages of this application more clearly, the following describes and explains this application with reference to the drawings and embodiments.

[0045] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meanings understood by those with ordinary skills in the technical field to which this application belongs. In this application, words such as "a", "an", "one kind", "the", "these", etc. do not indicate a limitation in quantity, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent in these processes, methods, products or devices. The words such as "connect", "be connected", "couple" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone. Usually, the character " / " indicates that the objects associated before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0046] The method embodiments provided in this embodiment can be executed on a terminal, a computer or a similar computing device. For example, when running on a terminal, Figure 1 is a block diagram of the hardware structure of the terminal for the intelligent allocation method of subjective question evaluation of the artificial intelligence agent in this embodiment. As Figure 1 shown, the terminal may include one or more ( Figure 1 only one is shown in Figure 1 the processor 102 and the memory 104 for storing data. Among them, the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above terminal. For example, the terminal may further include more or fewer components than Figure 1 shown in

[0047] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the intelligent allocation method for subjective question evaluation of the artificial intelligence agent in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, the above-mentioned method is implemented. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, intranet, local area network, mobile communication network, and their combinations.

[0048] The transmission device 106 is used to receive or send data via a network. The above-mentioned network includes the wireless network provided by the communication provider of the terminal. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (Radio Frequency, abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0049] In this embodiment, an intelligent allocation method for subjective question evaluation of an artificial intelligence agent is provided. Figure 2 It is the flowchart of the intelligent allocation method for subjective question evaluation of the artificial intelligence agent in this embodiment, as Figure 2 shown, the method includes the following steps:

[0050] Step S201, obtain the data to be evaluated uploaded by the developer; the data to be evaluated is the question-and-answer pair data and reference documents of the artificial intelligence agent.

[0051] Specifically, it supports uploading data to be evaluated in the user role of the developer. The data to be evaluated is the Q&A pair data of the AI Agent and the reference literature. Among them, the AI Agent is an intelligent entity that can perceive the environment, make decisions, and take actions to achieve specific goals. The Q&A pair data refers to the data set formed by the AI Agent answering user questions (including subjective questions and objective questions). During the process of the AI Agent answering questions, it may quote some external literature to support its answers. These literatures can be various types of knowledge sources such as academic papers, industry reports, books, and news. In this embodiment, subjective questions can be asked to the AI Agent to form the data to be evaluated based on the subjective questions, the answers of the AI Agent, and the reference literature, so as to allocate the evaluation tasks of the subjective questions of the AI Agent in the follow-up.

[0052] Since the application fields of the AI Agent include but are not limited to scientific research, medical, legal, education, finance, etc., the requirements and evaluation criteria for the AI Agent in each field are different. It also supports developers to upload multi-dimensional evaluation criteria corresponding to the data to be evaluated. In addition, it is also possible to format and standardize the data to be evaluated uploaded by developers to ensure the consistency and integrity of the data to be evaluated.

[0053] Step S202: Determine the matching object based on the domain label of the data to be evaluated.

[0054] Specifically, the domain labels include but are not limited to scientific research, medical, legal, education, finance, etc. Developers can assign domain labels to the data to be evaluated by themselves, or automatically assign domain labels to the data to be evaluated through semantic recognition to form a data to be evaluated distribution pool. The objects in this embodiment are used to evaluate the data to be evaluated. Specifically, they can be domain experts, evaluation tools, etc., and also have pre-set domain labels. Based on the domain label of the data to be evaluated, determine the matching object, which is a domain expert and / or an evaluation tool. For example, when there are few domain experts in a certain field, various objects can jointly evaluate the data to be evaluated.

[0055] Step S203: Allocate the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object.

[0056] Specifically, the developer points of a developer are mainly determined by the developer's point level, the quantity and quality of the uploaded data, and the object points of an object are mainly determined by the object's point level, the quantity and quality of the participated evaluations. For each data to be evaluated, after determining the matching object according to the domain label, the distribution weight of the developer is determined according to the developer points and point level of the developer who uploaded the data to be evaluated, and the distribution priorities are determined by sorting the distribution weights from high to low. Combining with the priorities sorted from high to low of the object points, each data to be evaluated is sequentially assigned to the corresponding target object according to the priorities of the developer points and the object points, so as to allocate the data to be evaluated more reasonably. Among them, the data to be evaluated is usually assigned to several target objects for independent evaluation, so as to comprehensively evaluate the performance of the AI Agent on subjective questions based on the evaluation results of each target object, and obtain the final evaluation result.

[0057] Through the above steps, the corresponding object is determined by matching according to the domain label of the data to be evaluated, making the subsequent distribution more accurate and targeted. Then, according to the developer points and the object points, the data to be evaluated is assigned to the target object, and the target object evaluates the data to be evaluated, realizing an intelligent distribution mechanism for the data to be evaluated, which is more reasonable and effective, solving the problem of unable to effectively and efficiently allocate evaluation tasks, and thus being able to better evaluate the performance of the AI Agent on subjective questions, so as to promote the wide application and performance improvement of the AI Agent in various fields.

[0058] In some of the embodiments, it further includes:

[0059] Define and manage user roles in the system, including developers and administrators. Among them, the administrator is responsible for the overall configuration and management of the system to ensure the security of data and the efficient operation of the system. The developer is responsible for uploading and maintaining the data to be evaluated and defining multi-dimensional evaluation criteria, and the object evaluates the data to be evaluated according to the multi-dimensional evaluation criteria. When the object is a domain expert, the user role also includes the domain expert. By refining the permission configuration in the role management module, the operation security and efficiency of each role in the system are ensured.

[0060] In some of the embodiments, in step S203 above, according to the developer points of the developer and the object points of the object, allocating the data to be evaluated to the corresponding target object includes the following steps:

[0061] Determine the distribution weight of the developer among all developers according to the developer points; according to the distribution weight and the object points, and combining with the current workload of the object, allocate the data to be evaluated.

[0062] Specifically, the integration level is determined based on the developer's points, and further calculation is performed to determine the allocation weight of the developer among all developers. The allocation weight represents the priority of the data to be evaluated uploaded by the developer in the intelligent allocation mechanism. The following is the calculation method of the allocation weight W d :

[0063] ;

[0064] where W d is the allocation weight of developer d, L d is the integration level of developer d (the lowest level is regarded as 1 and increases gradually), S d is the points of developer d. W k is the allocation weight of developer k, L k is the integration level of developer k, represents the weighted sum of the integration levels and points of all developers (a total of m developers).

[0065] Sort the allocation weights from high to low to determine the priority of the allocation of the data to be evaluated. Combine the priority of the object points sorted from high to low, and allocate each data to be evaluated to the corresponding target object in turn according to the priority of the developer points and the priority of the object points. In some embodiments, a daily evaluation upper limit can be set for each object. When the current workload of the object (i.e., the amount of data to be evaluated processed today) has reached the daily evaluation upper limit, no more data to be evaluated will be allocated to the object. The setting of the daily evaluation upper limit is related to the integration level of the object and is updated daily. The higher the integration level, the larger the daily evaluation upper limit.

[0066] In some embodiments, according to the allocation weight and object points, and in combination with the current workload of the object, the data to be evaluated is allocated, including the following steps:

[0067] Determine the current remaining quota of the object according to the daily evaluation upper limit and the current workload of the object; screen a number of objects with non-zero current remaining quotas as target objects according to the preset quantity and the descending order of object points; allocate the data to be evaluated to the number of target objects in turn according to the allocation weight.

[0068] Specifically, for an object with a daily evaluation upper limit set, determine the current remaining quota of the object according to its current workload. When the current remaining quota is non-zero, data to be evaluated can be allocated to the object. When the current remaining quota is zero, the evaluation quantity of the object today has reached the daily evaluation upper limit, and no data to be evaluated will be allocated to the object.

[0069] Sort the object integrals in descending order from high to low, filter out the objects with non-zero current remaining quotas as target objects with a preset quantity, sort the allocation weights from high to low to determine the priority of allocating the data to be evaluated, and allocate the data to be evaluated to the preset quantity of target objects in turn according to the priority of the allocation weights. Exemplarily, the preset quantity can be 3.

[0070] In addition, to ensure that the objects are not interfered with during the evaluation, after each object finishes processing the currently allocated data to be evaluated, the data to be evaluated is allocated according to the latest allocation weights. That is, for each object, only when the current remaining quota is non-zero and the evaluation of the data to be evaluated has been completed, the next data to be evaluated is allocated to the object.

[0071] The following is a calculation method for the object integral S e :

[0072] ;

[0073] where k e is a coefficient related to the integral level of the object, R i is the quality score of the object participating in the evaluation, N e is the number of evaluations participated by the object, and α and β are weight coefficients, which can be set to 1.

[0074] For each data to be evaluated, its final evaluation result is obtained by taking the average of the evaluation results of each target object participating in the evaluation. The quality score of the object participating in the evaluation is determined according to the gap between the evaluation result of each target object and the final evaluation result. Among them, taking three target objects as an example, the quality score of the object with the smallest gap between the evaluation result and the final evaluation result is 2, the second quality score is 1, and the quality score with the largest gap at the end is 0. The above is only an example, and the quality score can also be set according to the preset quantity of target objects, etc., without specific restrictions.

[0075] The following is a calculation method for the developer integral S d :

[0076] ;

[0077] where k d is a coefficient related to the integral level of the developer, U d is the number of all data to be evaluated uploaded by the developer (the number of question-and-answer pairs), Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients, which can be set to 1.

[0078] For each piece of data to be evaluated, its quality score is determined based on its ranking among all the final evaluation results of the data to be evaluated. In this ranking, corresponding quality scores can be set according to the order of the ranking. Exemplarily, for the top 30% in the ranking, the quality score is 3 points; for 40% - 90%, the quality score is 2 points; and for the last 10%, the quality score is 1 point.

[0079] Exemplarily, the integral levels of developers and objects can be divided into explorers (0 - 1000 points), go-getters (1001 - 5000 points), and elites (5001 points and above) according to the integral, and the corresponding coefficients related to the integral levels are 1.0, 1.1, and 1.2 respectively. It should be noted that the division of the integral levels of developers and objects can adopt different division criteria, or more or fewer integral levels can be set, and the coefficients related to the integral levels can also adopt other coefficients with a larger span, which are not specifically limited.

[0080] In this embodiment, the distribution weight is dynamically calculated based on the integral and integral level of the developer. According to the priority order of the distribution weight, the corresponding target object is determined based on the object integral among the objects that match the field, and then the data to be evaluated is assigned to the target object for evaluation. In the intelligent distribution mechanism, the data to be evaluated of developers with a high distribution weight is preferentially assigned. At the same time, the historical evaluation performance of the object is considered in combination with the object integral to ensure that high-quality data can be evaluated in a timely and high-quality manner.

[0081] In some of the embodiments, the step of obtaining the data to be evaluated uploaded by the developer in the above step S201 further includes the following steps:

[0082] Perform format standardization processing on the data to be evaluated; and determine the field label of the data to be evaluated.

[0083] Specifically, the developer uploads the data to be evaluated through the system interface, and the system conducts a preliminary check on the data to be evaluated to ensure that the format and content meet the requirements. In the data management module, after obtaining the data to be evaluated, format standardization processing is performed, and corresponding field labels are assigned. Among them, the text formats of the questions and answers in the data to be evaluated can be unified, for example, the basic format settings such as font, font size, and color are unified; the question-and-answer pairs are represented in a structured manner such as JSON (JavaScript Object Notation) or XML (Extensible Markup Language); key information such as reference documents in the data to be evaluated is marked and annotated, etc.

[0084] Domain tags include but are not limited to scientific research, medical, legal, education, finance, etc. Developers can assign domain tags to the data to be evaluated manually, or the system can automatically assign domain tags to the data to be evaluated through semantic recognition, forming a data distribution pool for the data to be evaluated.

[0085] By performing format standardization processing and assigning domain tags to the data to be evaluated in this embodiment, the consistency and integrity of the data can be ensured, and the domain tags of the data to be evaluated are assigned, providing a basis for data distribution for the subsequent intelligent allocation mechanism, enabling more accurate allocation.

[0086] In some of these embodiments, it further includes:

[0087] Obtain the multi-dimensional evaluation criteria corresponding to the data to be evaluated; several target objects evaluate the data to be evaluated according to the multi-dimensional evaluation criteria to obtain evaluation results.

[0088] Specifically, since the application fields of AI Agents include but are not limited to scientific research, medical, legal, education, finance, etc., and the requirements and evaluation criteria for AI Agents in each field are different, it also supports developers to upload the multi-dimensional evaluation criteria corresponding to the data to be evaluated.

[0089] For the target objects assigned the data to be evaluated, evaluate the data to be evaluated according to the multi-dimensional evaluation criteria provided by the developer. Finally, synthesize the evaluation results of each target object and calculate the mean to obtain the final evaluation result. In addition, when synthesizing the evaluation results of each target object, different weights can also be assigned to the evaluation results of the target objects according to the integral levels of the target objects, and the weighted average is calculated to obtain the final evaluation result.

[0090] As shown in Table 1, it is an example of the multi-dimensional evaluation criteria uploaded by the developer, which includes evaluation dimensions, descriptions, and scoring criteria, covering semantic understanding and application of domain-specific knowledge.

[0091] Table 1

[0092]

[0093] When the object is a domain expert, score the data to be evaluated from each dimension according to the data to be evaluated and the corresponding multi-dimensional evaluation criteria to obtain the evaluation result. When the object is an evaluation tool, the evaluation tool scores the data to be evaluated from each evaluation dimension by performing word segmentation processing, semantic similarity calculation, sentiment analysis, etc. on the data to be evaluated to obtain the evaluation result.

[0094] By providing multi-dimensional evaluation criteria for the data to be evaluated in this embodiment, a more comprehensive evaluation of the data to be evaluated can be achieved.

[0095] In some of these embodiments, it further includes:

[0096] Determine the quality score of the target object's participation in the evaluation according to the final evaluation result, and correspondingly update the object points; wherein, the object points S e are calculated as:

[0097] ;

[0098] wherein, k e is a coefficient related to the point level of the object, R i is the quality score of the object's participation in the evaluation, N e is the number of evaluations participated by the object, and α and β are weight coefficients.

[0099] Specifically, for each data to be evaluated, its final evaluation result is obtained by averaging the evaluation results of each target object participating in the evaluation. Determine the quality score of the object's participation in the evaluation according to the gap between the evaluation result of each target object and the final evaluation result. Among them, taking three target objects as an example, the quality score of the object with the smallest gap between the evaluation result and the final evaluation result is 2, the second quality score is 1, and the last one with the largest gap has a quality score of 0. The above is only an example, and the quality score can also be set according to a preset number of target objects, etc., without specific limitation.

[0100] In some of these embodiments, it further includes:

[0101] Determine the quality score of the data to be evaluated according to the final evaluation result, and correspondingly update the developer points; wherein, the developer points S d are calculated as:

[0102] ;

[0103] wherein, k d is a coefficient related to the point level of the developer, U d is the number of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

[0104] Specifically, for each data to be evaluated, its quality score is determined according to the ranking of the final evaluation results of all data to be evaluated. In this ranking, corresponding quality scores can be set according to the order of the ranking. Exemplarily, the quality score of the top 30% in the ranking is 3 points, 40% - 90% is 2 points, and the last 10% is 1 point.

[0105] After obtaining the final evaluation result of the data to be evaluated each time in this embodiment, the developer points and object points are updated in real time to ensure the accuracy and timeliness of the subsequent weighted allocation calculation, and a feedback mechanism is provided to continuously optimize and improve the evaluation process. In addition, it is also possible to quantify user contributions, provide a point incentive mechanism, and ensure the fairness and motivation of the incentive mechanism.

[0106] The following describes and illustrates this embodiment through preferred embodiments.

[0107] Figure 3 It is a flowchart of the intelligent allocation method for subjective question evaluation of the artificial intelligence agent in this embodiment, as Figure 3 shown, and this method includes the following steps:

[0108] Step S301: Obtain the data to be evaluated uploaded by the developer and the corresponding multi-dimensional evaluation criteria, perform format standardization processing on the data to be evaluated; and determine the domain label of the data to be evaluated.

[0109] Step S302: Determine the allocation weight of the developer among all developers according to the developer points.

[0110] Step S303: Based on the domain label of the data to be evaluated, determine the matching object.

[0111] Step S304: According to the preset quantity and the descending order of the object points, screen out several objects with non-zero current remaining quotas as target objects; and allocate the data to be evaluated to several target objects in turn according to the allocation weight.

[0112] Step S305: Have several target objects evaluate the data to be evaluated according to the multi-dimensional evaluation criteria, and obtain the final evaluation result by synthesizing the evaluation results of each target object.

[0113] Step S306: Determine the quality score of the target object's participation in the evaluation according to the final evaluation result, and correspondingly update the object points; determine the quality score of the data to be evaluated according to the final evaluation result, and correspondingly update the developer points.

[0114] Step S307: After a round of data distribution process to be evaluated ends, automatically prepare to receive new data to be evaluated and enter the next cycle.

[0115] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0116] In this embodiment, an intelligent allocation system for evaluating subjective questions of an artificial intelligence agent is also provided. This system is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated here. The following terms "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that can achieve a predetermined function. Although the system described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0117] Figure 4 is a structural block diagram of the intelligent allocation system for evaluating subjective questions of the artificial intelligence agent in this embodiment, as Figure 4 shown, this system includes:

[0118] A data management module 10, which is used to obtain the data to be evaluated uploaded by the developer; the data to be evaluated is the question-and-answer pair data and reference documents of the artificial intelligence agent.

[0119] A data distribution module 20, which is used to determine the matching object based on the domain label of the data to be evaluated; and allocate the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object.

[0120] Through the system provided in this embodiment, the corresponding object is determined by matching the domain label of the data to be evaluated, making the subsequent allocation more accurate and targeted. Then, according to the developer points and the object points, the data to be evaluated is allocated to the target object, and the target object evaluates the data to be evaluated, realizing an intelligent allocation mechanism for the data to be evaluated, which is more reasonable and effective, solving the problem of unable to effectively and efficiently allocate evaluation tasks, and further being able to better evaluate the performance of the AI Agent on subjective questions, so as to promote the wide application and performance improvement of the AI Agent in various fields.

[0121] In some of these embodiments, the data management module 10 is further used for:

[0122] Performing format standardization processing on the data to be evaluated; and determining the domain label of the data to be evaluated.

[0123] In some of these embodiments, the data distribution module 20 is further used for:

[0124] Determining the allocation weight of the developer among all developers according to the developer points; and allocating the data to be evaluated according to the allocation weight and the object points, combined with the current workload of the object.

[0125] In some of these embodiments, the data distribution module 20 is further used for:

[0126] Determine the current remaining quota of an object according to the daily evaluation limit and the current workload of the object; according to the preset quantity and the descending order of the object scores, filter out several objects with non-zero current remaining quotas as target objects; allocate the data to be evaluated to several target objects in turn according to the allocation weight.

[0127] In some of these embodiments, it further includes:

[0128] An evaluation module, configured to obtain multi-dimensional evaluation criteria corresponding to the data to be evaluated; several target objects evaluate the data to be evaluated according to the multi-dimensional evaluation criteria to obtain a final evaluation result.

[0129] In some of these embodiments, it further includes:

[0130] An integral level module, configured to determine the quality score of the target object participating in the evaluation according to the final evaluation result, and correspondingly update the object score; wherein, the object score S e is calculated as:

[0131] ;

[0132] wherein, k e is a coefficient related to the integral level of the object, R i is the quality score of the object participating in the evaluation, N e is the number of times the object participates in the evaluation, and α and β are weight coefficients.

[0133] In some of these embodiments, the integral level module is further configured to:

[0134] Determine the quality score of the data to be evaluated according to the final evaluation result, and correspondingly update the developer score; wherein, the developer score S d is calculated as:

[0135] ;

[0136] wherein, k d is a coefficient related to the integral level of the developer, U d is the number of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

[0137] In some of these embodiments, it further includes:

[0138] A role management module, configured to manage user roles; the user roles include administrators, developers, and domain experts.

[0139] Figure 5 is a schematic diagram of the architecture of the intelligent allocation system for subjective question evaluation of the artificial intelligence agent in this embodiment, asFigure 5 As shown in the figure, the system includes a data management module, a data distribution module, a role management module, an evaluation module, and a point and level module.

[0140] Among them, the data management module sends the data to be evaluated obtained to the data distribution module. The data distribution module, through an intelligent allocation mechanism, allocates the data to be evaluated to objects such as domain experts or evaluation tools according to the developer points and object points. In the evaluation module, the object evaluates the data to be evaluated, comprehensively obtains the final evaluation result, and sends the evaluated data and the final evaluation result to the point and level module. In the point and level module, the user points and point levels of the developer and the object are updated, and feedback is sent to the data distribution module for subsequent allocation of the data to be evaluated.

[0141] The role management module is used to manage user roles; the user roles include administrators, developers, and domain experts.

[0142] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented either by software or by hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination form.

[0143] In this embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0144] Optionally, the above computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0145] It should be noted that the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated in this embodiment.

[0146] In addition, in combination with the intelligent allocation method for evaluating subjective questions of an artificial intelligence agent provided in the above embodiments, a storage medium may also be provided to implement it in this embodiment. A computer program is stored on the storage medium; when the computer program is executed by a processor, it implements any one of the intelligent allocation methods for evaluating subjective questions of an artificial intelligence agent in the above embodiments.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties.

[0148] It should be understood that the specific embodiments described herein are for explaining this application rather than limiting it. All other embodiments obtained by those of ordinary skill in the art based on the embodiments provided in this application without creative efforts shall fall within the protection scope of this application.

[0149] Obviously, the accompanying drawings are only some examples or embodiments of this application. For those of ordinary skill in the art, this application can also be applied to other similar situations based on these drawings without creative efforts. Additionally, it can be understood that although the work done during this development process may be complex and time-consuming, for those of ordinary skill in the art, certain design, manufacturing, or production changes based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient disclosure of this application.

[0150] The term "embodiment" in this application means that the specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of this application. The phrase appears in various positions in the specification and does not necessarily mean the same embodiment, nor does it mean being independent or alternative to other embodiments and mutually exclusive. Those of ordinary skill in the art can clearly or implicitly understand that the embodiments described in this application can be combined with other embodiments without conflict.

[0151] The above-described embodiments only represent several implementation manners of this application. Their descriptions are relatively specific and detailed, but they should not be construed as limitations on the patent protection scope. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. An intelligent allocation method for subjective problem evaluation of an artificial intelligence agent, characterized in that, Including: Obtain the data to be evaluated uploaded by the developer; The data to be evaluated is the question-and-answer pair data and reference documents of the artificial intelligence agent; Based on the domain label of the data to be evaluated, determine the matching object; According to the developer points of the developer and the object points of the object, allocate the data to be evaluated to the corresponding target object; where it includes the following steps: Determine the current remaining quota of the object according to the daily evaluation limit and the current workload of the object; according to the preset quantity and the descending order of the object points, screen out several objects with non-zero current remaining quotas as target objects; sort the allocation weights from high to low to determine the priority of allocating the data to be evaluated, and sequentially allocate the data to be evaluated to the preset quantity of target objects according to the priority of the allocation weights; the allocation weight is determined by the developer points and the corresponding point levels.

2. The method according to claim 1, characterized in that, The obtaining of the data to be evaluated uploaded by the developer further includes: Perform format standardization processing on the data to be evaluated; and determine the domain label of the data to be evaluated.

3. The method according to claim 1, wherein It also includes: Obtain the multi-dimensional evaluation criteria corresponding to the data to be evaluated; Have several of the target objects evaluate the data to be evaluated according to the multi-dimensional evaluation criteria to obtain the final evaluation result.

4. The method according to claim 3, characterized in that It also includes: Determine the quality score of the target object participating in the evaluation according to the final evaluation result, and correspondingly update the object integral; where the object integral S e is calculated as follows: ； where k e is a coefficient related to the integral level of the said object, R i is the quality score of the said object participating in the evaluation, N e is the number of times the said object participates in the evaluation, and α and β are weight coefficients.

5. The method according to claim 3, wherein It also includes: Determine the quality score of the data to be evaluated according to the final evaluation result, and correspondingly update the developer points; wherein, the developer points S d are calculated as follows: ； Among them, k d is a coefficient related to the integral level of the developer, U d is the quantity of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

6. An intelligent allocation system for evaluating subjective questions of an artificial intelligence agent, characterized in that, Including: A data management module for obtaining the data to be evaluated uploaded by the developer; The data to be evaluated is the question-and-answer pair data and reference documents of the artificial intelligence agent; A data distribution module for determining the matching object based on the domain label of the data to be evaluated; According to the developer points of the developer and the object points of the object, allocate the data to be evaluated to the corresponding target object; where it includes the following steps: Determine the current remaining quota of the object according to the daily evaluation limit and the current workload of the object; according to the preset quantity and the descending order of the object points, screen out several objects with non-zero current remaining quotas as target objects; sort the allocation weights from high to low to determine the priority of allocating the data to be evaluated, and sequentially allocate the data to be evaluated to the preset quantity of target objects according to the priority of the allocation weights; the allocation weight is determined by the developer points and the corresponding point levels.

7. A computer device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is set to run the computer program to execute the intelligent allocation method for evaluating subjective questions of the artificial intelligence agent according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent allocation method for evaluating subjective questions of the artificial intelligence agent according to any one of claims 1 to 5.

Images