Intelligent allocation method and system for subjective question evaluation of artificial intelligence agent

Abstract

The invention relates to an intelligent allocation method and system for artificial intelligence agent subjective question evaluation, and the method comprises the steps: obtaining to-be-evaluated data uploaded by a developer; the to-be-evaluated data are question and answer pair data and reference literatures of an artificial intelligence agent; determining a matched object based on the field label of the to-be-evaluated data; and distributing the to-be-evaluated data to the corresponding target object according to the developer integral of the developer and the object integral of the object. According to the method and the device, the to-be-evaluated data can be allocated to the target object according to the developer integral and the object integral after the object corresponding to the to-be-evaluated data field label is determined, and the to-be-evaluated data is evaluated by the target object, so that an intelligent allocation mechanism of the to-be-evaluated data is realized, and the problem that the evaluation task cannot be effectively and efficiently allocated is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an intelligent allocation method and system for subjective problem evaluation by artificial intelligence agents. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the breakthrough in the field of large language models, AI Agent (Artificial Intelligence Agent) has outstanding performance in information retrieval, professional question answering, content generation, etc., relying on its powerful natural language processing capabilities. However, AI Agent's handling of subjective questions involves complex semantic understanding and professional knowledge in multiple fields, which is full of uncertainty and complexity. Correspondingly, the evaluation of AI Agent's subjective questions also faces many challenges.

[0003] Traditional evaluation methods usually rely on objective indicators, such as accuracy and response time, but these indicators are difficult to fully measure the performance of AI Agents on subjective issues. Therefore, the evaluation tasks of subjective issues can be assigned to intelligent analysis tools or domain experts for evaluation, especially in complex scenarios involving multiple fields and cultural backgrounds, which can provide more accurate evaluation conclusions. At present, although there are some frameworks and tools for AI Agent evaluation, they generally lack an intelligent allocation mechanism for evaluation tasks and cannot effectively and efficiently allocate evaluation tasks.

[0004] There is currently no effective solution to the problem that related technologies cannot effectively and efficiently allocate evaluation tasks. Summary of the invention

[0005] Based on this, it is necessary to provide an intelligent allocation method and system for subjective question evaluation of artificial intelligence agents, which can solve the above-mentioned technical problems and improve the quality and efficiency of AI Agents in subjective question evaluation.

[0006] In the first aspect, in this embodiment, a method for intelligently allocating subjective question evaluation by an artificial intelligence agent is provided, comprising: Obtain the data to be evaluated uploaded by the developer; the data to be evaluated is the question-answer pair data and references of the artificial intelligence agent; Determining matching objects based on the domain labels of the data to be evaluated; The data to be evaluated is distributed to the corresponding target object according to the developer points of the developer and the object points of the object.

[0007] In some embodiments, allocating the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object includes: Determine, according to the developer's points, a distribution weight of the developer among all developers; The data to be evaluated is allocated according to the allocation weight and the object score in combination with the current workload of the object.

[0008] In some embodiments, allocating the data to be evaluated according to the allocation weight and the object score in combination with the current workload of the object includes: Determine the current remaining quota of the object according to the daily evaluation upper limit of the object and the current workload; According to the preset quantity and the descending order of the object points, a number of objects whose current remaining quota is not zero are screened and obtained as the target objects; According to the allocation weights, the data to be evaluated are allocated to a number of the target objects in sequence.

[0009] In some of the embodiments, obtaining the data to be evaluated uploaded by the developer further includes: The format of the data to be evaluated is standardized; and the domain label of the data to be evaluated is determined.

[0010] In some of the embodiments, it also includes: Obtaining a multi-dimensional evaluation standard corresponding to the data to be evaluated; The target objects evaluate the data to be evaluated according to the multi-dimensional evaluation standard to obtain a final evaluation result.

[0011] In some of the embodiments, it also includes: The quality score of the target object participating in the evaluation is determined according to the final evaluation result, and the object score is updated accordingly; wherein the object score S e The calculation is: ; Among them, k e is the coefficient related to the integral level of the object, R i is the quality score of the object participating in the evaluation, N e is the number of objects participating in the evaluation, and α and β are weight coefficients.

[0012] In some of the embodiments, it also includes: Determine the quality score of the data to be evaluated according to the final evaluation result, and update the developer score accordingly; wherein the developer score Sd The calculation is: ; Among them, k d is a coefficient related to the developer's score level, U d The number of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

[0013] In the second aspect, in this embodiment, an intelligent allocation system for subjective question evaluation by an artificial intelligence agent is provided, including: A data management module is used to obtain the data to be evaluated uploaded by the developer; the data to be evaluated is the question-answer pair data and references of the artificial intelligence agent; The data distribution module is used to determine the matching object based on the domain label of the data to be evaluated; and distribute the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object.

[0014] In the third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the intelligent allocation method for subjective question evaluation of an artificial intelligence agent as described in the first aspect above is implemented.

[0015] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the intelligent allocation method for subjective problem evaluation of an artificial intelligence agent described in the first aspect above is implemented.

[0016] Compared with the related art, the intelligent allocation method and system for subjective question evaluation of artificial intelligence agents provided in this embodiment obtains the data to be evaluated uploaded by the developer; the data to be evaluated is the question-answer pair data and references of the artificial intelligence agent; based on the domain label of the data to be evaluated, the matching object is determined; according to the developer's developer points and the object points of the object, the data to be evaluated is allocated to the corresponding target object. Through this embodiment, after determining the object corresponding to the domain label of the data to be evaluated, the data to be evaluated is allocated to the target object according to the developer points and the object points, and the target object evaluates the data to be evaluated, realizing the intelligent allocation mechanism of the data to be evaluated, and solving the problem of being unable to effectively and efficiently allocate evaluation tasks.

[0017] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a hardware structure block diagram of a terminal of an intelligent allocation method for subjective question evaluation by an artificial intelligence agent in an embodiment; Figure 2 is a flow chart of a method for intelligently allocating subjective question evaluations by an artificial intelligence agent in one embodiment; Figure 3 is a flow chart of a method for intelligently allocating subjective question assessments by artificial intelligence agents in another embodiment; Figure 4 is a structural block diagram of an intelligent allocation system for subjective question evaluation by an artificial intelligence agent in one embodiment; Figure 5 The diagram is a schematic diagram of the architecture of an intelligent allocation system for subjective question evaluation by an artificial intelligence agent in one embodiment.

[0019] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, data management module; 20, data distribution module. DETAILED DESCRIPTION

[0020] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 1 is a hardware structure block diagram of a terminal of the intelligent allocation method for subjective question evaluation by artificial intelligence agent in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0023] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the intelligent allocation method of subjective question evaluation of artificial intelligence agents in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0025] In this embodiment, a method for intelligently assigning subjective question evaluation by an artificial intelligence agent is provided. Figure 2 is a flow chart of the intelligent allocation method of subjective question evaluation by artificial intelligence agent in this embodiment. Figure 2 As shown, the method comprises the following steps: Step S201, obtaining the data to be evaluated uploaded by the developer; the data to be evaluated is the question-answer pair data and references of the artificial intelligence agent.

[0026] Specifically, it supports uploading the data to be evaluated with the user role of the developer. The data to be evaluated is the question-answer data and references of the AI ​​Agent. Among them, the AI ​​Agent is an intelligent entity that can perceive the environment, make decisions and take actions to achieve specific goals. The question-answer data refers to the data set formed by the AI ​​Agent answering the user's questions (including subjective questions and objective questions). In the process of answering questions, the AI ​​Agent may cite some external documents to support its answer. These documents can be various types of knowledge sources such as academic papers, industry reports, books, news information, etc. In this embodiment, subjective questions can be asked to the AI ​​Agent to form the data to be evaluated based on the subjective questions, the answers of the AI ​​Agent and the references, so as to allocate the evaluation tasks of the AI ​​Agent's subjective questions in the future.

[0027] Since the application fields of AI Agent include but are not limited to scientific research, medical treatment, law, education, finance, etc., each field has different requirements and evaluation standards for AI Agent, and can also support developers to upload multi-dimensional evaluation standards corresponding to the data to be evaluated. In addition, the data to be evaluated uploaded by the developer can be formatted and standardized to ensure the consistency and integrity of the data to be evaluated.

[0028] Step S202: determining matching objects based on the domain labels of the data to be evaluated.

[0029] Specifically, domain labels include but are not limited to scientific research, medical treatment, law, education, finance, etc. The developer can assign domain labels to the data to be evaluated by himself, or the domain labels can be automatically assigned to the data to be evaluated through semantic recognition to form a distribution pool of data to be evaluated. The objects in this embodiment are used to evaluate the data to be evaluated, and can specifically be domain experts, evaluation tools, etc., which also have pre-set domain labels. Based on the domain labels of the data to be evaluated, the matching objects are determined, which are domain experts and / or evaluation tools. For example, when there are fewer domain experts in a certain field, various objects can jointly evaluate the data to be evaluated.

[0030] Step S203: distribute the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object.

[0031] Specifically, the developer's developer points are mainly determined by the developer's point level, the amount and quality of uploaded data, and the object points of the object are mainly determined by the object's point level, the amount and quality of participating in the evaluation. For each data to be evaluated, after determining the matching object according to the domain label, the developer's allocation weight is determined according to the developer's developer points and point level of the developer who uploaded the data to be evaluated, and the allocation priority is determined by sorting the allocation weights from high to low. Combined with the priority of the object points sorted from high to low, each data to be evaluated is assigned to the corresponding target object in turn according to the priority of the developer points and the priority of the object points, so as to allocate the evaluation data more reasonably. Among them, the data to be evaluated is usually assigned to several target objects for independent evaluation, so as to evaluate the performance of AIAgent on subjective issues by combining the evaluation results of each target object to obtain the final evaluation results.

[0032] Through the above steps, the corresponding objects are determined according to the domain label matching of the data to be evaluated, making the subsequent allocation more accurate and targeted. Then, according to the developer points and object points, the data to be evaluated is allocated to the target object, and the target object evaluates the data to be evaluated. This realizes an intelligent allocation mechanism for the data to be evaluated, which is more reasonable and effective, and solves the problem of being unable to effectively and efficiently allocate evaluation tasks. It can then better evaluate the performance of AI Agent on subjective issues, so as to promote the wide application and performance improvement of AI Agent in various fields.

[0033] In some of the embodiments, it also includes: Define and manage user roles in the system, including developers and administrators. The administrator is responsible for the overall configuration and management of the system to ensure data security and efficient operation of the system. Developers are responsible for uploading and maintaining the data to be evaluated and defining multi-dimensional evaluation standards. The subject will evaluate the data to be evaluated according to the multi-dimensional evaluation standards. When the subject is a domain expert, the user role also includes domain experts. In the role management module, by refining the permission configuration, ensure the safety and efficiency of each role in the system.

[0034] In some embodiments, the step S203 allocates the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object, including the following steps: According to the developer points, the distribution weight of the developer among all developers is determined; according to the distribution weight and object points, the data to be evaluated is distributed in combination with the current workload of the object.

[0035] Specifically, the points level is determined according to the developer's points, and the allocation weight of the developer among all developers is further calculated. The allocation weight represents the priority of the data to be evaluated uploaded by the developer in the intelligent allocation mechanism. d A way to calculate: ; Among them, W d is the allocation weight of developer d, L d is the score level of developer d (the lowest level is regarded as 1, and it increases step by step), S d is the points of developer d. k is the allocation weight of developer k, L k is the score level of developer k, Represents the weighted sum of the score levels and scores of all developers (a total of m developers).

[0036] The allocation weights are sorted from high to low to determine the priority of the allocation of the data to be evaluated. Combined with the priority of the object points sorted from high to low, each data to be evaluated is allocated to the corresponding target object in turn according to the priority of the developer points and the priority of the object points. In some embodiments, a daily evaluation limit can be allocated to each object. When the current workload of the object (i.e., the amount of data to be evaluated that has been processed today) has reached the daily evaluation limit, no more data to be evaluated will be allocated to the object. The setting of the daily evaluation limit is related to the object's score level and is updated daily. The higher the score level, the larger the daily evaluation limit.

[0037] In some embodiments, allocating the data to be evaluated according to the allocation weight and the object score in combination with the current workload of the object includes the following steps: Determine the current remaining quota of the object based on the daily assessment limit and current workload of the object; select several objects whose current remaining quota is not zero as target objects based on the preset quantity and descending sorting of the object points; and allocate the data to be assessed to several target objects in turn according to the allocation weights.

[0038] Specifically, for objects with a daily evaluation upper limit, the current remaining quota of the object is determined based on its current workload. When the current remaining quota is not zero, data to be evaluated can be allocated to the object. When the current remaining quota is zero, the number of evaluations of the object today has reached the daily evaluation upper limit, and no data to be evaluated will be allocated to the object.

[0039] The object scores are sorted in descending order from high to low, and the objects whose current remaining quota is not zero are selected as target objects according to a preset number, and the allocation weights are sorted from high to low to determine the priority of the data to be evaluated, and the data to be evaluated is sequentially allocated to the preset number of target objects according to the priority of the allocation weights. For example, the preset number can be 3.

[0040] In addition, in order to ensure that the object is not disturbed by other factors during evaluation, after each object has processed the currently allocated data to be evaluated, the data to be evaluated is allocated according to the latest allocation weight. That is, for each object, the next data to be evaluated is allocated to the object only after the current remaining quota is not zero and the evaluation of the data to be evaluated has been completed.

[0041] The following is the object integral S e A way to calculate: ; Among them, k e is the coefficient related to the integral level of the object, R i is the quality score of the object participating in the evaluation, N e is the number of objects participating in the evaluation, α and β are weight coefficients, which can be set to 1.

[0042] For each data to be evaluated, its final evaluation result is obtained by averaging the evaluation results of each target object participating in the evaluation. The quality score of the object participating in the evaluation is determined based on the difference between the evaluation result of each target object and the final evaluation result. For example, taking three target objects as an example, the quality score of the object with the smallest difference between the evaluation result and the final evaluation result is 2, the next best quality score is 1, and the final quality score with the largest difference is 0. The above is only an example, and the quality score can also be set according to a preset number of target objects, etc., without specific limitation.

[0043] The following is the developer points S d A way to calculate: ; Among them, k d is the coefficient related to the developer's score level, U d The number of all data to be evaluated uploaded by developers (number of question and answer pairs), Q d is the quality score of the data to be evaluated uploaded by the developer. δ and ε are weight coefficients and can be set to 1.

[0044] For each data to be evaluated, its quality score is determined according to the ranking of the final evaluation results among all the data to be evaluated. In this ranking, the corresponding quality score can be set according to the order of ranking. For example, the quality score of the top 30% in the ranking is 3 points, the quality score of 40% to 90% is 2 points, and the quality score of the last 10% is 1 point.

[0045] For example, the points level of developers and objects can be divided into explorers (0-1000 points), enterprising (1001-5000 points), and excellent (5001 points and above) according to the points, and the coefficients related to the points level are 1.0, 1.1, and 1.2 respectively. It should be noted that the points level division of developers and objects can adopt different division standards, and more or fewer points levels can be set. The coefficients related to the points level can also adopt other coefficients with a larger span, which are not specifically limited.

[0046] In this embodiment, allocation weights are dynamically calculated based on the developer's points and point levels. According to the priority order of the allocation weights, the corresponding target objects are determined according to the object points in the field-matching objects, and then the data to be evaluated are allocated to the target objects for evaluation. In the intelligent allocation mechanism, the data to be evaluated of developers with high allocation weights are allocated first. At the same time, the historical evaluation performance of the object is taken into account in combination with the object points to ensure that high-quality data is evaluated in a timely and high-quality manner.

[0047] In some embodiments, the step S201 of obtaining the data to be evaluated uploaded by the developer further includes the following steps: Standardize the format of the data to be evaluated and determine the domain labels of the data to be evaluated.

[0048] Specifically, developers upload the data to be evaluated through the system interface, and the system performs a preliminary check on the data to be evaluated to ensure that the format and content meet the requirements. In the data management module, the data to be evaluated is obtained and the format is standardized, and the corresponding field labels are assigned. Among them, the text format of the questions and answers in the data to be evaluated can be unified, for example, the basic format settings such as font, font size, and color can be unified; structured methods such as JSON (JavaScript Object Notation) or XML (Extensible Markup Language) are used to represent question and answer pairs; key information such as references in the data to be evaluated is marked and annotated, etc.

[0049] Domain labels include but are not limited to scientific research, medical care, law, education, finance, etc. Developers can assign domain labels to the data to be evaluated by themselves, or they can automatically assign domain labels to the data to be evaluated through semantic recognition to form a distribution pool for data to be evaluated.

[0050] By standardizing the format of the data to be evaluated and assigning domain labels in this embodiment, the consistency and integrity of the data can be ensured, and the domain labels of the data to be evaluated are assigned to provide a basis for data distribution for the subsequent intelligent allocation mechanism, which can allocate more accurately.

[0051] In some of the embodiments, it also includes: Obtain multi-dimensional evaluation standards corresponding to the data to be evaluated; a number of target objects evaluate the data to be evaluated according to the multi-dimensional evaluation standards to obtain evaluation results.

[0052] Specifically, since the application fields of AI Agents include but are not limited to scientific research, medical care, law, education, finance, etc., each field has different requirements and evaluation standards for AI Agents. It can also support developers to upload multi-dimensional evaluation standards corresponding to the data to be evaluated.

[0053] For the target objects assigned to the data to be evaluated, the data to be evaluated is evaluated according to the multi-dimensional evaluation standards provided by the developer, and finally the evaluation results of each target object are combined and the average is calculated to obtain the final evaluation result. In addition, when combining the evaluation results of each target object, different weights can be assigned to the evaluation results of the target object according to the score level of the target object, and the weighted average is calculated to obtain the final evaluation result.

[0054] As shown in Table 1, this is an example of a multi-dimensional evaluation standard uploaded by a developer, which includes evaluation dimensions, descriptions, and scoring criteria, covering semantic understanding and domain-specific knowledge application.

[0055] Table 1

[0056] When the object is a domain expert, the evaluation data is scored from each dimension according to the data to be evaluated and the corresponding multi-dimensional evaluation criteria to obtain the evaluation results. When the object is an evaluation tool, the evaluation tool scores the evaluation data from each evaluation dimension by performing word segmentation, semantic similarity calculation, sentiment analysis, etc. on the evaluation data to obtain the evaluation results.

[0057] By providing multi-dimensional evaluation criteria for the data to be evaluated in this embodiment, a more comprehensive evaluation of the data to be evaluated can be achieved.

[0058] In some of the embodiments, it also includes: The quality score of the target object participating in the evaluation is determined according to the final evaluation results, and the object score is updated accordingly; among which, the object score S e The calculation is: ; Among them, k e is the coefficient related to the integral level of the object, R i is the quality score of the object participating in the evaluation, N e is the number of objects participating in the evaluation, and α and β are weight coefficients.

[0059] Specifically, for each data to be evaluated, its final evaluation result is obtained by averaging the evaluation results of each target object participating in the evaluation. The quality score of the object participating in the evaluation is determined based on the difference between the evaluation result of each target object and the final evaluation result. Taking three target objects as an example, the quality score of the object with the smallest difference between the evaluation result and the final evaluation result is 2, the next best quality score is 1, and the final best quality score is 0. The above is only an example, and the quality score can also be set according to a preset number of target objects, etc., without specific limitation.

[0060] In some of the embodiments, it also includes: The quality score of the data to be evaluated is determined based on the final evaluation results, and the developer points are updated accordingly; among them, the developer points S d The calculation is: ; Among them, k d is the coefficient related to the developer's score level, U d The number of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

[0061] Specifically, for each data to be evaluated, its quality score is determined according to the ranking of the final evaluation results among all the data to be evaluated. In this ranking, the corresponding quality score can be set according to the order of ranking. For example, the quality score of the top 30% in the ranking is 3 points, the quality score of 40% to 90% is 2 points, and the quality score of the last 10% is 1 point.

[0062] In this embodiment, after obtaining the final evaluation results of the data to be evaluated, the developer points and object points are updated in real time to ensure the accuracy and timeliness of the subsequent weight calculation, and provide a feedback mechanism to continuously optimize and improve the evaluation process. In addition, it can also quantify user contributions, provide a points incentive mechanism, and ensure the fairness and incentive of the incentive mechanism.

[0063] The present embodiment is described and illustrated below through preferred embodiments.

[0064] Figure 3 is a flow chart of the intelligent allocation method of the subjective question evaluation of the artificial intelligence agent of this embodiment. Figure 3 As shown, the method comprises the following steps: Step S301, obtaining the data to be evaluated and the corresponding multi-dimensional evaluation standards uploaded by the developer, performing format standardization processing on the data to be evaluated; and determining the field label of the data to be evaluated.

[0065] Step S302: Determine the distribution weight of the developer among all developers based on the developer points.

[0066] Step S303: determining matching objects based on the domain labels of the data to be evaluated.

[0067] Step S304, according to the preset quantity and descending order of the object points, a number of objects with current remaining quotas not being zero are screened and obtained as target objects; and the data to be evaluated are sequentially allocated to the target objects according to the allocation weights.

[0068] Step S305 , a number of target objects evaluate the evaluation data according to the multi-dimensional evaluation standard, and the evaluation result of each target object is combined to obtain a final evaluation result.

[0069] Step S306, determine the quality score of the target object participating in the evaluation according to the final evaluation result, and update the object score accordingly; determine the quality score of the data to be evaluated according to the final evaluation result, and update the developer score accordingly.

[0070] Step S307: after a round of distribution process of the data to be evaluated is completed, the system automatically prepares to receive new data to be evaluated and enters the next cycle.

[0071] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0072] In this embodiment, an intelligent allocation system for subjective question evaluation by an artificial intelligence agent is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below may be a combination of software and / or hardware that implements predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0073] Figure 4: is a structural block diagram of the intelligent distribution system of the artificial intelligence agent subjective question evaluation of this embodiment, such as Figure 4 As shown, the system includes: The data management module 10 is used to obtain the data to be evaluated uploaded by the developer; the data to be evaluated is the question-answer pair data and references of the artificial intelligence agent; The data distribution module 20 is used to determine matching objects based on the domain labels of the data to be evaluated; and distribute the data to be evaluated to corresponding target objects according to the developer points of the developer and the object points of the object.

[0074] Through the system provided in this embodiment, the corresponding object is determined according to the matching of the field label of the data to be evaluated, so that the subsequent allocation is more accurate and targeted, and then the data to be evaluated is allocated to the target object according to the developer points and the object points, and the target object evaluates the data to be evaluated, thereby realizing an intelligent allocation mechanism for the data to be evaluated, which is more reasonable and effective, and solves the problem of being unable to effectively and efficiently allocate evaluation tasks, so as to better evaluate the performance of AI Agent on subjective problems, so as to promote the wide application and performance improvement of AI Agent in various fields.

[0075] In some of the embodiments, the data management module 10 is further configured to: Standardize the format of the data to be evaluated and determine the domain labels of the data to be evaluated.

[0076] In some embodiments, the data distribution module 20 is further configured to: According to the developer points, the distribution weight of the developer among all developers is determined; according to the distribution weight and object points, the data to be evaluated is distributed in combination with the current workload of the object.

[0077] In some embodiments, the data distribution module 20 is further configured to: Determine the current remaining quota of the object based on the daily assessment limit and current workload of the object; select several objects whose current remaining quota is not zero as target objects based on the preset quantity and descending sorting of the object points; and allocate the data to be assessed to several target objects in turn according to the allocation weights.

[0078] In some of the embodiments, it also includes: The evaluation module is used to obtain the multi-dimensional evaluation standards corresponding to the data to be evaluated; a number of target objects evaluate the data to be evaluated according to the multi-dimensional evaluation standards to obtain the final evaluation results.

[0079] In some of the embodiments, it also includes: The score level module is used to determine the quality score of the target object participating in the evaluation according to the final evaluation results, and update the object score accordingly; among them, the object score S e The calculation is: ; Among them, k e is the coefficient related to the integral level of the object, R i is the quality score of the object participating in the evaluation, N e is the number of objects participating in the evaluation, and α and β are weight coefficients.

[0080] In some of the embodiments, the score level module is further configured to: The quality score of the data to be evaluated is determined based on the final evaluation results, and the developer points are updated accordingly; among them, the developer points S d The calculation is: ; Among them, k d is the coefficient related to the developer's score level, U d The number of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

[0081] In some of the embodiments, it also includes: The role management module is used to manage user roles; user roles include administrators, developers, and domain experts.

[0082] Figure 5 is a schematic diagram of the architecture of the intelligent allocation system for subjective question evaluation by artificial intelligence agents in this embodiment. Figure 5 As shown, the system includes a data management module, a data distribution module, a role management module, an evaluation module and a score level module.

[0083] Among them, the data management module sends the obtained data to be evaluated to the data distribution module. The data distribution module distributes the data to be evaluated to objects such as domain experts or evaluation tools according to the developer points and object points through an intelligent allocation mechanism. In the evaluation module, the object evaluates the data to be evaluated, and comprehensively obtains the final evaluation results. The evaluated data and the final evaluation results are sent to the point level module. In the point level module, the user points and point levels of the developer and the object are updated, and fed back to the data distribution module for subsequent distribution of the data to be evaluated.

[0084] The role management module is used to manage user roles; user roles include administrators, developers, and domain experts.

[0085] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0086] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0087] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0088] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0089] In addition, in combination with the intelligent allocation method of subjective problem evaluation of artificial intelligence agents provided in the above embodiments, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any intelligent allocation method of subjective problem evaluation of artificial intelligence agents in the above embodiments is implemented.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0091] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0092] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0093] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0094] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An intelligent allocation method for subjective question evaluation by artificial intelligence agents, characterized in that: include: Get the data to be evaluated uploaded by the developer; The data to be evaluated are question-answer pair data and references of the artificial intelligence agent; Determining matching objects based on the domain labels of the data to be evaluated; The data to be evaluated is distributed to the corresponding target object according to the developer points of the developer and the object points of the object.

2. The method according to claim 1, characterized in that The allocating the to-be-assessed data to the corresponding target object according to the developer points of the developer and the object points of the object includes: Determine, according to the developer's points, a distribution weight of the developer among all developers; The data to be evaluated is allocated according to the allocation weight and the object score in combination with the current workload of the object.

3. The method according to claim 2, characterized in that The allocating the data to be evaluated according to the allocation weight and the object score in combination with the current workload of the object includes: Determine the current remaining quota of the object according to the daily evaluation upper limit of the object and the current workload; According to the preset quantity and the descending order of the object points, a number of objects whose current remaining quota is not zero are screened and obtained as the target objects; According to the allocation weights, the data to be evaluated are allocated to a number of the target objects in sequence.

4. The method according to claim 1, characterized in that The obtaining of the data to be evaluated uploaded by the developer also includes: The format of the data to be evaluated is standardized; and the domain label of the data to be evaluated is determined.

5. The method according to claim 1, characterized in that Also includes: Obtaining a multi-dimensional evaluation standard corresponding to the data to be evaluated; The target objects evaluate the data to be evaluated according to the multi-dimensional evaluation standard to obtain a final evaluation result.

6. The method according to claim 5, characterized in that Also includes: The quality score of the target object participating in the evaluation is determined according to the final evaluation result, and the object score is updated accordingly; wherein the object score S e The calculation is: ； Among them, k e is the coefficient related to the integral level of the object, R i is the quality score of the object participating in the evaluation, N e is the number of objects participating in the evaluation, and α and β are weight coefficients.

7. The method according to claim 5, characterized in that Also includes: Determine the quality score of the data to be evaluated according to the final evaluation result, and update the developer score accordingly; wherein the developer score S d The calculation is: ； Among them, k d is a coefficient related to the developer's score level, U d The number of all data to be evaluated uploaded by the developer, Q d is the quality score of the data to be evaluated uploaded by the developer, and δ and ε are weight coefficients.

8. An intelligent distribution system for subjective question evaluation by artificial intelligence agents, characterized in that: include: The data management module is used to obtain the data to be evaluated uploaded by the developer; The data to be evaluated are question-answer pair data and references of the artificial intelligence agent; The data distribution module is used to determine the matching object based on the domain label of the data to be evaluated; and distribute the data to be evaluated to the corresponding target object according to the developer points of the developer and the object points of the object.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the intelligent allocation method for subjective question evaluation of an artificial intelligence agent as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent allocation method for subjective question evaluation of an artificial intelligence agent as described in any one of claims 1 to 7 are implemented.

Images