A large model user annotation quality calculation method and electronic device

Abstract

The present application discloses a large model user annotation quality calculation method and electronic device, belonging to the field of large model technology. The method includes: the target annotation user performs Z annotations, and the Z annotations include N first-class annotations that trigger weak models and M normal annotations that do not trigger weak models, wherein Z=N+M, and Z, M, and N are positive integers; each of the Z annotations performed by the target annotation user corresponds to a annotation score, and an instruction score is calculated based on the annotation score; an instruction diversity score is calculated based on the input instruction content of the target annotation user; based on the instruction score and the instruction diversity score, the quality ranking score of the target annotation user is calculated. The present application greatly improves the timeliness and accuracy of quality verification, can promptly discover problems and correct them, and ensure the accuracy and efficiency of annotation.

Description

Technical Field

[0001] The present application belongs to the technical field of large models, and in particular relates to a large model user annotation quality calculation method and electronic equipment. Background Art

[0002] In the field of artificial intelligence, large language models have shown powerful capabilities, but evaluating these models and ensuring that their outputs are consistent with human preferences remains a huge challenge. The Large Model Content Battle Evaluation product is an open, crowdsourced large model annotation platform whose main goal is to collect human feedback and evaluate the capabilities of large models in real-world scenarios.

[0003] Chatbot Arena is an open dataset, model, system and evaluation tool first created by students and teachers from the University of California, Berkeley, in collaboration with the University of California, San Diego and Carnegie Mellon University. The model is evaluated by collecting feedback and preference votes from a large number of human participants. This crowdsourcing evaluation method can make full use of the collective wisdom of the vast number of users, evaluate the model from multiple perspectives, and make the evaluation results more representative and comprehensive.

[0004] In the existing technology, there are obvious deficiencies in the large-scale model content battle evaluation platform. The primary problem is that the platform currently does not rank the users who participate in the crowdsourcing evaluation or conduct separate manual annotation quality inspections. This means that the professional level, annotation accuracy, and work efficiency of the annotators cannot be effectively distinguished and evaluated; this situation will lead to uneven annotation quality, which in turn affects the accuracy and reliability of large-scale model evaluation.

[0005] In view of the above problems, a large model user annotation quality calculation method and electronic device are proposed in the present application. Summary of the invention

[0006] In order to address the deficiencies of the prior art, the present application provides a large model user annotation quality calculation method and electronic device, which is used to solve the problems in the prior art such as uneven quality of manual annotation, which affects the accuracy and reliability of large model evaluation.

[0007] The technical effects to be achieved by this application are achieved through the following solutions:

[0008] In a first aspect, the present application provides a method for calculating the quality of user annotations of a large model, the method comprising:

[0009] The target annotation user performs Z annotations, wherein the Z annotations include N first-class annotations that trigger the weak model and M normal annotations that do not trigger the weak model, where Z=N+M, and Z, M and N are positive integers;

[0010] Each of the Z annotations performed by the target annotation user corresponds to a annotation score, and an instruction score is calculated based on the annotation scores;

[0011] Calculating an instruction diversity score based on the input instruction content of the target labeled user;

[0012] A quality ranking score of the target annotated user is calculated based on the instruction score and the instruction diversity score.

[0013] In some embodiments, the step of obtaining the annotation score of the first type of annotation includes:

[0014] Obtaining an input instruction from the target annotation user, inputting the input instruction into a normal annotation model, and obtaining a normal output result;

[0015] Inputting the input instruction into the weak model to obtain a weak model output result;

[0016] Based on the normal output result and the weak model output result, the target annotation user obtains a determination result;

[0017] A labeling score is determined based on the determination result.

[0018] In some embodiments, determining the annotation score based on the determination result includes:

[0019] If the determination result determines that the output result of the weak model is correct, and the determination result is inconsistent with the result of the normal algorithm, then the corresponding data is erroneous data, and the corresponding annotation score is 0;

[0020] If the determination result determines that the output result of the weak model is wrong, and the determination result is consistent with the result of the normal algorithm, then the corresponding data is correct data, and the corresponding annotation score is 1;

[0021] Among them, the result of the normal algorithm indicates that the output result of the weak model is wrong.

[0022] In some embodiments, calculating the instruction score based on the annotation score includes:

[0023] The instruction score is obtained by adding up the annotation scores corresponding to all annotations in the Z-th annotation.

[0024] In some embodiments, the instruction diversity score is calculated based on the input instruction content of the target annotated user according to the following formula:

[0025] ;

[0026] The instruction diversity score is a score that quantifies the novelty of the input instruction content. Indicates that the score of each input instruction is accumulated; is the embedding matrix used to map each input instruction into a vector space for similarity comparison; is the embedding matrix The transpose of is an identity matrix used to obtain the matrix product to ensure that the diagonal elements do not affect the maximum value; Represented in the matrix Find the maximum value in each row of , where i and j are both integers.

[0027] In some embodiments, calculating the quality ranking score of the target annotated user includes:

[0028] ;

[0029] Among them, weight 1 represents the proportion of the instruction score; weight 2 represents the proportion of the instruction diversity score; the number of times the weak model is preferred represents the number of times the target annotation user in the N first-category annotations believes that the output result of the weak model is correct; the number of times the weak model is drawn represents the number of times the weak model is triggered N.

[0030] In some embodiments, the steps for obtaining the annotation scores corresponding to M normal annotations that do not trigger the weak model are as follows:

[0031] Obtaining an input instruction from the target annotation user, inputting the input instruction into a normal annotation model, and obtaining a normal output result;

[0032] The normal output result is compared with the system algorithm result. If it is consistent with the algorithm result, the normal output result is considered correct and the corresponding annotation score is 1. If it is inconsistent with the algorithm result, the normal output result is considered wrong and the corresponding annotation score is 0.

[0033] In some embodiments, the method further comprises:

[0034] After obtaining the quality ranking score, the annotation quality of the target annotation user is evaluated based on the quality ranking score.

[0035] In a second aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the computer program.

[0036] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the aforementioned method.

[0037] Through the large-model user annotation quality calculation method and electronic device of the present application, the method can ensure that the evaluation results are consistent with human preferences through real-time verification and user ranking mechanism, and realize daily verification results output of the annotation quality in crowdsourcing evaluation, thereby greatly improving the timeliness and accuracy of quality verification, and can promptly discover and correct problems, ensuring the accuracy and efficiency of annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flowchart of a method for calculating the quality of large model user annotations in one embodiment of the present application;

[0040] Figure 2 A system implementation diagram of a large model user annotation quality calculation method in one embodiment of the present application;

[0041] Figure 3 It is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0043] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0044] In the related technologies, the current large-scale model content battle evaluation product (Chatbot Arena) has some shortcomings in terms of operation:

[0045] 1. The large-scale content battle evaluation product (Chatbot Arena) does not rank the users of crowdsourced evaluations, nor does it have a separate manual annotation quality detection mechanism, which may affect the reliability of the evaluation results.

[0046] 2. Currently, the global large-scale model content battle evaluation product (Chatbot Arena) uses a post-hoc mechanism for annotation quality verification. Although this mechanism can detect problems to a certain extent, it cannot perform real-time verification during the evaluation process, and thus cannot correct annotation errors in a timely manner or improve annotation efficiency. This lag affects the real-time and dynamic nature of large-scale model evaluation. Without real-time verification during the evaluation, this may delay the discovery and resolution of problems and affect evaluation efficiency.

[0047] To solve the above problems, this application proposes an innovative solution; this solution can verify the annotation quality in real time during crowdsourcing evaluation and produce verification results in a timely manner. By real-time monitoring and evaluating the annotation quality, the reliability of the evaluation results can be ensured and the evaluation efficiency can be improved.

[0048] This application also supports ranking users who participate in crowdsourcing evaluations to encourage excellent labelers and thus improve the overall labeling quality.

[0049] This application aims to solve the current annotation quality problems in large model evaluation, improve the reliability and efficiency of evaluation results, ensure that evaluation results are consistent with human preferences through real-time verification and user ranking mechanisms, and achieve daily verification results of annotation quality in crowdsourcing evaluations, thereby greatly improving the timeliness and accuracy of quality verification, and being able to discover and correct problems in a timely manner, ensuring that the annotation work is carried out with high quality.

[0050] This application proposes a method for controlling the annotation quality of crowdsourcing evaluation, aiming to improve the accuracy and reliability of data annotation. This method comprehensively evaluates the performance of the annotator and introduces a random verification mechanism to ensure the annotation quality.

[0051] The following is an explanation of the relevant technical terms involved in this application:

[0052] Prompt: The instructions or requests entered by the user when annotating.

[0053] Weak model: refers to data samples that are pre-set in the system and contain known errors or low-quality annotations.

[0054] Embedding matrix (M): In natural language processing, a matrix representation that converts text or instructions into a numeric vector.

[0055] GPU & CuPy: Leverage the Graphics Processing Unit (GPU) and the CuPy library for efficient matrix operations.

[0056] The specific technical solution is as follows:

[0057] Build a crowdsourcing scoring mechanism:

[0058] This application calculates the score based on the number of annotations provided by the annotator, the diversity and complexity of the instructions, and the accuracy of the annotation results.

[0059] Instruction effectiveness: Assessing whether the instructions provided by the annotator are clear and specific will help improve the consistency and accuracy of annotations.

[0060] Instruction richness: measures the diversity and detail of the annotator’s instructions to ensure that a variety of annotation scenarios and needs are covered.

[0061] Number of annotations: Count the number of valid annotations completed by the annotator to reflect his workload and enthusiasm.

[0062] Judging quality: Based on the output results of the two models, the accuracy of the annotator’s judgment on the annotation results is evaluated.

[0063] The scoring algorithm of this application will take into account the complexity of the annotation task and the efficiency and accuracy of the annotator in completing the task.

[0064] Verify model trigger mechanism:

[0065] Verification model setting: During the annotation process, this method randomly triggers a preset "weak model" for verification with an exponentially distributed probability.

[0066] Weak model design:

[0067] A "weak model" is a model with relatively low performance, and its main function is to provide a benchmark to test the judgment of the annotator.

[0068] When the "weak model" is triggered, the weak model will return a result generated based on the user input. In this application, the user's input part in the weak model will be rewritten, so that the output result of the weak model will be significantly different from the normal output result. The weak model will correspond to a preset output and be compared with the final choice of the annotated user (the choice is: the weak model is worse than the normal model or not good at all).

[0069] In this application, weak models will not produce correct results. Therefore, when the annotation user or annotator encounters a weak model, a verification mechanism will be triggered to compare the annotation user's results with the pre-set results in the normal algorithm. If the results are consistent (that is, the normal algorithm determines that the output result of the weak model is wrong, and the annotation user determines that the output result of the weak model is wrong), the result is 1; if the results are inconsistent (that is, the normal algorithm determines that the output result of the weak model is wrong, and the annotation user determines that the output result of the weak model is correct), the result is 0. The comparison result is used as an algorithm verification data score, that is, as a corresponding annotation score, and is calculated as one of the data parameters in the final ranking calculation.

[0070] The trigger probability setting of the weak model can be, for example, that the "weak model" will be activated approximately 5 times out of every 100 valid annotations to balance the system resource usage and verification frequency; the 5 activations here are exemplary, and other feasible times known to technical personnel in this field can also be applied here. The number of activations can be adjusted according to the actual situation, and there is no specific limitation on this.

[0071] Definition of valid annotation: The annotator needs to enter a prompt content and get at least one complete response from any model, and then complete the vote to select the model, which is considered a valid annotation. The annotations in this application are all valid annotations, unless otherwise specified.

[0072] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0073] First, refer to Figure 1 , the large model user annotation quality calculation method of this application is described in detail.

[0074] The present application provides a method for calculating the quality of large model user annotations, the method comprising:

[0075] S1: The target annotation user performs Z annotations, wherein the Z annotations include N first-class annotations that trigger the weak model and M normal annotations that do not trigger the weak model, where Z=N+M, and Z, M and N are positive integers;

[0076] S2: Each of the Z annotations performed by the target annotation user corresponds to a annotation score, and an instruction score is calculated based on the annotation scores;

[0077] S3: Calculating an instruction diversity score based on the input instruction content of the target labeled user;

[0078] S4: Calculate the quality ranking score of the target annotating user based on the instruction score and the instruction diversity score.

[0079] Through the large-model user annotation quality calculation method and electronic device of the present application, the method can ensure that the evaluation results are consistent with human preferences through real-time verification and user ranking mechanism, and realize daily verification results output of the annotation quality in crowdsourcing evaluation, thereby greatly improving the timeliness and accuracy of quality verification, and can promptly discover and correct problems, ensuring the accuracy and efficiency of annotation.

[0080] In some embodiments, the step of obtaining the annotation score of the first type of annotation includes:

[0081] Obtaining an input instruction from the target annotation user, inputting the input instruction into a normal annotation model, and obtaining a normal output result;

[0082] Inputting the input instruction into the weak model to obtain a weak model output result;

[0083] Based on the normal output result and the weak model output result, the target annotation user obtains a determination result;

[0084] A labeling score is determined based on the determination result.

[0085] First, this application will monitor and collect the data generated by the user during the annotation process in real time, including each annotation operation of the annotating user and the annotation results of the weak model data embedded in the system. In order to calculate the annotation quality score, the system needs to record the following information:

[0086] The goal is to mark all the instructions (prompts) that the user has currently contributed;

[0087] The number of times the target labeler draws a “weak model”;

[0088] The number of times the target annotation user annotated the “weak model” better.

[0089] Specifically, examples of weak models include:

[0090] 1. When the target annotation user performs a certain operation (for example, the weak model is triggered when the fifth annotation is performed) and inputs the prompt content, the output result A of the normal model and the output result B of the weak model will be obtained at the same time. The output result B of the weak model is an error result and is different from the normal result (for example, the target annotation user inputs: apples are red, the output result A of the normal model is: apples are red, and the output result B of the weak model is: the weather in Singapore today is minus 10 degrees Celsius).

[0091] 2. When the target annotation user compares result A and result B and selects result B as the correct result (that is, determines that the result of the weak model is correct), the data is incorrect data.

[0092] 3. After recording the data, record the data as an error data, and aggregate the error data with all the annotation data of the target annotation user to obtain the indicator score.

[0093] 4. Based on all the erroneous data and all the data that has been annotated by the target annotator, calculate the corresponding annotation score and indicator score.

[0094] 5. The annotation score of each incorrect data is recorded as 0, and the annotation score of each correct data is recorded as 1.

[0095] 6. Finally, the quality ranking score of the target annotation user is calculated using the number of incorrect data, the number of correct data, and the number of total data as the basic data for the verification data score.

[0096] 7. For example, the target annotation user made 10 annotations in total, of which 3 annotations were correct for the weak model and 7 annotations were wrong for the weak model. Then the scoring data is: correct data: 7 times, wrong data: 3 times, and the total data is 10 times.

[0097] In some embodiments, determining the annotation score based on the determination result includes:

[0098] If the determination result determines that the output result of the weak model is correct, and the determination result is inconsistent with the result of the normal algorithm, then the corresponding data is erroneous data, and the corresponding annotation score is 0;

[0099] If the determination result determines that the output result of the weak model is wrong, and the determination result is consistent with the result of the normal algorithm, then the corresponding data is correct data, and the corresponding annotation score is 1;

[0100] Among them, the result of the normal algorithm indicates that the output result of the weak model is wrong.

[0101] When the target annotation user is faced with the output of the "weak model", if he chooses "the normal model is better (more accurate)" or "both are worse", the choice is recorded as the key verification data. These two results are the correct answers for the algorithm verification and will be included in the subsequent result verification part of this application.

[0102] As described above: If the target annotation user selects "normal model is better (more accurate)" or "both are worse", and it is consistent with the result of the system algorithm (the system algorithm determines that the weak model loses, and the user determines that the weak model loses), the result is 1; if it is inconsistent with the result of the system algorithm (the system algorithm determines that the weak model loses, and the user determines that the weak model wins), the result is 0. This comparison result is used as an algorithm verification data score and is calculated as one of the data parameters in the final ranking calculation.

[0103] This application collects these verification data and all the labeled data, and uses a specially designed algorithm to evaluate the overall performance of the annotators.

[0104] The annotation quality calculation algorithm in this application can analyze the performance of annotators in different situations and adjust their scores and rankings accordingly.

[0105] By implementing the above technical solutions, the system can effectively motivate annotators to provide high-quality annotation data, while ensuring high efficiency and accuracy of annotation through continuous quality monitoring and feedback mechanisms.

[0106] In some embodiments, calculating the instruction score based on the annotation score includes:

[0107] The instruction score is obtained by adding up the annotation scores corresponding to all annotations in the Z-th annotation.

[0108] In some embodiments, the instruction diversity score is calculated based on the input instruction content of the target annotated user according to the following formula:

[0109] ;

[0110] The instruction diversity score is a score that quantifies the novelty of the input instruction content. Indicates that the score of each input instruction is accumulated; is the embedding matrix used to map each input instruction into a vector space for similarity comparison; is the embedding matrix The transpose of is an identity matrix used to obtain the matrix product to ensure that the diagonal elements do not affect the maximum value; Represented in the matrix Find the maximum value in each row of , where i and j are both integers.

[0111] In some embodiments, calculating the quality ranking score of the target annotated user includes:

[0112] ;

[0113] Among them, weight 1 represents the proportion of the instruction score; weight 2 represents the proportion of the instruction diversity score; the number of times the weak model is preferred represents the number of times the target annotation user in the N first-category annotations believes that the output result of the weak model is correct; the number of times the weak model is drawn represents the number of times the weak model is triggered N.

[0114] In some embodiments, the steps for obtaining the annotation scores corresponding to M normal annotations that do not trigger the weak model are as follows:

[0115] Obtaining an input instruction from the target annotation user, inputting the input instruction into a normal annotation model, and obtaining a normal output result;

[0116] The normal output result is compared with the system algorithm result. If it is consistent with the algorithm result, the normal output result is considered correct and the corresponding annotation score is 1. If it is inconsistent with the algorithm result, the normal output result is considered wrong and the corresponding annotation score is 0.

[0117] In some embodiments, the method further comprises:

[0118] After obtaining the quality ranking score, the annotation quality of the target annotation user is evaluated based on the quality ranking score.

[0119] Reference below Figure 2 Another embodiment of the present application is described in detail. The present application also discloses a system, which is mainly composed of the following modules: Data collection module: responsible for real-time monitoring of the user's annotation behavior and collecting relevant data. Data processing module: responsible for summarizing and preliminarily processing the collected data. Score calculation module: calculates the annotation score (0 or 1) and instruction diversity score based on the data provided by the data processing module. Quality ranking module: calculates the final quality ranking score based on the results of the score calculation module.

[0120] The system workflow is as follows Figure 2 As shown:

[0121] 1. After entering the FlagEval platform, the user needs to enter FlagEval-enter the large model arena to mark.

[0122] 2. During the first annotation, verification extraction is triggered based on the annotation user UID to determine whether to extract a normal model or a verification model from the model library. The extraction method is: each user will randomly draw about 5 verification models every 100 annotations. In each annotation, the annotation service will determine whether to call the verification model based on the number of annotations by the annotation user. When the annotation user triggers the annotation extraction probability, a battle between the verification model and the normal model is triggered.

[0123] 3. After the model is extracted and the annotated user performs annotation, the annotation results will be summarized and stored in the data processing module for the validity calculation stage.

[0124] 4. Each verification and annotation result is used as a validity indicator. Through the validity calculation algorithm, the user annotation quality ranking score is obtained, and finally the quality ranking score is displayed.

[0125] This application proposes a highly time-effective annotation quality verification method, which can verify the annotation quality at the daily level in the evaluation annotation process, improving the current crowdsourcing evaluation annotation, which can only use the posterior quality statistics solution, and improves the annotation quality problem in large model evaluation through the following methods:

[0126] 1. Real-time verification of annotation quality: This solution can verify the annotation quality in real time during crowdsourcing evaluation and produce verification results in a timely manner. This real-time monitoring and evaluation mechanism can ensure the reliability of evaluation results and significantly improve evaluation efficiency.

[0127] 2. User ranking and incentive mechanism: This solution supports ranking users who participate in crowdsourcing evaluation, so as to motivate excellent annotators and improve the overall annotation quality. By establishing a ranking mechanism, annotators can be encouraged to pay more attention to annotation quality and efficiency, thus forming a healthy competition environment.

[0128] 3. Improve the timeliness and accuracy of quality verification: By producing verification results every day, this solution greatly improves the timeliness and accuracy of quality verification. This not only enables problems to be discovered and corrected in a timely manner, but also ensures that the labeling work can be carried out continuously with high quality.

[0129] This application scheme effectively solves the current annotation quality problems in large model evaluation through innovative measures such as real-time verification, user ranking and incentive mechanisms, and improving the timeliness and accuracy of quality verification, improves the reliability and efficiency of evaluation results, and ensures that the evaluation results are consistent with human preferences.

[0130] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0131] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also discloses an electronic device;

[0133] Specifically, Figure 3 The hardware structure diagram of the electronic device provided in this embodiment is shown, and the device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other in communication within the device through the bus 450.

[0134] The processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0135] The memory 420 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0136] The input / output interface 430 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0137] The communication interface 440 is used to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (for example, USB, network cable, etc.) or a wireless manner (for example, mobile network, Wi-Fi, Bluetooth, etc.).

[0138] The bus 450 includes a path that transmits information between the various components of the device (eg, the processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 ).

[0139] It should be noted that, although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440 and the bus 450, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0140] The electronic device of the above embodiment is used to implement the corresponding large model user annotation quality calculation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0141] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, one or more embodiments of the present application also provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the large model user annotation quality calculation method as described in any of the above embodiments.

[0142] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0143] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the large model user annotation quality calculation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0144] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0145] In addition, to simplify the description and discussion, and in order not to make one or more embodiments of the present application difficult to understand, the well-known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided figures. In addition, the device may be shown in the form of a block diagram to avoid making one or more embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that one or more embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0146] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0147] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for calculating the quality of large model user annotations, characterized in that: The method comprises: The target annotation user performs Z annotations, including N first-class annotations that trigger the weak model and M normal annotations that do not trigger the weak model, where Z=N+M, and Z, M and N are positive integers; wherein the weak model is a pre-set model containing data samples with known errors or low-quality annotations, and the weak model will not produce correct results; Each of the Z annotations performed by the target annotation user corresponds to a annotation score, and the instruction score is calculated based on the annotation score; the instruction score is calculated based on the annotation score, including: adding the annotation scores corresponding to all the annotations in the Z annotations to obtain the instruction score; According to the following formula, the command diversity score is calculated based on the input command content of the target annotated user: ； The instruction diversity score is a score that quantifies the novelty of the input instruction content. Indicates that the score of each input instruction is accumulated; is the embedding matrix used to map each input instruction into a vector space for similarity comparison; is the embedding matrix The transpose of is an identity matrix used to obtain the matrix product to ensure that the diagonal elements do not affect the maximum value; Represented in the matrix Find the maximum value in each row of , where i and j are both integers; Based on the instruction score and the instruction diversity score, a quality ranking score of the target annotating user is calculated; wherein the calculating the quality ranking score of the target annotating user includes: ； Among them, weight 1 represents the proportion of the instruction score; weight 2 represents the proportion of the instruction diversity score; the number of times the weak model is preferred represents the number of times the target annotation user in the N first-class annotations believes that the output result of the weak model is correct; the number of times the weak model is drawn represents the number of times the weak model is triggered N; The steps for obtaining the annotation score of the first type of annotation include: Obtaining an input instruction from the target annotation user, inputting the input instruction into a normal annotation model, and obtaining a normal output result; Inputting the input instruction into the weak model to obtain a weak model output result; Based on the normal output result and the weak model output result, the target annotation user obtains a determination result; A labeling score is determined based on the determination result.

2. The large model user annotation quality calculation method according to claim 1, characterized in that: The determining of the annotation score based on the determination result includes: If the determination result determines that the output result of the weak model is correct, and the determination result is inconsistent with the result of the normal algorithm, then the corresponding data is erroneous data, and the corresponding annotation score is 0; If the determination result determines that the output result of the weak model is wrong, and the determination result is consistent with the result of the normal algorithm, then the corresponding data is correct data, and the corresponding annotation score is 1; Among them, the result of the normal algorithm indicates that the output result of the weak model is wrong.

3. The large model user annotation quality calculation method according to claim 1, characterized in that: The steps for obtaining the annotation scores corresponding to M normal annotations that do not trigger the weak model are as follows: Obtaining an input instruction from the target annotation user, inputting the input instruction into a normal annotation model, and obtaining a normal output result; The normal output result is compared with the system algorithm result. If it is consistent with the algorithm result, the normal output result is considered correct and the corresponding annotation score is 1. If it is inconsistent with the algorithm result, the normal output result is considered wrong and the corresponding annotation score is 0.

4. The large model user annotation quality calculation method according to claim 3, characterized in that: The method further comprises: After obtaining the quality ranking score, the annotation quality of the target annotation user is evaluated based on the quality ranking score.

5. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 4.

Images