Labeling method and device, equipment and storage medium
By using the response model to generate response data and the user selects data that meets the annotation standards as the annotation data, the problems of low manual annotation efficiency and poor user experience are solved, and efficient and high-quality annotation data generation is achieved.
Patent Information
- Application Number
- CN202311034260.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-16
- Publication Date
- 2025-02-21
AI Technical Summary
In the prior art, manual labeling data is inefficient and user experience is poor, making it difficult to ensure the quality of labeling data.
By receiving the instruction data entered by the user, the response data is generated using at least one response model, and the response data that meets the target annotation criteria is determined as the annotation data in the user selection.
Improves labeling efficiency and user experience, while ensuring the quality of labeling data.
Smart Images
Figure CN119493986A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a labeling method, device, equipment and storage medium. Background Art
[0002] Large language models are the latest advances in deep learning models for human language. The training of large language models is usually divided into two stages: pre-training and fine-tuning. Among them, fine-tuning is to further train large language models (such as GPT) after pre-training to adapt to specific tasks or fields. Since fine-tuning is supervised training, it requires a large amount of labeled data.
[0003] At present, the main method of labeling data is manual labeling. Since manual labeling is usually performed by users according to labeling guidelines and labeling rules, and then the management user performs a quality check on the labeled data, the data quality of the labeled data can be guaranteed.
[0004] However, manual labeling often requires users to manually write instruction data and response data corresponding to the instruction data, resulting in low labeling efficiency and reduced user labeling experience. Summary of the invention
[0005] The embodiments of the present application provide a labeling method, apparatus, device and storage medium, which can improve the labeling efficiency and the user's labeling experience while ensuring the data quality of the labeled data.
[0006] In a first aspect, an embodiment of the present application provides a marking method, the method comprising:
[0007] Receive instruction data input by the user;
[0008] Respond to the instruction data through at least one response model to obtain a plurality of response data corresponding to the instruction data;
[0009] displaying the plurality of response data;
[0010] In response to a target input by the user to determine target response data from among the plurality of response data, the target response data is determined from among the plurality of response data, the target response data being response data determined by the user and meeting a target annotation standard, the target annotation standard being an annotation standard corresponding to a task type of the instruction data;
[0011] The instruction data and the target response data are determined as a labeled data pair.
[0012] In a possible implementation, in response to the user inputting a target to determine the target response data from the plurality of response data, determining the target response data from the plurality of response data includes:
[0013] receiving a first input of the user selecting first response data from the plurality of response data, the first response data being response data determined by the user to meet the target annotation standard;
[0014] In response to the first input, selecting the first response data from among the plurality of response data;
[0015] The first response data is determined as the target response data.
[0016] In a possible implementation, in response to the user inputting a target to determine the target response data from the plurality of response data, determining the target response data from the plurality of response data includes:
[0017] receiving a second input of the user selecting second response data from the plurality of response data, the second response data being response data determined by the user and meeting a portion of the target annotation standard;
[0018] In response to the second input, selecting the second response data from the plurality of response data;
[0019] receiving a third input from the user for modifying the second response data;
[0020] In response to the third input, the second response data is modified to obtain the target response data.
[0021] In a possible implementation, the target annotation standard includes a content annotation standard and a format annotation standard, the content annotation standard includes a content relevance standard between the response data and the instruction data, the format annotation standard includes a response format template, the second response data is the first sub-response data, and receiving a second input of the user selecting the second response data from the plurality of response data includes:
[0022] A second input is received from the user selecting the first sub-answer data from among the plurality of answer data, wherein the first sub-answer data is answer data determined by the user to meet the content relevance standard and the answer format template.
[0023] In a possible implementation, the target annotation standard includes a content annotation standard and a format annotation standard, the content annotation standard includes a content relevance standard between the response data and the instruction data, the format annotation standard includes a response format template, the second response data is second sub-response data, and receiving a second input of the user selecting the second response data from the plurality of response data includes:
[0024] A second input is received from the user selecting the second sub-answer data from among the plurality of answer data, wherein the second sub-answer data is answer data determined by the user to meet the content relevance standard but not meet the answer format template.
[0025] In a possible implementation manner, after determining the instruction data and the target response data as a labeled data pair, the method further includes:
[0026] For a first target task type, i response data and j third sub-response data output by the at least one response model within a first preset period are obtained, wherein the first target task type is any one of a plurality of task types, and the third sub-response data is response data determined by the user and meeting the labeling standard corresponding to the first target task type, and i and j are positive integers;
[0027] Calculate the ratio of j to i to obtain a first ratio of the j third sub-response data to the i response data;
[0028] When the first ratio is less than a first preset threshold, acquiring a plurality of first data pairs corresponding to the first target task type, the plurality of first data pairs including data pairs corresponding to the first target task type among a plurality of labeled data pairs;
[0029] The response model corresponding to the first target task type is iteratively optimized using the multiple first data pairs.
[0030] In a possible implementation manner, after determining the instruction data and the target response data as a labeled data pair, the method further includes:
[0031] Obtain m response data and n fourth sub-response data corresponding to the second target task type output by the target response model within a second preset period, wherein the fourth sub-response data is response data determined by the user and meets the labeling standard corresponding to the second target task type, the target response model is any one of the at least one response model, the second target task type is any one of the at least one task type corresponding to the target response model, and m and n are positive integers;
[0032] Calculate the ratio of n to m to obtain a second ratio of the n fourth sub-response data to the m response data;
[0033] When the second ratio is less than a second preset threshold, acquiring a plurality of second data pairs corresponding to the second target task type, the plurality of second data pairs including data pairs corresponding to the second target task type among a plurality of labeled data pairs;
[0034] The target response model is iteratively optimized using the plurality of second data pairs.
[0035] In a second aspect, an embodiment of the present application provides a labeling device, the device comprising:
[0036] A receiving module, used for receiving instruction data input by a user;
[0037] A response module, used to respond to the instruction data through at least one response model to obtain a plurality of response data corresponding to the instruction data;
[0038] A display module, used for displaying the plurality of response data;
[0039] a first determining module, configured to determine the target response data from among the plurality of response data in response to a target input by the user to determine the target response data from among the plurality of response data, wherein the target response data is response data determined by the user and conforming to a target annotation standard, wherein the target annotation standard is an annotation standard corresponding to a task type of the instruction data;
[0040] The second determination module is used to determine the instruction data and the target response data as a marked data pair.
[0041] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;
[0042] When the processor executes the computer program instructions, the processor implements any possible implementation method of the first aspect described above.
[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements a method in any possible implementation method of the first aspect described above.
[0044] The annotation method, apparatus, device and storage medium of the embodiment of the present application, after receiving the instruction data input by the user, responds to the instruction data through at least one response model, obtains and displays multiple response data corresponding to the instruction data, so that the user can obtain multiple optional response data without manual writing, thereby improving the annotation efficiency and the user's annotation experience. Since the target response data is the response data determined by the user that meets the target annotation standard, and the target annotation standard is the annotation standard corresponding to the task type of the instruction data, therefore, by responding to the target input, determining the target response data among the multiple response data, and determining the instruction data and the target response data as an annotation data pair, the data quality of the annotation data pair can be guaranteed. In this way, through the embodiment of the present application, the annotation efficiency and the user's annotation experience can be improved on the basis of ensuring the data quality of the annotation data. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0046] Figure 1 It is a flowchart of a marking method provided in an embodiment of the present application;
[0047] Figure 2 It is a flowchart of another marking method provided in an embodiment of the present application;
[0048] Figure 3 It is a flowchart of another marking method provided in an embodiment of the present application;
[0049] Figure 4 is a structural schematic diagram of a marking device provided in an embodiment of the present application;
[0050] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0053] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0054] As described in the background technology section, in order to solve the problems of the prior art, the embodiments of the present application provide a labeling method, device, equipment and storage medium. Among them, the labeling method can be applied to a labeling platform.
[0055] Before labeling, the management user can first formulate the labeling task according to the fine-tuning requirements and tasks of the large language model. In an embodiment of the present application, the fine-tuning requirements and tasks may be to enable the large language model to accurately understand the user's instructions and respond based on the understanding. Based on this, the labeling task may be a labeling instruction-response data pair. The labeling task may include multiple task types. Task types may include, for example, encyclopedia tasks, question-and-answer tasks, summary tasks, plan-making tasks, and the like. Among them, summary tasks may, for example, be summarizing a summary from an article. Plan-making tasks may, for example, include formulating work plans, study plans, travel plans, and the like for users.
[0056] After determining the labeling tasks and their corresponding task types, management users can formulate clear and explicit labeling standards for each task type. The labeling standards can include label definitions, labeling rules, and examples to ensure that labelers can understand the labeling standards and perform labeling work in accordance with the labeling standards.
[0057] The following first introduces the marking method provided in the embodiment of the present application.
[0058] Figure 1 FIG. 1 is a flow chart of a marking method provided in an embodiment of the present application. Figure 1 As shown, the marking method provided in the embodiment of the present application includes steps S110 to S150.
[0059] S110, receiving instruction data input by a user.
[0060] Here, the instruction data may be voice data, image data, text data, etc., which are not limited here. If the instruction data is text data, the instruction data may be an instruction statement written by a user according to the annotation standard. The user may be an annotator.
[0061] As an example, the user may first determine the task type, and then write the instruction statement according to the annotation standard corresponding to the task type. Specifically, the user may use the annotation tool to write the instruction statement in the annotation platform. The process of the user writing the instruction statement may be the process of inputting instruction data into the annotation platform. That is, in the process of the user writing the instruction data in the annotation platform, the annotation platform may continue to receive the instruction data input by the user.
[0062] S120, responding to the instruction data through at least one response model to obtain a plurality of response data corresponding to the instruction data.
[0063] Here, the response model can be an open source language model, or a language model trained based on open source instruction-response data pairs, which is not limited here. If the response model is an open source language model, it can be considered that the response model does not correspond to any task type. If the response model is a language model trained based on open source instruction-response data pairs, the response model can correspond to at least one task type. Among them, the language model can be a large language model or a non-large language model. In addition, the annotation platform may include one response model or multiple response models. Among them, if the annotation platform includes one response model, one response model can output multiple response data corresponding to the instruction data. If the annotation platform includes multiple response models, each of the multiple response models can output one response data or multiple response data, which is not limited here.
[0064] In addition, if the annotation platform includes a response model, one response model can correspond to all task types. If the annotation platform includes multiple response models, the number and type of task types corresponding to each response model may be different. For example, the annotation platform may include a first response model, a second response model, and a third response model. Among them, the first response model can correspond to encyclopedia tasks, the second response data model can correspond to question-and-answer tasks, summary tasks, and plan-making tasks, and the third response model can correspond to encyclopedia tasks and plan-making tasks. In addition, since the third response model corresponds to two types of tasks, summary tasks and plan-making tasks, the third response model can be trained together through article-abstract data pairs and plan requirements-plan response data pairs.
[0065] In fact, if the task types corresponding to multiple response models are the same, the differences between the multiple response data may be small, which is not conducive to the user's selection of response data. Therefore, by arranging the number and types of task types corresponding to the response models, the diversity of the multiple response data output by the model can be improved.
[0066] S130, displaying a plurality of response data.
[0067] Here, after the response model outputs a plurality of response data, the plurality of response data may be displayed on the annotation platform. The user may judge the plurality of response data respectively based on the target annotation standard to determine whether each response data meets the target annotation standard.
[0068] S140, in response to the user's target input of determining target response data among multiple response data, target response data is determined among the multiple response data, the target response data is response data determined by the user that meets the target annotation standard, and the target annotation standard is the annotation standard corresponding to the task type of the instruction data.
[0069] Here, the task type corresponding to the response data and the task type corresponding to the instruction data may be the same. The target response data may be one or more, which is not limited here. In addition, the target input may be an operation of the user determining the target response data from multiple response data. The target input may include an operation of the user selecting the target response data from multiple response data, and an operation of the user modifying the response data.
[0070] As an example, if the user determines that the response data fully meets the target annotation standard, the response data can be directly determined as the target response data. If the user determines that the response data meets part of the target annotation standard, the response data can be modified so that the response data fully meets the annotation standard, and the modified response data is determined as the target response data.
[0071] In addition, the target annotation standard (i.e., the annotation standard corresponding to each task type) may include a content annotation standard and a format annotation standard. Among them, the content annotation standard may include a content relevance standard between the response data and the instruction data, and the format annotation standard includes a response format template. Specifically, compliance with the content annotation standard may include that the content relevance between the response data and the instruction data is relatively large, the response data does not include content unrelated to the instruction data, and the response data does not include personal sensitive information and other related information. Compliance with the format annotation standard may include that the response data complies with the format annotation template, and the segmentation and punctuation in the response data are correct. Among them, the format annotation template may include a total-to-specific mode, a sub-total mode, a total-to-specific-total mode, and the like.
[0072] It should be noted that since it is the user who determines whether the response data meets the annotation standard, the judgment result may be subjective. For example, for the same response data, when judging the content relevance between the response data and the instruction data, user A may think that the task content relevance is greater, while user B may think that the content relevance is less. However, since the annotation standard contains guidelines, rules, examples, etc., in most cases, the judgment results of different users for the same response data can be the same.
[0073] Based on this, in order to improve the data quality of the labeled data pair, in some embodiments, the above S140 may specifically include:
[0074] Receiving a first input of a user selecting first response data from a plurality of response data, the first response data being response data determined by the user and meeting a target annotation standard;
[0075] In response to the first input, selecting first response data among the plurality of response data;
[0076] The first response data is determined as the target response data.
[0077] Here, the first response data may be one or more, which is not limited here. In addition, the first input may be an operation in which the user selects the first response data from multiple response data and determines the first response data as the target response data. In addition, the user's determination that the first response data meets the target annotation standard may include that the user believes that the content correlation between the first response data and the instruction data is large, the first response data does not include content unrelated to the instruction data, the first response data does not include personal sensitive information and other relevant information, the first response data complies with the format annotation template, and the segmentation and punctuation in the first response data are correct, etc.
[0078] As an example, if the user determines that the first response data fully meets the target annotation standard, the user can first click the button corresponding to the first response data in the annotation platform, and then click the OK button. After receiving the user clicking the button corresponding to the first response data, the annotation platform can select the first response data. After receiving the user clicking the OK button, the annotation platform can determine the first response data as the target response data.
[0079] In this way, by selecting and confirming the first response data based on user operations, response data that fully complies with the labeling standard can be obtained, thereby improving the labeling quality of the labeling data pair.
[0080] Based on this, in order to improve the data quality of the labeled data pair, in some embodiments, the above S140 may specifically include:
[0081] receiving a second input of a user selecting second response data from a plurality of response data, the second response data being response data determined by the user and meeting a partial target labeling standard;
[0082] In response to a second input, selecting second response data from among the plurality of response data;
[0083] receiving a third input from a user for modifying the second response data;
[0084] In response to the third input, the second response data is modified to obtain the target response data.
[0085] Here, the second input may be an operation in which the user selects the second response data from a plurality of response data. The third input may be an operation in which the user modifies the second response data. In addition, since responding to an instruction is usually responding to the specific content of the instruction, when the user determines whether the response statement meets the annotation criteria, the user may first determine the correlation between the response data and the instruction data. Therefore, the user's determination that the second response data meets some of the target annotation criteria may at least include the user's belief that the content correlation between the second response data and the instruction data is large.
[0086] As an example, if the user has determined that the first response data is not included in the multiple response data, the second response data can continue to be searched in the multiple response data. If the user determines that the second response data meets some of the target annotation standards, the button corresponding to the second response data can be clicked in the annotation platform. After receiving the user's click on the button corresponding to the second response data, the annotation platform can select the second response data. Afterwards, the user can click the Modify button. After receiving the user's click on the Modify button, the annotation platform can display an edit box corresponding to the second response data. The user can modify the second response statement in the edit box. After the modification is completed, you can click the OK button so that the annotation platform will determine the modified second response data as the target response data.
[0087] As another example, after the user selects the second response data, the second response data may be directly copied to an edit box named "annotation reply", and the second response data may be modified in the edit box.
[0088] In this way, by selecting the second response data and modifying the second response statement based on the user operation, the response data that fully complies with the annotation standard can be obtained, thereby improving the annotation quality of the annotation data pair.
[0089] Based on this, if the second response data is the first sub-response data, then in some embodiments, the above-mentioned receiving the second input of the user selecting the second response data from the multiple response data may specifically include:
[0090] A second input is received in which the user selects first sub-answer data from among a plurality of answer data, where the first sub-answer data is answer data determined by the user to meet the content relevance standard and the answer format template.
[0091] Here, if the user has determined that the first response data is not included in the multiple response data, the first sub-response data can be continued to be searched in the multiple response data. That is, if the user has determined that all response data do not fully comply with the labeling standard, the first sub-response data that has a greater content correlation between the response data and the instruction data and complies with the response format template can be searched in the response data. Afterwards, the user can modify the first sub-response data. Specifically, if the first sub-response includes sensitive information, other related information, etc. in the data, the sensitive information and other related information can be deleted. If the content in the first sub-response data is not segmented, the content can be segmented to obtain response data that fully complies with the labeling standard.
[0092] Based on this, if the second response data is the second sub-response data, then in some embodiments, the above-mentioned receiving the second input of the user selecting the second response data from the multiple response data may specifically include:
[0093] A second input is received in which the user selects second sub-answer data from among the plurality of answer data, where the second sub-answer data is answer data determined by the user to meet the content relevance standard but not meet the answer format template.
[0094] Here, if the user has determined that the first sub-response data is not included in the multiple response data, the second sub-response data can be continued to be searched in the multiple response data. That is, the user has determined that all the response data do not conform to the response format template. Then, among the response data with greater content relevance, the one with the least errors can be selected to obtain the second response data. If the response data with the least errors is selected, the user's subsequent modification workload can be reduced. Among them, the least errors can be, for example, that although the response statement does not conform to the response format template, there is no sensitive information, other related information, etc., and the punctuation is correct. Of course, the determination of the number of errors is related to the subjectivity of the user, which will not be elaborated here.
[0095] S150, determining the instruction data and the target response data as a marked data pair.
[0096] Here, if the target response data is one, one labeled data pair can be obtained. If the target response data is multiple, multiple labeled data pairs can be obtained.
[0097] The annotation method of the embodiment of the present application responds to the instruction data through at least one response model after receiving the instruction data input by the user, obtains and displays multiple response data corresponding to the instruction data, so that the user can obtain multiple optional response data without manual writing, thereby improving the annotation efficiency. Since the target response data is the response data determined by the user that meets the target annotation standard, and the target annotation standard is the annotation standard corresponding to the task type of the instruction data, therefore, by responding to the target input, determining the target response data among the multiple response data, and determining the instruction data and the target response data as an annotation data pair, the data quality of the annotation data pair can be guaranteed. In this way, through the embodiment of the present application, the annotation efficiency can be improved on the basis of ensuring the data quality of the annotation data.
[0098] See also Figure 2 The marking method provided in the embodiment of the present application may further include, after S150 shown in the above embodiment:
[0099] S210: Store the labeled data pairs to obtain a plurality of labeled data pairs after storage.
[0100] After obtaining the annotated data pairs, the annotated data pairs can be stored in a database or file for subsequent use. Specifically, the stored multiple annotated data pairs can be used to fine-tune the large language model after pre-training, and can also be used for data exploration and analysis to iteratively optimize the response model in the embodiment of the present application according to the analysis results.
[0101] Therefore, by storing the labeled data pairs and using the stored multiple labeled data to iteratively optimize the response model, the accuracy of the response data can be continuously improved, thereby improving the labeling efficiency and enhancing the user's labeling experience.
[0102] In addition, the other steps of the method in the embodiment of the present application can be seen above Figure 1 The relevant description of the illustrated embodiment will not be repeated here.
[0103] Based on this, in order to continuously improve the accuracy of the response data corresponding to the first target task type output by the response model, in some embodiments, after the above S150 or S210, the following may also be included:
[0104] For a first target task type, i response data and j third sub-response data output by at least one response model within a first preset period are obtained, the first target task type is any one of a plurality of task types, the third sub-response data is response data determined by a user and meeting a labeling standard corresponding to the first target task type, and i and j are positive integers;
[0105] Calculate the ratio of j to i to obtain a first ratio of j third sub-response data to i response data;
[0106] When the first ratio is less than a first preset threshold, acquiring a plurality of first data pairs corresponding to a first target task type, the plurality of first data pairs including data pairs corresponding to the first target task type among the plurality of labeled data pairs;
[0107] A response model corresponding to the first target task type is iteratively optimized using the plurality of first data pairs.
[0108] Here, the first preset period may be a statistical period for integrating and analyzing the data. The first preset period may be, for example, 1 day, 2 days, 3 days, etc. The first preset threshold may be a preset ratio threshold. If the first ratio is less than the first preset threshold, the response model has a weak response capability to the first target task type. If the first ratio is greater than the first preset threshold, the response model has a strong response capability to the first target task type. In addition, in addition to including data pairs corresponding to the first target task type in multiple annotated data pairs, the multiple first data pairs may also include open source data pairs corresponding to the first target task type, and user-written data pairs corresponding to the first target task type.
[0109] As an example, the annotation platform can save the first response data and the second response data determined by the user for each output of multiple response data. Among them, the first response data may include third sub-response data. At every first preset period, the annotation platform can perform statistical analysis on the saved response data, and calculate the first proportion of response data output from the response model that meets the annotation standard for each task type. If the first proportion is small, that is, the response model has a weak response capability for this task type, multiple first data pairs can be obtained to iteratively optimize the response model. In order to continue to differentiate between the multiple response data output by the response model, only the response model corresponding to the first target task type can be iteratively optimized.
[0110] In addition, when the response model has a strong response capability for the task type, it is not necessary to iteratively optimize the response model.
[0111] In this way, by iteratively optimizing the response model corresponding to the first target task type using multiple first data pairs every first preset period when the first ratio is less than the first preset threshold, the accuracy of the response data corresponding to the first target task type output by the response model can be continuously improved.
[0112] Based on this, in order to continuously improve the accuracy of the response data output by the target response model, in some embodiments, after the above S150 or S210, the following steps may also be included:
[0113] Obtain m response data and n fourth sub-response data corresponding to the second target task type output by the target response model within the second preset period, where the fourth sub-response data is response data determined by the user and meets the labeling standard corresponding to the second target task type, the target response model is any one of the at least one response model, the second target task type is any one of the at least one task type corresponding to the target response model, and m and n are positive integers;
[0114] Calculate the ratio of n to m to obtain a second ratio of the n fourth sub-response data to the m response data;
[0115] When the second ratio is less than a second preset threshold, acquiring a plurality of second data pairs corresponding to a second target task type, the plurality of second data pairs including data pairs corresponding to the second target task type among the plurality of labeled data pairs;
[0116] The target response model is iteratively optimized using the plurality of second data pairs.
[0117] Here, the second preset period may be a statistical period for integrating and analyzing the data. The second preset period may be, for example, 1 day, 2 days, 3 days, and the like. The second preset period may be the same as or different from the first preset period, and is not limited here. In addition, the second preset threshold may be a preset ratio threshold. If the second ratio is less than the second preset threshold, the target response model has a weaker response capability to the second target task type. If the second ratio is greater than the second preset threshold, the target response model has a stronger response capability to the second target task type. The second preset threshold and the first preset threshold may be the same or different, and are not limited here. In addition, in addition to including data pairs corresponding to the second target task type in multiple annotated data pairs, the multiple second data pairs may also include open source data pairs corresponding to the second target task type, and data pairs written by users corresponding to the second target task type.
[0118] As an example, the annotation platform can save the first response data and the second response data determined by the user for each output of multiple response data. Among them, the first response data may include fourth sub-response data. Every second preset period, the annotation platform can perform statistical analysis on the saved response data. Specifically, for each task type corresponding to each response model, the annotation platform can calculate the second proportion of response data output from the response model that meets the annotation standard. If the second proportion is small, that is, the response model has a weak response capability for this task type, multiple second data pairs can be obtained to iteratively optimize the response model.
[0119] In addition, if the target response model has a strong response capability to a certain task type it corresponds to, it is not necessary to use a plurality of second data pairs corresponding to the task type to iteratively optimize the target response model.
[0120] In this way, by iteratively optimizing the target response model using multiple second data pairs every second preset period when the second ratio is less than the second preset threshold, the accuracy of the response data output by the target response model can be continuously improved.
[0121] In addition, if the user determines that multiple response data are not related to the content of a certain instruction data, multiple data pairs corresponding to the task type of the instruction data can be re-generated, and the response model can be iteratively optimized using the multiple data pairs so that the response model can output response data related to the content of the instruction data. The multiple data pairs can be open source data pairs or written by the user.
[0122] In order to better describe the entire solution, some specific examples are given based on the above embodiments.
[0123] For example, Figure 3 As shown, the marking method in the embodiment of the present application may include the following steps:
[0124] S31, the annotation platform receives instruction data input by the user;
[0125] S32, responding to the instruction data through at least one response model, obtaining and displaying a plurality of response data;
[0126] S33, the user determines whether the response data fully meets the target labeling standard according to the target labeling standard, if yes, execute S37, if no, execute S34;
[0127] S34, the user determines whether the response data meets both the content relevance standard and the response format template according to the target annotation standard. If so, execute S36; if not, execute S35;
[0128] S35, the user determines whether the response data meets the content relevance standard but does not meet the response format template according to the target annotation standard. If so, execute S36, if not, execute S38;
[0129] S36, modifying the response data;
[0130] S37, storing the labeled data pair including the instruction data and the response data, and returning to execute S31;
[0131] S38, integrating and analyzing the data at every preset period;
[0132] S39. Iteratively optimize the response model based on the results of integration and analysis.
[0133] Therefore, through the embodiments of the present application, the annotation efficiency and user annotation experience can be improved while ensuring the data quality of the annotation data. In addition, by integrating and analyzing the data, and iteratively optimizing the response model based on the results of the integration and analysis, the accuracy of the response data output by the response model can be continuously improved, further improving the annotation efficiency.
[0134] In addition, if a response model is a large language model, and the response model corresponds to all task types in the labeling task, the response model after iterative optimization can be directly used as the large language model after fine-tuning.
[0135] Based on the marking method provided in the above embodiment, the present application also provides a specific implementation of the marking device. Please refer to the following embodiment.
[0136] like Figure 4 As shown, the labeling device 400 provided in the embodiment of the present application includes the following modules:
[0137] The receiving module 410 is used to receive instruction data input by the user;
[0138] A response module 420, configured to respond to the instruction data through at least one response model to obtain a plurality of response data corresponding to the instruction data;
[0139] Display module 430, used to display multiple response data;
[0140] A first determining module 440 is configured to determine target response data from among the plurality of response data in response to a target input by the user to determine target response data from among the plurality of response data, wherein the target response data is response data determined by the user and meets a target annotation standard, and the target annotation standard is an annotation standard corresponding to the task type of the instruction data;
[0141] The second determination module 450 is used to determine the instruction data and the target response data as a marked data pair.
[0142] The above-mentioned marking device 400 is described in detail below, as follows:
[0143] In some embodiments, the first determining module 440 may specifically include:
[0144] A first receiving submodule is used to receive a first input from a user selecting first response data from a plurality of response data, where the first response data is response data determined by the user and meets a target annotation standard;
[0145] A first selection submodule, configured to select first response data from a plurality of response data in response to a first input;
[0146] The first determining submodule is used to determine the first response data as target response data.
[0147] In some embodiments, the first determining module 440 may further include:
[0148] A second receiving submodule is used to receive a second input from a user selecting second response data from a plurality of response data, where the second response data is response data determined by the user and conforming to a partial target labeling standard;
[0149] A second selection submodule, configured to select second response data from the plurality of response data in response to a second input;
[0150] A third receiving submodule, used for receiving a third input from a user for modifying the second response data;
[0151] The modification submodule is used to modify the second response data in response to the third input to obtain target response data.
[0152] In some embodiments, the target annotation standard includes a content annotation standard and a format annotation standard, the content annotation standard includes a content correlation standard between the response data and the instruction data, the format annotation standard includes a response format template, and the second response data is the first sub-response data.
[0153] Based on this, the second receiving submodule may specifically include:
[0154] The first receiving unit is used to receive a second input of a user selecting first sub-response data from multiple response data, where the first sub-response data is response data determined by the user and meets the content relevance standard and the response format template.
[0155] In some of these embodiments, the second response data is second sub-response data.
[0156] Based on this, the second receiving submodule may further include:
[0157] The second receiving unit is used to receive a second input from a user selecting second sub-response data from multiple response data, where the second sub-response data is response data determined by the user that meets the content relevance standard but does not meet the response format template.
[0158] In some embodiments, the marking device 400 may further include:
[0159] A first acquisition module is used to acquire, for a first target task type, i pieces of response data and j pieces of third sub-response data output by at least one response model within a first preset period, wherein the first target task type is any one of a plurality of task types, and the third sub-response data is response data determined by a user and meeting a labeling standard corresponding to the first target task type, and i and j are positive integers;
[0160] A first calculation module is used to calculate the ratio of j to i to obtain a first ratio of j third sub-response data to i response data;
[0161] A second acquisition module is used to acquire a plurality of first data pairs corresponding to a first target task type when the first ratio is less than a first preset threshold, the plurality of first data pairs including data pairs corresponding to the first target task type among the plurality of labeled data pairs;
[0162] The first iteration module is used to iteratively optimize the response model corresponding to the first target task type using multiple first data pairs.
[0163] In some embodiments, the marking device 400 may further include:
[0164] A third acquisition module is used to obtain m response data and n fourth sub-response data corresponding to the second target task type output by the target response model within a second preset period, where the fourth sub-response data is response data determined by a user and meets the labeling standard corresponding to the second target task type, the target response model is any one of the at least one response model, the second target task type is any one of the at least one task type corresponding to the target response model, and m and n are positive integers;
[0165] A second calculation module is used to calculate the ratio of n to m to obtain a second ratio of the n fourth sub-response data to the m response data;
[0166] A fourth acquisition module, configured to acquire, when the second ratio is less than a second preset threshold, a plurality of second data pairs corresponding to a second target task type, the plurality of second data pairs including data pairs corresponding to the second target task type among the plurality of labeled data pairs;
[0167] The second iteration module is used to iteratively optimize the target response model using multiple second data pairs.
[0168] The annotation device of the embodiment of the present application responds to the instruction data through at least one response model after receiving the instruction data input by the user, obtains and displays multiple response data corresponding to the instruction data, so that the user can obtain multiple optional response data without manual writing, thereby improving the annotation efficiency. Since the target response data is the response data determined by the user that meets the target annotation standard, and the target annotation standard is the annotation standard corresponding to the task type of the instruction data, therefore, by responding to the target input, determining the target response data among the multiple response data, and determining the instruction data and the target response data as an annotation data pair, the data quality of the annotation data pair can be guaranteed. In this way, through the embodiment of the present application, the annotation efficiency can be improved on the basis of ensuring the data quality of the annotation data.
[0169] Based on the marking method provided in the above embodiment, the embodiment of the present application also provides a specific implementation of the electronic device. Figure 5 A schematic diagram of an electronic device 500 provided in an embodiment of the present application is shown.
[0170] The electronic device 500 may include a processor 510 and a memory 520 storing computer program instructions.
[0171] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0172] The memory 520 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 520 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 520 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 520 is a non-volatile solid-state memory.
[0173] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present application.
[0174] The processor 510 implements any one of the marking methods in the above embodiments by reading and executing computer program instructions stored in the memory 520 .
[0175] In one example, the electronic device 500 may further include a communication interface 530 and a bus 540. Figure 5 As shown, the processor 510, the memory 520, and the communication interface 530 are connected via a bus 540 and communicate with each other.
[0176] The communication interface 530 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0177] Bus 540 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 540 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
[0178] Exemplarily, the electronic device 500 may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).
[0179] The electronic device can execute the marking method in the embodiment of the present application, thereby realizing the combination Figures 1 to 4 Described is a marking method and apparatus.
[0180] In addition, in combination with the marking method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the marking methods in the above embodiment is implemented.
[0181] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0182] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0183] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0184] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.
[0185] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A marking method, characterized in that: include: Receive instruction data input by the user; Respond to the instruction data through at least one response model to obtain a plurality of response data corresponding to the instruction data; displaying the plurality of response data; In response to a target input by the user to determine target response data from among the plurality of response data, the target response data is determined from among the plurality of response data, the target response data being response data determined by the user and meeting a target annotation standard, the target annotation standard being an annotation standard corresponding to a task type of the instruction data; The instruction data and the target response data are determined as a labeled data pair.
2. The method according to claim 1, characterized in that In response to the user inputting a target to determine the target response data from among the plurality of response data, determining the target response data from among the plurality of response data comprises: receiving a first input of the user selecting first response data from the plurality of response data, the first response data being response data determined by the user to meet the target annotation standard; In response to the first input, selecting the first response data from among the plurality of response data; The first response data is determined as the target response data.
3. The method according to claim 1, characterized in that In response to the user inputting a target to determine the target response data from among the plurality of response data, determining the target response data from among the plurality of response data comprises: receiving a second input of the user selecting second response data from the plurality of response data, the second response data being response data determined by the user and meeting a portion of the target annotation standard; In response to the second input, selecting the second response data from the plurality of response data; receiving a third input from the user for modifying the second response data; In response to the third input, the second response data is modified to obtain the target response data.
4. The method according to claim 3, characterized in that The target annotation standard includes a content annotation standard and a format annotation standard, the content annotation standard includes a content correlation standard between the response data and the instruction data, the format annotation standard includes a response format template, the second response data is the first sub-response data, and receiving a second input of the user selecting the second response data from the plurality of response data includes: A second input is received from the user selecting the first sub-answer data from among the plurality of answer data, wherein the first sub-answer data is answer data determined by the user to meet the content relevance standard and the answer format template.
5. The method according to claim 3, characterized in that: The target annotation standard includes a content annotation standard and a format annotation standard, the content annotation standard includes a content correlation standard between the response data and the instruction data, the format annotation standard includes a response format template, the second response data is a second sub-response data, and receiving a second input of the user selecting the second response data from the plurality of response data includes: A second input is received from the user selecting the second sub-answer data from among the plurality of answer data, wherein the second sub-answer data is answer data determined by the user to meet the content relevance standard but not meet the answer format template.
6. The method according to claim 1, characterized in that After determining the instruction data and the target response data as a labeled data pair, the method further includes: For a first target task type, i response data and j third sub-response data output by the at least one response model within a first preset period are obtained, wherein the first target task type is any one of a plurality of task types, and the third sub-response data is response data determined by the user and meeting the labeling standard corresponding to the first target task type, and i and j are positive integers; Calculate the ratio of j to i to obtain a first ratio of the j third sub-response data to the i response data; When the first ratio is less than a first preset threshold, acquiring a plurality of first data pairs corresponding to the first target task type, the plurality of first data pairs including data pairs corresponding to the first target task type among a plurality of labeled data pairs; The response model corresponding to the first target task type is iteratively optimized using the multiple first data pairs.
7. The method according to claim 1, characterized in that After determining the instruction data and the target response data as a labeled data pair, the method further includes: Obtain m response data and n fourth sub-response data corresponding to the second target task type output by the target response model within a second preset period, wherein the fourth sub-response data is response data determined by the user and meets the labeling standard corresponding to the second target task type, the target response model is any one of the at least one response model, the second target task type is any one of the at least one task type corresponding to the target response model, and m and n are positive integers; Calculate the ratio of n to m to obtain a second ratio of the n fourth sub-response data to the m response data; When the second ratio is less than a second preset threshold, acquiring a plurality of second data pairs corresponding to the second target task type, the plurality of second data pairs including data pairs corresponding to the second target task type among a plurality of labeled data pairs; The target response model is iteratively optimized using the plurality of second data pairs.
8. A marking device, characterized in that: The device comprises: A receiving module, used for receiving instruction data input by a user; A response module, used to respond to the instruction data through at least one response model to obtain a plurality of response data corresponding to the instruction data; A display module, used for displaying the plurality of response data; a first determining module, configured to determine the target response data from among the plurality of response data in response to a target input by the user to determine the target response data from among the plurality of response data, wherein the target response data is response data determined by the user and conforming to a target annotation standard, wherein the target annotation standard is an annotation standard corresponding to a task type of the instruction data; The second determination module is used to determine the instruction data and the target response data as a marked data pair.
9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the marking method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the marking method according to any one of claims 1 to 7 is implemented.