Data crowdsourcing annotation distribution method and device, electronic equipment and storage medium
By distributing and receiving annotation results, performing truth inference and interval estimation, the problem of additional costs caused by redundant annotations in data crowdsourcing annotation is solved, and the number of annotations is reduced while ensuring quality, thus saving costs.
Patent Information
- Application Number
- CN202210834624.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-14
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2042-07-14
AI Technical Summary
The additional costs caused by redundant annotations in existing crowdsourced annotation, especially under a specified degree of annotation redundancy, result in too many redundant annotations for each question, affecting the availability of the dataset and the annotation cost.
By distributing annotation tasks and receiving current annotation results, updating the annotation result set, performing truth inference and interval estimation, and deciding whether to stop distributing tasks based on the interval range of label probabilities, redundant annotations are reduced.
While ensuring the quality of data crowdsourcing, reduce the number of annotations and save annotation costs.
Smart Images
Figure CN115270940B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data crowdsourcing annotation, and in particular to a data crowdsourcing annotation distribution method and device, electronic equipment and a storage medium. BACKGROUND
[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission that the prior art is prior art nor does it constitute an admission of any description in this section as prior art to an application described herein and / or in another application also owned by the applicant of the present application.
[0003] In model training of artificial intelligence, data annotation demand is often large and tedious, and is usually realized by crowdsourcing annotation of non-expert personnel. This annotation method improves the difficulty of low efficiency of obtaining labeled data, but the unpredictable annotation accuracy greatly affects the usability of the data set. In view of this, data crowdsourcing introduces redundant annotation, each sample is annotated by multiple annotators, and a true value inference algorithm is used to improve the quality of the final output label. However, in actual deployment, data crowdsourcing usually uses a specified annotation redundancy, that is, a fixed number of annotators are used to carry out annotation for each question. This process is prone to cause excessive redundant annotation for each question, resulting in additional cost. SUMMARY
[0004] Therefore, the purpose of the present application is to provide a data crowdsourcing annotation distribution method, device, electronic equipment and storage medium.
[0005] In order to achieve the above purpose, the present application provides a data crowdsourcing annotation distribution method, which comprises:
[0006] distributing a question title and receiving a current annotation result of the question title, updating a set of annotation results of the question title based on the current annotation result;
[0007] performing true value inference on the updated set of annotation results to obtain a current probability corresponding to a label of the question title;
[0008] interval estimation of the probability of the label based on the current probability to obtain an interval range of the probability of the label;
[0009] in response to determining that the interval ranges of the probabilities of all labels of the question title are not greater than a preset convergence threshold, stopping distribution of the question title.
[0010] In some embodiments, interval estimation of the probability of the label based on the current probability specifically comprises:
[0011] obtaining a plurality of historical probabilities of the label; wherein each label corresponds to a historical probability each time true value inference is performed before the current probability is obtained;
[0012] interval estimation is performed on the probability of the label based on the current probability and the plurality of historical probabilities.
[0013] In some embodiments, the interval estimation is performed on the probability of the label based on the current probability and the plurality of historical probabilities, specifically comprising:
[0014] determining the mean value of the current probability and the plurality of historical probabilities, and the variance of the current probability and the plurality of historical probabilities;
[0015] interval estimation is performed on the probability of the label based on the variance and the mean value.
[0016] In some embodiments, the plurality of historical probabilities of the label is obtained, specifically comprising:
[0017] determining the number of all historical probabilities of the label;
[0018] in response to determining that the number of all historical probabilities is greater than a preset number, obtaining a preset number of historical probabilities in the order from the last to the first according to the time sequence of performing the true value inference, starting from the historical probability closest to the current probability among the all historical probabilities.
[0019] In some embodiments, the plurality of historical probabilities of the label is obtained, specifically further comprising:
[0020] in response to determining that the number of all historical probabilities is not greater than a preset number, obtaining the all historical probabilities.
[0021] In some embodiments, after stopping distributing the target task, the method further comprises:
[0022] determining a target label with the largest probability from the probabilities of all labels;
[0023] taking the target label as the annotation label of the target task.
[0024] In some embodiments, after updating the annotation result set of the target task based on the current annotation result, the method further comprises:
[0025] determining whether the number of annotation results in the annotation result set is less than a preset threshold;
[0026] in response to determining that the number of annotation results in the annotation result set is less than a preset threshold, no longer performing true value inference on the updated annotation result set.
[0027] In some embodiments, distributing the target task specifically comprises:
[0028] creating an annotation task; wherein the annotation task at least includes the target task.
[0029] distributing the annotation task.
[0030] Based on the same inventive concept, the exemplary embodiments of the present application further provide a data crowdsourcing annotation distribution device, comprising:
[0031] a distribution recovery module configured to distribute a target topic and receive a current annotation result of the target topic, update an annotation result set of the target topic based on the current annotation result;
[0032] an inference module configured to perform ground truth inference on the updated annotation result set to obtain a current probability of a label corresponding to the target topic;
[0033] an interval estimation module configured to perform interval estimation on a probability of the label based on the current probability to obtain an interval range of the probability of the label;
[0034] a control module configured to, in response to determining that the interval range of the probability of all labels of the target topic is not greater than a preset convergence threshold, stop distributing the target topic.
[0035] In some embodiments, the interval estimation module is specifically configured to:
[0036] obtain a plurality of historical probabilities of the label; wherein each label corresponds to one historical probability each time ground truth inference is performed before the current probability is obtained;
[0037] perform interval estimation on the probability of the label based on the current probability and the plurality of historical probabilities.
[0038] In some embodiments, the interval estimation module is specifically configured to:
[0039] determine a mean value of the current probability and the plurality of historical probabilities, and a variance of the current probability and the plurality of historical probabilities;
[0040] perform interval estimation on the probability of the label based on the variance and the mean value.
[0041] In some embodiments, the interval estimation module is specifically configured to:
[0042] determine a number of all historical probabilities of each label;
[0043] in response to determining that the number of all historical probabilities is greater than a preset number, obtain a preset number of historical probabilities in a time sequence from back to front in which ground truth inference is performed, starting from a historical probability closest to the current probability among the all historical probabilities.
[0044] In some embodiments, the interval estimation module is specifically configured to:
[0045] In response to determining that the number of all the historical probabilities is not greater than the preset number, obtaining the all the historical probabilities.
[0046] In some embodiments, the apparatus further comprises a determination module configured to:
[0047] determine whether the number of the labeled results in the labeled result set is less than a preset threshold;
[0048] In response to determining that the number of the labeled results in the labeled result set is less than the preset threshold, no longer performing ground truth inference on the updated labeled result set.
[0049] In some embodiments, the distribution recovery module is specifically configured to:
[0050] creating a labeling task; wherein the labeling task at least comprises the objective;
[0051] distributing the labeling task.
[0052] Based on the same inventive concept, the example embodiments of the present application further provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the data crowdsourcing labeling distribution method as described above when executing the program.
[0053] Based on the same inventive concept, the example embodiments of the present application further provide a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the data crowdsourcing labeling distribution method as described above.
[0054] As can be seen from the above, the data crowdsourcing labeling distribution method, apparatus, electronic device and storage medium provided by the present application distribute an objective and receive a current labeled result of the objective, update a labeled result set of the objective based on the current labeled result; perform ground truth inference on the updated labeled result set to obtain a current probability of a label corresponding to the objective; perform interval estimation on a probability of the label based on the current probability to obtain an interval range of the probability of the label; in response to determining that the interval ranges of the probabilities of all the labels of the objective are all not greater than a preset convergence threshold, stop distributing the objective, thereby reducing the number of labeled objectives under the premise of ensuring the quality of data crowdsourcing, and further saving labeling cost. BRIEF DESCRIPTION OF DRAWINGS
[0055] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description only are the embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without any creative effort.
[0056] Figure 1 A flowchart of a data crowdsourcing annotation distribution method in the prior art;
[0057] Figure 2 A flowchart of a data crowdsourcing annotation distribution method in an embodiment of the application;
[0058] Figure 3 A flowchart of another data crowdsourcing annotation distribution method in an embodiment of the application;
[0059] Figure 4 A structural diagram of a data crowdsourcing annotation distribution device in an embodiment of the application;
[0060] Figure 5 A structural diagram of a specific electronic device in an embodiment of the application. DETAILED DESCRIPTION
[0061] The principles and spirits of the application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the application, and do not limit the scope of the application in any way. On the contrary, these embodiments are provided to make the application more thorough and complete, and to fully convey the scope of the application to those skilled in the art.
[0062] According to the embodiments of the application, a data crowdsourcing annotation distribution method, device, electronic device and storage medium are provided.
[0063] In this document, it should be understood that any number of elements in the drawings are used for illustration only, not for limitation, and any naming is only for distinction, not for any limiting meaning.
[0064] The principles and spirits of the application will be described below with reference to several exemplary embodiments. SUMMARY
[0066] In the prior art, reference is made to Figure 1When the labeling task of the to-be-labeled data is initiated, based on the redundant labeling quantity preset by the administrator, the question is distributed to a fixed number of labelers and the label results are collected, based on the label data, the true value inference algorithm carries out an inference, and the final data label is output. For example, when there are n labeling task requirements, the data crowdsourcing platform sends each labeling task to m labelers for labeling based on the preset redundant quantity m, and each question is recycled m label results. However, in the prior art, the redundant labeling quantity is generally set according to the experience of the administrator, that is, how many times each to-be-labeled question is distributed is set in advance by the administrator. Since, in order to ensure the effect of data crowdsourcing true value inference as much as possible, the redundant labeling quantity is often set to be large, and then when unnecessary more labels are introduced, the labeling task continues to be distributed to other labelers. At the same time, each labeler needs to be given a corresponding reward for each task. Therefore, the unreasonable adjustment of the labeling quantity directly leads to the increase of the labeling task cost, which affects the original intention of the data crowdsourcing to obtain data labels at low cost and quickly.
[0067] To solve the above problems, the present application provides a data crowdsourcing labeling distribution method, specifically comprising:
[0068] Distribute the question title and receive the current labeling result of the question title, update the labeling result set of the question title based on the current labeling result, perform true value inference on the updated labeling result set to obtain the current probability of the label corresponding to the question title, estimate the interval range of the probability of the label based on the current probability, and obtain the interval range of the probability of the label. In response to determining that the interval range of the probability of all labels of the question title is not greater than a preset convergence threshold, stop distributing the question title, thereby reducing the labeling quantity of the question title under the premise of ensuring the quality of data crowdsourcing, and saving the labeling cost.
[0069] After introducing the basic principles of the present application, the various non-limiting embodiments of the present application will be specifically introduced.
[0070] Overview of application scenarios
[0071] In some specific application scenarios, the data crowdsourcing labeling distribution method of the present application can be applied to various data crowdsourcing labeling distribution systems, and optionally, the system can be a data crowdsourcing platform.
[0072] In some specific application scenarios, the data crowdsourcing labeling distribution method of the present application can be directly applied locally or run in a cloud server. When running in a cloud server, the obtained to-be-processed data is sent to the cloud server through a network, and the to-be-processed data is processed by the server through the data crowdsourcing labeling distribution method of the present application, and the processing result is sent to the local through the network.
[0073] The data crowdsourcing labeling distribution method according to the exemplary embodiments of the present application will be described below in combination with specific application scenarios. It should be noted that the above-mentioned application scenarios are only shown for the purpose of facilitating understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.
[0074] Exemplary method
[0075] Reference Figure 2 The embodiments of the present application provide a data crowdsourcing labeling distribution method, comprising the following steps:
[0076] S101, distribute a target topic and receive a current labeling result of the target topic, and update a labeling result set of the target topic based on the current labeling result.
[0077] In specific implementation, the to-be-labeled target topic is first distributed, and after the target topic is labeled by a labeler, the current labeling result of the target topic is received, and the labeling result set of the target topic is updated according to the current labeling result. The target labeling result set saves a plurality of labeling results of the target topic, and each time a labeling result of the target topic is newly received, the target topic is saved into the target labeling result set, thereby completing the update of the labeling result set.
[0078] In some embodiments, distributing a target topic specifically comprises:
[0079] Creating a labeling task; wherein the labeling task at least includes the target topic;
[0080] Distributing the labeling task.
[0081] In specific implementation, the target topic can be distributed together with the created labeling task. Optionally, in addition to the target topic, the labeling task can also include other to-be-labeled topics.
[0082] S102, performing truth value inference on the updated labeling result set to obtain a current probability of a label corresponding to the target topic.
[0083] In actual implementation, after the set of labeling results is updated, the updated set of labeling results is subjected to true value inference to obtain the current probability of the label corresponding to the objective. The objective generally has multiple labels, and each label corresponds to an answer. For example, an objective of judging right or wrong has two labels, a determination label and a negative label. After the true value inference is completed, the respective probabilities of the determination label and the negative label can be obtained. Optionally, the specific method of true value inference is not limited herein, and a suitable true value inference method can be selected from the prior art as needed.
[0084] In S103, the probability of the label is subjected to interval estimation based on the current probability to obtain the interval range of the probability of the label.
[0085] In actual implementation, after the current probability of the label corresponding to the objective is obtained, the probability of the label is subjected to interval estimation according to the current probability of the label to obtain the interval range of the probability of the label. Optionally, the probability of the label can be subjected to interval estimation, and then the interval range of the probability of each label can be obtained according to the upper and lower bounds of the probability interval.
[0086] In some embodiments, the probability of each label is subjected to interval estimation based on the current probability, and specifically includes:
[0087] A plurality of historical probabilities of the label are obtained. Each label corresponds to a historical probability each time the true value inference is performed before the current probability is obtained;
[0088] The probability of the label is subjected to interval estimation based on the current probability and the plurality of historical probabilities.
[0089] In actual implementation, since a historical probability of a label is obtained each time the true value inference is performed, each label corresponds to a plurality of historical probabilities, and the historical probabilities are historical probabilities obtained before the current probability is obtained. After the plurality of historical probabilities of the label are obtained, the probability of the label is subjected to interval estimation according to the current probability and the plurality of historical probabilities. Optionally, the current probability and the plurality of historical probabilities can be combined to form a probability set, and then the probability of the label is subjected to interval estimation according to the probability set.
[0090] It should be noted that the specific method of interval estimation is not limited herein, and any interval estimation method in the prior art can be selected as needed. Optionally, the interval estimation can also be expressed by a confidence interval.
[0091] In some embodiments, the probability of the label is subjected to interval estimation based on the current probability and the plurality of historical probabilities, and specifically includes:
[0092] Determine the average of the current probability and the plurality of historical probabilities, and the variance of the current probability and the plurality of historical probabilities;
[0093] The probability of the label is estimated from the variance and mean.
[0094] In practice, the probability of the label can be estimated from an interval based on the current probability and the average and variance of the multiple historical probabilities. Optionally, the upper and lower bounds of the probability interval of the label can be determined using the following formula:
[0095]
[0096] Where u represents the average of the current probability and the plurality of historical probabilities, s represents the variance of the current probability and the plurality of historical probabilities, n represents the number of the current probability and the plurality of historical probabilities, and t_ val This represents the t-score corresponding to the pre-set reliability in the t-distribution.
[0097] In some embodiments, obtaining multiple historical probabilities of the tag specifically includes:
[0098] Determine the number of all historical probabilities for the label;
[0099] In response to the determination that the number of all historical probabilities is greater than a preset number, the preset number of historical probabilities are obtained sequentially from the historical probability closest to the current probability, according to the time order of truth inference.
[0100] In practice, first determine the number of all historical probabilities for each tag starting from the first truth inference. When the number of all historical probabilities is greater than a preset number, start from the historical probability closest to the current probability and sequentially obtain the preset number of historical probabilities from the back to the front according to the time sequence of the truth inference. For example, the preset number is w, and the current probability is A. n The closest historical probability to the current probability is A. n-1 The probability of sequentially obtaining a preset number of historical records is A. n-1 A n-2 A n-3 A n-4 …A n-w It should be noted that, initially, the number of labeled results in the dataset used for truth inference is small, resulting in low accuracy. Therefore, when obtaining historical probabilities, it is advisable to select from historical probabilities that are closest to the current probability. The preset number can be set as needed and is not limited here.
[0101] In some embodiments, the plurality of historical probabilities of the label are obtained, specifically comprising:
[0102] The plurality of historical probabilities of the label are obtained, specifically comprising:
[0103] In response to determining that the number of all historical probabilities is not greater than a preset number, all historical probabilities are obtained.
[0104] In practice, since the number of historical probabilities obtained is small at the beginning of the true value inference, when the number of all historical probabilities is not greater than a preset number, all historical probabilities are obtained.
[0105] S104, in response to determining that the interval ranges of the probabilities of all labels of the target topic are all not greater than a preset convergence threshold, the distribution of the target topic is stopped.
[0106] In practice, after obtaining the current probability of the label corresponding to the target topic, it is determined whether the interval ranges of the probabilities of all labels of the target topic are all not greater than a preset convergence threshold. If yes, the distribution of the target topic is stopped, so that the number of the distribution of the target topic can be timely controlled. Since the interval ranges of the probabilities of all labels of the target topic have converged to the preset convergence threshold, the effect of the true value inference can be guaranteed. Optionally, the preset convergence threshold can be set according to needs, which is not limited herein. When it is determined that the interval ranges of the probabilities of all labels of the target topic are greater than the preset convergence threshold, the distribution of the target topic is continued.
[0107] In some embodiments, after the distribution of the target topic is stopped, the method further comprises:
[0108] A target label with the maximum probability is determined from the probabilities of all labels.
[0109] The target label is taken as the annotation label of the target topic.
[0110] In some embodiments, after the annotation result set of the target topic is updated based on the current annotation result, the method further comprises:
[0111] It is determined whether the number of annotation results in the annotation result set is less than a preset threshold.
[0112] In response to determining that the number of annotation results in the annotation result set is less than a preset threshold, the updated annotation result set is no longer subjected to the true value inference.
[0113] In a specific implementation, when the number of annotation results in the annotation result set is particularly small, the accuracy of the true value inference result obtained is low, and the number of historical probabilities of the labels obtained is also small. In this case, the interval range of the probability of the label is generally large, that is, it does not converge to the preset convergence threshold, and the subject title is generally not stopped from being distributed. Therefore, in order to improve the efficiency, when the number of annotation results in the annotation result set is less than the preset threshold, the updated annotation result set is no longer subjected to true value inference, and the distribution of the subject title is continued.
[0114] In some embodiments, the reference Figure 3 In the process of gradually distributing and collecting the data to be annotated, the true value inference calculation is performed after each distribution of the subject title and collection of the annotation, and the output result of the true value inference is dynamically observed through the interval estimation evaluation. When the interval of the output result of the true value inference is narrowed to a preset convergence value under the condition of meeting a certain confidence, an annotation stop signal is sent, the annotation process is interrupted in time, and the data annotation result is output, so that the annotation cost is reduced as much as possible under the premise of ensuring the quality of the data crowdsourcing.
[0115] The data crowdsourcing annotation distribution method provided in the application distributes a subject title and receives a current annotation result of the subject title, updates an annotation result set of the subject title based on the current annotation result, performs true value inference on the updated annotation result set, obtains a current probability of a label corresponding to the subject title, performs interval estimation on the probability of the label based on the current probability, obtains an interval range of the probability of the label, and stops distributing the subject title in response to determining that the interval ranges of the probabilities of all labels of the subject title are not greater than a preset convergence threshold, thereby reducing the number of annotations of the subject title under the premise of ensuring the quality of the data crowdsourcing, and saving the annotation cost.
[0116] Exemplary device
[0117] Corresponding to any of the above-mentioned embodiment methods based on the same inventive concept, the application further provides a data crowdsourcing annotation distribution device.
[0118] With reference to Figure 4 The data crowdsourcing annotation distribution device comprises:
[0119] The distribution and collection module 201 distributes a subject title and receives a current annotation result of the subject title, and updates an annotation result set of the subject title based on the current annotation result.
[0120] The inference module 202 performs true value inference on the updated annotation result set, and obtains a current probability of a label corresponding to the subject title.
[0121] An interval estimation module 203 is configured to perform interval estimation on the probability of the label based on the current probability, to obtain an interval range of the probability of the label.
[0122] A control module 204 is configured to, in response to determining that the interval range of the probability of all labels of the target topic is not greater than a preset convergence threshold, stop distributing the target topic.
[0123] In some embodiments, the interval estimation module is specifically configured to:
[0124] Obtain a plurality of historical probabilities of the label, wherein each label corresponds to one historical probability each time a true value inference is performed before the current probability is obtained;
[0125] Perform interval estimation on the probability of the label based on the current probability and the plurality of historical probabilities.
[0126] In some embodiments, the interval estimation module is specifically configured to:
[0127] Determine the mean value of the current probability and the plurality of historical probabilities, and the variance of the current probability and the plurality of historical probabilities;
[0128] Perform interval estimation on the probability of the label based on the variance and the mean value.
[0129] In some embodiments, the interval estimation module is specifically configured to:
[0130] Determine the number of all historical probabilities of each label;
[0131] In response to determining that the number of all historical probabilities is greater than a preset number, obtain a preset number of historical probabilities in the order from the most recent historical probability to the earliest historical probability in the time sequence of performing true value inference.
[0132] In some embodiments, the interval estimation module is specifically configured to:
[0133] In response to determining that the number of all historical probabilities is not greater than a preset number, obtain all historical probabilities.
[0134] In some embodiments, the apparatus further comprises a judgment module configured to:
[0135] Determine whether the number of annotation results in the annotation result set is less than a preset threshold;
[0136] In response to determining that the number of annotation results in the annotation result set is less than a preset threshold, stop performing true value inference on the updated annotation result set.
[0137] In some embodiments, the distribution and collection module is specifically configured to:
[0138] create a labeling task; wherein the labeling task at least includes the target objective;
[0139] distribute the labeling task.
[0140] For the convenience of description, the above apparatus is described in various modules in terms of functions. Of course, in the implementation of the present application, the functions of each module can be implemented in one or more software and / or hardware.
[0141] The apparatus of the above embodiments is used to implement the data crowdsourcing labeling distribution method of any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here again.
[0142] Based on the same inventive concept, corresponding to any of the above method embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data crowdsourcing labeling distribution method of any of the above embodiments.
[0143] Figure 5 A more specific hardware structure of an electronic device is shown in the embodiment, which can include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040 and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030 and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.
[0144] The processor 1010 can be implemented in the form of a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present specification.
[0145] The memory 1020 can be implemented in the form of a ROM (Read Only Memory, read-only memory), a RAM (Random Access Memory, random access memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are saved in the memory 1020 and executed by the processor 1010.
[0146] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.
[0147] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).
[0148] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.
[0149] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0150] The electronic device of the above embodiments is used to implement the data crowdsourcing annotation distribution method of the corresponding embodiments of any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.
[0151] Exemplary program product
[0152] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the data crowdsourcing annotation distribution method of any of the above embodiments.
[0153] The computer readable media of the embodiments can include permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.
[0154] The storage medium of the above embodiments stores computer instructions for causing the computer to perform a data crowd labeling distribution method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.
[0155] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present application (including claims) is limited to these examples; the above embodiments or technical features between different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of the different aspects of the embodiments of the present application as described above. In order to be brief, they are not provided in detail.
[0156] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented the embodiments of the present application (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe the exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.
[0157] While the present application has been described in connection with certain embodiments thereof, many modifications, substitutions, changes, and of forms will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.
[0158] Embodiments of the present application are intended to cover all such alterations, modifications, and variations as they can come within the scope of the appended claims. Accordingly, although specific embodiments have been furthered in connection with the present application, any omission, substitution, change, improvement, etc. made by one of ordinary skill in the art to the disclosed embodiments should be considered to be within the scope of the present application.
Claims
1. A method for distributing data crowdsourced annotations, characterized in that, include: Distribute target questions and receive the current annotation results of the target questions, and update the annotation result set of the target questions based on the current annotation results; Determine whether the number of annotation results in the annotation result set is less than a preset threshold; In response to determining that the number of labeled results in the labeled result set is not less than a preset threshold, truth inference is performed on the updated labeled result set to obtain the current probability of the tag corresponding to the target question; Based on the current probability, the probability of the tag is estimated within an interval to obtain the range of the tag's probability. In response to the fact that the probability range of all tags of the target question is no greater than a preset convergence threshold, the distribution of the target question is stopped; The method further includes: in response to determining that the number of labeled results in the labeled result set is less than a preset threshold, ceasing to perform truth inference on the updated labeled result set, thereby continuing to distribute the target question.
2. The method according to claim 1, characterized in that, Based on the current probability, an interval estimation of the probability of the label is performed, specifically including: Obtain multiple historical probabilities of the label; wherein, before obtaining the current probability, each label corresponds to a historical probability for each truth inference performed; The probability of the label is estimated from the current probability and the multiple historical probabilities.
3. The method according to claim 2, characterized in that, Based on the current probability and the multiple historical probabilities, an interval estimation of the probability of the label is performed, specifically including: Determine the average of the current probability and the plurality of historical probabilities, and the variance of the current probability and the plurality of historical probabilities; The probability of the label is estimated from the variance and mean.
4. The method according to claim 2, characterized in that, Obtaining multiple historical probabilities of the label specifically includes: Determine the number of all historical probabilities for the label; In response to the determination that the number of all historical probabilities is greater than a preset number, the preset number of historical probabilities are obtained sequentially from the historical probability closest to the current probability, according to the time order of truth inference.
5. The method according to claim 4, characterized in that, Obtaining multiple historical probabilities of the label specifically includes: In response to determining that the total number of all historical probabilities is not greater than a preset number, all historical probabilities are acquired.
6. The method according to claim 1, characterized in that, After ceasing the distribution of the target question, the method further includes: Determine the target label with the highest probability from all the probabilities of the labels; The target label is used as the annotation label for the target question.
7. The method according to claim 1, characterized in that, The specific topics to be distributed include: Create a labeling task; wherein the labeling task includes at least the target title; Distribute the annotation task.
8. A data crowdsourcing annotation distribution device, characterized in that, include: The distribution and recycling module distributes target questions and receives the current annotation results of the target questions, and updates the annotation result set of the target questions based on the current annotation results; The determination module determines whether the number of annotation results in the annotation result set is less than a preset threshold; The inference module, in response to determining that the number of labeled results in the labeled result set is not less than a preset threshold, performs truth inference on the updated labeled result set to obtain the current probability of the tag corresponding to the target question; The interval estimation module performs interval estimation on the probability of the label based on the current probability to obtain the interval range of the probability of the label; The control module stops distributing the target question in response to determining that the probability range of all tags of the target question is no greater than a preset convergence threshold. The device further includes a continuing distribution module, which, in response to determining that the number of labeled results in the labeled result set is less than a preset threshold, stops performing truth inference on the updated labeled result set and continues to distribute the target question.
9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
top-N service re-recommendation method based on crowdsourcing and top-N service re-recommendation system
CN112148986A