Method, device and computer program product for file annotation

Abstract

Embodiments of the present disclosure relate to a method, device and computer program product for file annotation. The method comprises: processing a plurality of files to be annotated by using an annotation model to determine a first performance of the annotation model, the first performance being related to a confidence of model annotation results generated by the annotation model; if the first performance is lower than a predetermined threshold, determining a set of target files from the plurality of files based on at least the confidence of the model annotation results; obtaining ground truth annotation information of the set of target files for retraining the annotation model; and if a second performance of the retrained annotation model is higher than or equal to the predetermined threshold, determining annotation information for at least part of the plurality of files by using the retrained annotation model. In this way, automatic annotation of files can be achieved by using less ground truth annotation information, thereby reducing annotation cost.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computers, and more specifically, to methods, apparatus, and computer program products for document annotation. Background Technology

[0002] In file management within large file systems, it's often necessary to annotate files to improve efficiency. Such annotations help in faster file retrieval and categorization. For example, text files can be annotated to indicate the topics covered by the text content, such as economics, education, or entertainment.

[0003] Currently, manual annotation of documents is costly. Some solutions attempt to use machine learning-based annotation models to perform automatic annotation, but these models still rely on ground truth annotations obtained from manual annotation. Therefore, there is a desire to obtain sufficiently accurate annotation models using less ground truth annotation information. Summary of the Invention

[0004] The embodiments of this disclosure provide a scheme for document annotation.

[0005] According to a first aspect of this disclosure, a method for document annotation is proposed. The method includes: processing multiple documents to be annotated using an annotation model to determine a first performance of the annotation model, the first performance being related to the confidence level of model annotation results generated by the annotation model; if the first performance is lower than a predetermined threshold, determining a set of target documents from the multiple documents based at least on the confidence level of the model annotation results; obtaining ground truth annotation information for the set of target documents for retraining the annotation model; and if a second performance of the retrained annotation model is higher than or equal to a predetermined threshold, using the retrained annotation model to determine annotation information for at least a portion of the multiple documents.

[0006] According to a second aspect of this disclosure, an electronic device is proposed. The device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform an action comprising: processing a plurality of files to be labeled using an annotation model to determine a first performance of the annotation model, the first performance being related to the confidence level of model annotation results generated by the annotation model; if the first performance is lower than a predetermined threshold, determining a set of target files from the plurality of files based at least on the confidence level of the model annotation results; obtaining ground truth annotation information for the set of target files for retraining the annotation model; and if a second performance of the retrained annotation model is higher than or equal to the predetermined threshold, determining annotation information for at least a portion of the files using the retrained annotation model.

[0007] In a third aspect of this disclosure, a computer program product is provided. The computer program product is stored in a non-transitory computer storage medium and includes machine-executable instructions that, when executed in a device, cause the device to perform any step of the method described in the first aspect of this disclosure.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0009] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0010] Figure 1 The illustration shows a schematic diagram of an exemplary environment in which embodiments of the present disclosure may be implemented;

[0011] Figure 2 The illustration is a schematic diagram of a document annotation process according to some embodiments of the present disclosure;

[0012] Figure 3 The illustration shows a schematic diagram of a process for filtering target files according to some embodiments of the present disclosure; and

[0013] Figure 4 A schematic block diagram of an example device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0014] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0015] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0016] As discussed above, when a large number of documents need to be labeled, some solutions train machine learning models by manually labeling a subset of documents, thus reducing the overall cost of document labeling. Therefore, there is a growing focus on how to further reduce the cost of manual labeling while ensuring the accuracy of machine learning model labeling.

[0017] According to embodiments of this disclosure, a document annotation scheme is provided. In this scheme, an annotation model is used to process multiple documents to be annotated to determine a first performance of the annotation model, wherein the first performance is related to the confidence level of the model annotation results generated by the annotation model. If the first performance is lower than a predetermined threshold, a set of target documents is determined from the multiple documents, at least based on the confidence level of the model annotation results. Subsequently, ground truth annotation information for the set of target documents is obtained for retraining the annotation model. If the second performance of the retrained annotation model is higher than or equal to a predetermined threshold, the retrained annotation model is used to determine annotation information for at least a portion of the multiple documents. Based on this approach, embodiments of this disclosure can obtain an annotation model with satisfactory performance using less ground truth annotation information, thereby further reducing the cost of document annotation.

[0018] The present disclosure will now be described with reference to the accompanying drawings.

[0019] Figure 1 An example environment 100 in which embodiments of this disclosure may be executed is shown. For example... Figure 1 As shown, the environment 100 includes an annotation device 120, which can receive multiple files 110 to be annotated.

[0020] like Figure 1As shown, the annotation device 120 can acquire a machine learning-based annotation model 130. Examples of such annotation models 130 include, but are not limited to, various deep neural networks (DNNs), convolutional neural networks (CNNs), support vector machines (SVMs), decision trees, random forest models, and so on.

[0021] In some embodiments, the annotation model 130 may be a model designed for a specific training dataset. In some examples, the training dataset may be, for example, a set of documents that the user expects to annotate. For example, the user may expect to annotate 100,000 documents, and the annotation device 120 may select a certain number of documents (e.g., 10,000 documents) from the 100,000 documents and send them to the annotation service provider 140 to obtain ground truth annotation information for the 10,000 documents.

[0022] Such ground truth annotation information can be used to initially train the annotation model 130. Specifically, the input used to train the annotation model 130 may include the feature representations of these documents, and the training objective of the annotation model 130 may be to reduce the difference between the model annotation results determined by the annotation model 130 and the corresponding ground truth annotation information.

[0023] The feature representation of a file can be determined using any suitable method. For example, a trained encoder can be used to generate the feature representation of a file based on its attribute information. This disclosure is not intended to limit the method of generating the feature representation of a file.

[0024] The annotation device 120 can further utilize the annotation model 130 to process multiple files 110, thereby determining the confidence level of the model annotation results determined by the annotation model 130. Furthermore, the annotation device 120 can determine the performance of the annotation model 130 based on these confidence levels.

[0025] If the performance reaches a predetermined threshold, the annotation device 120 can, for example, directly determine the model annotation result determined by the annotation model 130 as the annotation information of multiple files 110.

[0026] Conversely, if the performance does not reach the threshold, the annotation device 120 can further filter a set of target files from the multiple files 110 and obtain the ground truth annotation information of the set of target files to retrain the annotation model 130. For example, the annotation device 120 can send the set of target files to the annotation service provider 140 to obtain the corresponding ground truth annotation information.

[0027] The annotation device 120 can further determine the performance of the retrained annotation model 130, and, if the performance reaches a predetermined threshold, use the retrained annotation model 130 to determine annotation information 150-1, 150-2, ... 150-N (individually or collectively referred to as annotation information 150) for at least a portion of the multiple files 110. Detailed procedures for generating annotation information 150 will be described in [reference needed]. Figure 2 Describe it.

[0028] Figure 2 A flowchart of a document-marked process 200 according to some embodiments of the present disclosure is shown. Process 200 may, for example, be... Figure 1 The annotation device 120 shown is used to implement this.

[0029] like Figure 2 As shown, the annotation device 120 uses the annotation model 130 to process multiple files 110 to be annotated in order to determine the first performance of the annotation model 130, wherein the first performance is related to the confidence level of the model annotation results generated by the annotation model 130.

[0030] In some embodiments, as discussed above, the plurality of files 110 may be a first subset of a user-labeled file set, and the labeling model 130 is initially trained using ground truth labeling information from a second subset of the file set to be labeled.

[0031] For example, the set of files to be annotated by the user may include 100,000 files, of which 10,000 files may be randomly selected as a second subset. In some embodiments, the truth annotation information of the second subset may be determined, for example, by the annotation service provider 140. Such an annotation service provider 140 may, for example, include any suitable individual or organization.

[0032] Furthermore, it should be understood that the "truth value annotation information" in this article refers to annotation information with a high confidence level that is directly used to train the annotation model 130, and does not necessarily require that this "truth value annotation information" be objectively accurate. For example, some experts may still make some incorrect annotations when manually annotating documents, but such annotation information is still regarded as "truth value annotation information".

[0033] In some embodiments, the confidence level of the model annotation results can be characterized, for example, by an uncertainty measure of the model annotation results. Such an uncertainty measure is, for example, a parameter determined by the machine learning model to characterize the degree of certainty regarding the provided model annotation results. For example, the Bayesian Active Learning by Disagreement (BALD) measure can be used as the uncertainty measure in this disclosure. It should be understood that any other suitable uncertainty measure can also be used. It should be understood that a larger value of the uncertainty measure indicates a higher degree of uncertainty in the model annotation results.

[0034] In some embodiments, the annotation device 120 may further determine a first performance based on an uncertainty measure of the model prediction results of the plurality of files 110. For example, the annotation device 120 may determine the first performance based on the average of the uncertainty measures of the plurality of files 110. Alternatively, the annotation device 120 may determine the first performance based on the maximum value of the uncertainty measures of the plurality of files 110.

[0035] Alternatively, the annotation device 120 can also base its annotation on the number or proportion of files in the plurality of files 110 whose uncertainty measure is greater than a threshold. For example, if the plurality of files 110 includes 100,000 files, and the number of files whose uncertainty measure is greater than a predetermined threshold is 50,000, then, for example, the first performance can be determined as a proportion of 0.5.

[0036] In box 204, annotation device 120 determines whether the first performance of the annotated model 130 is below a predetermined threshold. If so, process 200 proceeds to box 206, where annotation device 120 determines a set of target files from multiple files 110 based at least on the confidence level of the model annotation results.

[0037] In some embodiments, the annotation device 120 may, for example, select files with a confidence level lower than a predetermined level from a plurality of files 110 as the target files in the group, based on the confidence level. In this way, files that are currently difficult for the annotation model 120 to annotate accurately can be selected.

[0038] In some embodiments, the annotation device 120 may also consider, for example, the expected annotation cost required to obtain truth annotation information of a document, in order to achieve a balance between improving the accuracy and cost of the annotation model. The following will refer to... Figure 3 To describe the process of filtering target files, specifically, Figure 3 A flowchart of a process 300 for screening target files according to some embodiments of the present disclosure is shown.

[0039] like Figure 3As shown, in block 302, the annotation device 120 can utilize a cost prediction model to determine the expected annotation cost of acquiring ground truth annotation information for multiple files, wherein the cost prediction model is trained based on the historical annotation costs of a set of training files. In some embodiments, the expected annotation cost is determined by at least one of the expected time required to annotate the files and the expected cost required to annotate the files.

[0040] In some embodiments, a cost prediction model can be constructed using 140 pairs of historical annotations from an annotation service provider. Examples of such cost prediction models may include, but are not limited to, various deep neural networks (DNNs), convolutional neural networks (CNNs), support vector machines (SVMs), decision trees, random forest models, and so on.

[0041] During training, the feature representation of the file can be input into the cost prediction model, and the objective function can be constructed based on the actual annotation cost of the annotation service provider 140, so that the trained cost prediction model can determine the expected annotation cost based on the feature representation of the file.

[0042] It should be understood that the training of the cost prediction model can be performed by the same or different devices as the annotation device 120. For example, the annotation device 120 can send the feature representations of multiple files 120 to another device to obtain the expected annotation cost. Alternatively, the cost prediction model can also be deployed on the annotation device 120 to determine the expected annotation cost.

[0043] In box 304, annotation device 120 can select a set of target files from multiple files based on the uncertainty metric and expected annotation cost corresponding to each file.

[0044] In some embodiments, the annotation device 120 may determine the annotation score for each document based on an uncertainty metric and an expected annotation cost, wherein the annotation score increases with an increase in the uncertainty metric and decreases with an increase in the expected annotation cost.

[0045] For example, the annotation score can be represented as a function of the uncertainty measure and the expected annotation cost:

[0046] Score(x)=α*Uncertainty(x)+β*1 / Effort(x) (1)

[0047] Where Score(x) represents the annotation score of file x, Uncertainty(x) represents the uncertainty measure of file x, Effort(x) represents the expected annotation cost of file x, and α and β are weighting coefficients. It should be understood that α and β can be predetermined constants, or specified by the user according to the actual scenario.

[0048] It should be understood that other appropriate formulas can also be used to determine the annotation score.

[0049] Furthermore, the annotation device 120 can select a set of target files from multiple files based on the annotation scores.

[0050] In some embodiments, the annotation device 120 may select, for example, a predetermined number of files with the highest annotation scores as the target files in the group. Alternatively, the annotation device 120 may also determine files with annotation scores greater than a predetermined threshold as target files, regardless of the total number of target files in the group.

[0051] Continue to refer to Figure 2 In box 208, the annotation device 120 acquires a set of ground truth annotation information for the target files for retraining the annotation model.

[0052] In some embodiments, the annotation device 120 may send the set of object files to the annotation service provider 140 to obtain ground truth annotation information for the set of object files. Furthermore, the annotation device 120 may use the ground truth annotation information to retrain the annotation model 130.

[0053] In some embodiments, the annotation device 120 may perform additional training based on an existing model, utilizing the feature representations of the set of target files and the corresponding ground truth annotation information, to obtain a retrained annotation device.

[0054] In other embodiments, the annotation device 120 may, for example, add the set of object files to a previous training dataset to construct a new training dataset. The annotation device 120 may then use this new training dataset to retrain the annotation model 130.

[0055] Subsequently, process 200 returns to box 204, where annotation device 120 further determines whether the second performance of the retrained annotation model is below a predetermined threshold.

[0056] In some embodiments, the annotation device 120 may determine the second performance of the retrained annotation model in accordance with the manner in which the first performance was determined above. Specifically, the annotation device 120 may utilize the retrained annotation model to process the remaining files in the plurality of files 110 for which ground truth annotation information has not been obtained, and determine the second performance based on the confidence level of the model annotation results.

[0057] If it is determined in box 204 that the second performance is higher than or equal to a predetermined threshold, then process 200 can proceed to box 210, whereby the annotation device 120 can use the retrained annotation model to determine annotation information for at least a portion of the multiple files.

[0058] Specifically, the annotation device 120 can directly use the model annotation results determined by the annotation model for files for which truth value annotation information has not been obtained as the annotation information 150 for these files.

[0059] Based on this approach, embodiments of this disclosure can always obtain truth labeling information for documents that greatly benefit the improvement of model accuracy, thereby reducing the cost of document labeling while ensuring the accuracy of the labeled model.

[0060] Continue to refer to Figure 2 If it is determined in box 204 that the second performance of the retrained labeled model is still below the threshold, steps 206 and 207 can be performed iteratively until the performance of the retrained labeled model is above the predetermined threshold.

[0061] Specifically, the annotation device 120 can further determine a new set of target files from files that have never had ground truth annotation information, and obtain ground truth annotation information from the new set of target files in order to retrain the annotation model.

[0062] Based on this approach, embodiments of this disclosure can iteratively optimize the performance of the annotation model, thereby further reducing, for example, the number of documents that require manual annotation, and thus further reducing the initial level of document annotation while ensuring the accuracy of the model.

[0063] Figure 4 A schematic block diagram of an example device 400 that can be used to implement embodiments of the present disclosure is shown. For example, the labeled device 120 according to an embodiment of the present disclosure can be implemented by device 400. As shown, device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 402 or loaded from storage unit 408 into random access memory (RAM) 403. Various programs and data required for the operation of device 400 may also be stored in RAM 403. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0064] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0065] The various processes and handling described above, such as process 200, can be executed by processing unit 401. For example, in some embodiments, process 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by CPU 401, one or more actions of process 200 described above may be performed.

[0066] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0067] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0068] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0069] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0070] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0072] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0074] Various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for document annotation, comprising: The annotation model is used to annotate multiple files to be annotated in order to determine the first performance of the annotation model, the first performance being related to a confidence level, which is a function of the aggregate uncertainty measure of the model annotation results generated by the annotation model; If the first performance is lower than a predetermined threshold, a set of target files is determined from the plurality of files based at least on the confidence level of the model annotation results, wherein determining the set of target files from the plurality of files includes: The expected annotation cost for obtaining truth labeling information for each file in the plurality of files is determined using a cost prediction model trained based on the historical annotation costs of a set of training files; and Based on the corresponding uncertainty metric for each file and the corresponding expected annotation cost for each file, the set of target files is selected from the plurality of files; Obtain the truth value annotation information of the set of target files; The annotation model is retrained based on the ground truth annotation information of the set of target files to generate a retrained annotation model; and If the second performance of the retrained annotation model is higher than or equal to the predetermined threshold, the retrained annotation model is used to determine annotation information for at least a portion of the plurality of files.

2. The method according to claim 1, wherein the confidence level is a first confidence level, and wherein the second confidence level of the annotation prediction results of the set of target files is selected to be below a predetermined level.

3. The method of claim 1, wherein selecting the set of target files from the plurality of files comprises: Based on the corresponding uncertainty measure and the corresponding expected annotation cost, a corresponding annotation score for each file is determined. The corresponding annotation score increases with the increase of the corresponding uncertainty measure and decreases with the increase of the corresponding expected annotation cost. The determination of the corresponding annotation score for each file generates the corresponding annotation scores for the files in the plurality of files. as well as Based on the corresponding labeled scores, select the set of target files from the plurality of files.

4. The method of claim 1, wherein the corresponding expected annotation cost for each file is determined based on at least one of the expected time required to annotate the file or the expected cost required to annotate the file.

5. The method according to claim 1, further comprising: If the second performance is determined to be below the predetermined threshold, the retrained labeled model is iteratively retrained to produce a further retrained labeled model, wherein the iterative retraining includes: Determine a new set of target files from files for which truth labeling information has never been obtained; and Obtain the ground truth annotation information of the new set of target files for the iterative retraining of the retrained annotation model to generate the further retrained annotation model.

6. The method according to claim 1, wherein obtaining the truth value annotation information of the set of target files includes: Send the set of target files to the annotation service provider; as well as Receive truth value annotation information for the set of target files from the annotation service provider.

7. The method of claim 1, wherein the first performance is determined based on one of the following: The average of the corresponding uncertainty measure for each of the multiple files, The maximum value of the corresponding uncertainty metric for each of the multiple files, or The number or proportion of files whose corresponding uncertainty measure is greater than the threshold among the multiple files.

8. The method of claim 1, wherein the plurality of files is a first subset of a set of files to be labeled, and the labeling model is initially trained using ground truth labeling information from a second subset of the set of files to be labeled.

9. An electronic device, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform an action, the action including: The annotation model is used to annotate multiple files to be annotated in order to determine the first performance of the annotation model, the first performance being related to a confidence level, which is a function of the aggregate uncertainty measure of the model annotation results generated by the annotation model; If the first performance is lower than a predetermined threshold, a set of target files is determined from the plurality of files based at least on the confidence level of the model annotation results, wherein determining the set of target files from the plurality of files includes: The expected annotation cost for obtaining truth labeling information for each file in the plurality of files is determined using a cost prediction model trained based on the historical annotation costs of a set of training files; and Based on the corresponding uncertainty metric for each file and the corresponding expected annotation cost for each file, the set of target files is selected from the plurality of files; Obtain the truth value annotation information of the set of target files; The annotation model is retrained based on the ground truth annotation information of the set of target files to generate a retrained annotation model; and If the second performance of the retrained annotation model is higher than or equal to the predetermined threshold, the retrained annotation model is used to determine annotation information for at least a portion of the plurality of files.

10. The electronic device of claim 9, wherein the confidence level is a first execution level, and the second confidence level of the labeled prediction results of the set of target files is selected to be below a predetermined level.

11. The electronic device of claim 9, wherein selecting the set of target files from the plurality of files comprises: Based on the uncertainty measure and the corresponding expected annotation cost, a corresponding annotation score for each file is determined. The corresponding annotation score increases with the increase of the corresponding uncertainty measure and decreases with the increase of the corresponding expected annotation cost. The determination of the corresponding annotation score for each file generates the corresponding annotation scores for the files in the plurality of files. as well as Based on the corresponding labeled scores, select the set of target files from the plurality of files.

12. The electronic device of claim 9, wherein the expected annotation cost corresponding to each file is determined based on at least one of the expected time required to annotate the file or the expected cost required to annotate the file.

13. The electronic device according to claim 9, wherein the action further includes: If the second performance is lower than the predetermined threshold, the retrained labeled model is iteratively retrained, the iterative retraining including: Determine a new set of target files from files for which truth labeling information has never been obtained; and Obtain the ground truth annotation information of the new set of target files for the iterative retraining of the retrained annotation model, resulting in a further retrained annotation model.

14. The electronic device according to claim 9, wherein obtaining the truth value annotation information of the set of target files includes: Send the set of target files to the annotation service provider; as well as Receive truth value annotation information for the set of target files from the annotation service provider.

15. The electronic device of claim 9, wherein the first performance is determined based on one of the following: The average of the corresponding uncertainty measure for each of the multiple files, The maximum value of the corresponding uncertainty metric for each of the multiple files, or The number or proportion of files whose corresponding uncertainty measure is greater than the threshold among the multiple files.

16. The electronic device of claim 9, wherein the plurality of files is a first subset of a set of files to be labeled, and the labeling model is initially trained using ground truth labeling information of a second subset of the set of files to be labeled.

17. A computer program product stored in a non-transient computer storage medium and comprising machine-executable instructions that, when executed in a device, cause the device to perform the method according to any one of claims 1-8.

Images