Distributed data annotation method and industry document importance identification method

Abstract

The present application relates to a distributed data annotation method and an industry file importance identification method, wherein the industry file importance identification method includes: inputting a question set Q into a large language model, asking the large language model about the characteristic attributes of the target industry file, and obtaining an answer set A output by the large language model in response to the question set Q; matching the answer set A in the target feature library, determining the weights of each characteristic attribute category of the target industry file, the target feature library has a mapping relationship between different characteristic attributes and different categories, and the mapping relationship between different characteristic attributes and different categories is obtained through a target data annotation method, and different categories are used to represent different degrees of importance; determining the importance level of the target industry file based on the category weights of each characteristic attribute of the target industry file. This solves the problem that the current industry file importance identification method is difficult to accurately identify semi-structured or unstructured industry files.

Description

Technical Field

[0001] The present application relates to the field of data identification, and in particular to a distributed data annotation method and an industry document importance identification method. Background Art

[0002] In each industry, it may be necessary to identify the importance of industry files, that is, to determine the importance level of a certain industry file. Generally speaking, if the industry file uses structured data, it is easier to identify. If the industry file uses semi-structured data or unstructured data, the current industry file importance identification method is difficult to identify the industry file more accurately. At the same time, the importance identification of industry files mainly involves first identifying the characteristic attributes of the industry file, then matching the categories of each characteristic attribute, and finally determining the importance level of the industry file based on the characteristic attribute category of the industry file. Among them, the category of each characteristic attribute needs to be labeled in advance. Faced with the task of labeling massive data, traditional data labeling methods have the problem of low labeling efficiency.

[0003] The current methods for identifying the importance of industry documents are difficult to accurately identify semi-structured or unstructured industry documents, and no effective solution has been proposed yet. Summary of the invention

[0004] The present invention provides a distributed data labeling method and an industry document importance identification method to solve the problem that the current industry document importance identification method is difficult to accurately identify semi-structured or unstructured industry documents.

[0005] In a first aspect, the present invention provides a method for identifying the importance of industry documents, comprising:

[0006] Acquire a target industry file to be identified, where the target industry file uses semi-structured data or unstructured data;

[0007] Construct a question set Q for the target industry file, input the question set Q into a large language model, ask the large language model questions about characteristic attributes of the target industry file, and obtain an answer set A output by the large language model in response to the question set Q;

[0008] Matching the answer set A in a target feature library to determine the weights of each feature attribute category of the target industry file, wherein the target feature library has a mapping relationship between different feature attributes and different categories, and the mapping relationship between different feature attributes and different categories is obtained by a target data annotation method, and different categories are used to represent different degrees of importance;

[0009] The importance level of the target industry file is determined based on the category weights of the characteristic attributes of the target industry file.

[0010] In a second aspect, the present invention provides a distributed data labeling method, wherein the target data labeling method is implemented by a distributed data labeling server system, and the target data labeling method includes:

[0011] receiving one or more target source data;

[0012] selecting one or more annotators for one or more of the target source data;

[0013] Creating one or more of the annotation tasks for the selected one or more of the annotators;

[0014] Requesting one or more annotations for one or more of the target source data by creating one or more of the annotation tasks;

[0015] receiving one or more of the annotations;

[0016] Determine metadata of one or more target source data through one or more annotations, wherein the metadata of the target source data includes target source data features;

[0017] Determining metadata of one or more of the annotators through one or more of the annotations and one or more of the target source data, wherein the metadata of the annotators includes the identified annotator features;

[0018] The true value of one or more target source data is determined according to metadata of one or more target source data and metadata of one or more taggers.

[0019] In a third aspect, the present invention provides an industry document importance identification device, comprising:

[0020] A file acquisition module, used to acquire a target industry file to be identified, wherein the target industry file adopts semi-structured data or unstructured data;

[0021] A feature extraction module is used to construct a question set Q for the target industry file, input the question set Q into a large language model, ask the large language model questions about the characteristic attributes of the target industry file, and obtain an answer set A output by the large language model in response to the question set Q;

[0022] A category matching module is used to match the answer set A in the target feature library to determine the weight of each feature attribute category of the target industry file, wherein the target feature library has a mapping relationship between different feature attributes and different categories, and the mapping relationship between different feature attributes and different categories is obtained by a target data annotation method, and different categories are used to represent different degrees of importance;

[0023] The level determination module is used to determine the importance level of the target industry file based on the category weights of the various characteristic attributes of the target industry file.

[0024] In a fourth aspect, the present invention provides a distributed data annotation system for use in the method for identifying the importance of industry documents described in the first aspect.

[0025] Compared with related technologies, the data in the semi-structured or unstructured target industry files are identified through a large language model to obtain the characteristic attributes of the target industry files. In this process, some extractions related to data identification can be input into the large language model, such as "Please identify the keywords in the file." The answer collection A of the large language model is the characteristic attribute of the target industry file. The present invention uses a large language model to perform data identification of target industry files, which can more accurately understand and interpret the content in semi-structured or unstructured data, and can adapt to data of various formats and structures, so as to more accurately determine the importance level of the target industry files. It solves the problem that the current industry file importance identification method is difficult to more accurately identify semi-structured or unstructured industry files.

[0026] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a method for identifying the importance of industry documents provided in an embodiment of the present invention;

[0028] Figure 2 is a flow chart of a target data labeling method provided in an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of metadata of source data provided in an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of metadata of a tagger provided in an embodiment of the present invention;

[0031] Figure 5 It is a schematic diagram of the annotation of source data provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0033] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0034] In an embodiment of the present invention, a method for identifying the importance of industry documents is provided. Figure 1 is a flow chart of a method for identifying the importance of industry documents provided in an embodiment of the present invention. Figure 1 As shown, the process includes step S110, step S120, step S130 and step S140.

[0035] Step S110, obtaining a target industry file to be identified, where the target industry file uses semi-structured data or unstructured data.

[0036] Step S120, construct a question set Q for target industry documents, input the question set Q into the large language model, ask the large language model about the characteristic attributes of the target industry documents, and obtain the answer set A output by the large language model in response to the question set Q.

[0037] Step S130, matching the answer set A in the target feature library, determining the weights of each feature attribute category of the target industry file, the target feature library has a mapping relationship between different feature attributes and different categories, the mapping relationship between different feature attributes and different categories is obtained through the target data labeling method, and different categories are used to represent different degrees of importance.

[0038] Step S140: determining the importance level of the target industry file based on the category weights of the characteristic attributes of the target industry file.

[0039] In the above technical solution, the data in the semi-structured or unstructured target industry files are identified through a large language model to obtain the characteristic attributes of the target industry files. In this process, some extractions related to data identification can be input into the large language model, such as "Please identify the keywords in the file." The answer collection A of the large language model is the characteristic attribute of the target industry file. The present invention uses a large language model to perform data identification of target industry files, which can more accurately understand and interpret the content in semi-structured or unstructured data, and can adapt to data of various formats and structures, so as to more accurately determine the importance level of the target industry file. It solves the problem that the current industry file importance identification method is difficult to more accurately identify semi-structured or unstructured industry files.

[0040] In some of the embodiments, the target feature library includes a Chinese feature library and / or an English feature library; in the Chinese feature library, the mapping relationship between different feature attributes and different categories is represented by an inverted index; in the English feature library, the mapping relationship between different feature attributes and different categories is represented by a Trie tree. In this embodiment, the Chinese feature library and the English feature library are established respectively using an inverted index and a Trie tree, and feature matching is achieved using an efficient index structure to identify the importance of files, so as to improve the accuracy and efficiency of identification.

[0041] The target data labeling method may adopt the traditional data labeling method, but the traditional data labeling method has the problem of low labeling efficiency. Therefore, the target data labeling method preferably adopts the data labeling method provided by the present invention.

[0042] In some of the embodiments, the target data annotation method is implemented through a distributed data annotation server system. Figure 2 is a flow chart of a target data labeling method provided in an embodiment of the present invention, referring to Figure 2 , the target data labeling method includes steps S210 to S280.

[0043] Step S210: the distributed data annotation server system receives one or more target source data, where the target source data includes different characteristic attributes of industry files.

[0044] Step S220: the distributed data annotation server system selects one or more annotators for one or more target source data.

[0045] Step S230: the distributed data annotation server system creates one or more annotation tasks for the selected one or more annotators.

[0046] Step S240: the distributed data annotation server system requests one or more annotations for one or more target source data through the created one or more annotation tasks.

[0047] Step S250: the distributed data annotation server system receives one or more annotations, and the annotations are determined according to the industry data classification and grading standard file.

[0048] Step S260: the distributed data annotation server system determines metadata of one or more target source data through one or more annotations, where the metadata of the target source data includes characteristics of the target source data.

[0049] Step S270 , the distributed data annotation server system determines metadata of one or more annotators through one or more annotations and one or more target source data, where the metadata of the annotators includes the identified annotator features.

[0050] Step S280: the distributed data annotation server system determines the true value of one or more target source data according to the metadata of one or more target source data and the metadata of one or more annotators.

[0051] Among them, step S280 specifically includes step S281 and step S282.

[0052] Step S281, iteratively updating metadata of one or more target source data and metadata of one or more annotators.

[0053] Step S282, when the iterative update is terminated, determining the true value of one or more target source data according to the updated metadata of one or more target source data and the metadata of one or more annotators.

[0054] The iterative update is implemented based on iterative maximum a posteriori estimation. The iterative update steps include: updating the current metadata of one or more target source data based on the previous metadata of one or more target source data and the previous metadata of one or more target source data; updating the current metadata of one or more annotators based on the previous metadata and annotations of one or more target source data.

[0055] The above-mentioned target data labeling method is implemented through a distributed data labeling server system, so massive data can be labeled in parallel based on parallel computing technology, thereby improving the data labeling efficiency and solving the problem of low labeling efficiency of traditional data labeling methods when facing the labeling task of massive data.

[0056] Specifically, various features can be used to describe the annotator applied to the source data, the source data, and the annotations. These features can be determined using various probabilistic techniques. According to an embodiment of the present invention, the concepts of source data metadata, annotator metadata, and annotations used by the distributed data annotation system are as follows: Figure 3 , Figure 4 and Figure 5 shown.

[0057] Figure 3 Schematic diagram of metadata of source data provided in an embodiment of the present invention. Figure 3 , the metadata 410 of the source data describes the attributes of a piece of source data, including a true value 412, source data features 414, and a record of a previous annotator 416 of the source data. The true value 412 describes the content contained in the source data, corresponding to the metadata 410 of the source data; the true value 412 may or may not correspond to the actual content of the source data. The true value 412 is the best estimate of the basic information based on the annotation of the source data. In the illustrated embodiment, the metadata 410 of the source data also includes a record 416 of the annotator that previously annotated the metadata 410 of the source data. In some embodiments, the record of the previous annotator 416 is a reference to the metadata of one or more annotators stored in the annotator metadata database. The metadata 410 of the source data is stored in the source data-metadata database.

[0058] Source data features 414 describe the features of the source data and can be used to determine the difficulty of annotating the source data. Source data features 414 of image data include, but are not limited to, the specimen captured in the image, the position of the image, the perspective of the image, the pose of the image, the weather captured in the image, the camera angle of the image, the height of the image, and the color in the image. In some embodiments, the source data is a multidimensional embedding of the image, and the source data features 414 include, but are not limited to, the height of the image in the multidimensional embedding and the color of the image in the multidimensional embedding. Other source data features 414 of other types of source data may include other factors not specifically listed above and / or various factors suitable for specific application requirements. Source data features 414 may be affected by random variability when creating source data metadata 410. In various embodiments, one or more source data features 414 are weighted. Source data feature weights are determined using various techniques, including but not limited to using an ideal annotator, using an adversarial annotator, using an ordinary annotator, and using a combination of annotators.

[0059] Figure 4 Schematic diagram of metadata of annotator provided in an embodiment of the present invention. Figure 4 , which illustrates metadata 420 for annotators, describing properties of the annotator, including noise features 422, noise feature weights 424, and error thresholds 426. Noise features 422 describe how the annotator annotates source data. Noise features 422 can be modeled based on various factors, including, but not limited to, the type of source data being annotated. For image data, noise features 422 include, but are not limited to, visual acuity, attention to the task at hand, gaze direction, and knowledge of the annotated subject. Noise features 422 for other types of source data may include other factors not specifically listed above and / or various factors suitable for specific application requirements. Noise features 422 may vary between annotators for specific source data, and the same annotator may vary between different source data. The capabilities of annotators described using annotator metadata 420 can be determined using noise features 422; noise features 422 may be above or below a threshold that can be dynamically determined or determined. Annotator metadata 420 includes noise feature weights 424; noise feature weights 424 describe how the annotator utilizes one or more noise features 422 to annotate source data.

[0060] The annotator's interpretation of the ground truth of the source data based on the noise features 422 and the noise feature weights 424 may not match the actual ground truth of the actual source data, causing the annotator to generate incorrect annotations. The annotator metadata 420 includes an error threshold 426 that describes the annotator's bias towards annotation errors. The error threshold 426 is combined with the annotator's interpretation of the source data to determine the annotations that the annotator will generate for the source data. In various embodiments, the error threshold 426 describes whether the annotator is prone to false positives or false negatives in the annotation of the source data.

[0061] In several embodiments, annotators may be described according to noise features 422, noise feature weights 424, and error thresholds 426 in metadata 420 for the annotator. An "ideal" annotator is an annotator that is not affected by noise features 422, so an ideal annotator can correctly annotate source data. In many embodiments, an ideal annotator has a zero noise feature weight 424 for each noise feature 422. In contrast, an "adversarial" annotator is an annotator that identifies correct annotations for source data; however, the adversarial annotator sometimes or always provides incorrect annotations for the source data. An adversarial annotator may be described as having an error threshold 426, causing the annotator to provide incorrect annotations despite correctly identifying the correct annotations. A "normal" annotator is an annotator that attempts to correctly annotate source data, but, due to noise and / or error thresholds, may make mistakes in some annotations. A normal annotator has at least one noise feature 422, a non-zero noise feature weight 424, and / or an error threshold 426, causing the annotator to provide false positive and / or false negative annotations.

[0062] Figure 5 Schematic diagram of the source data provided in the embodiment of the present invention. Figure 5 , the annotation 430 of the source data includes annotation data 432, source data annotation definitions 434, and task data 436. Annotation data 432 is a label and / or other annotation generated by the annotator for the source data. Source data annotation definitions 434 are the annotator's views on the source data when generating annotations 430 based on the annotator's characteristics and the characteristics of the source data. Task data 436 indicates which annotator generated the annotation data 432 and the source data to which the annotation 430 applies. In many embodiments, the annotator assigns annotations 430 to the metadata 410 of the source data using a linear classifier. In various embodiments, the annotations 430 are stored in a source data database and / or a distributed data annotation server system.

[0063] The above text describes specific source data, annotators and annotations according to the accompanying drawings. Various features can be assigned to source data, annotators and annotations, and various statistical techniques can be used according to embodiments of the present invention. The following describes the process of determining source data features and annotator features according to embodiments of the present invention.

[0064] Reference Figure 4 , the source data features described by the metadata 410 of the source data and the noise features described by the metadata 420 of the annotator can only be determined using the annotated data 432 and various statistical techniques. In several embodiments, the probability of a particular source data feature given a particular annotation set of a set of source data, the noise features of the annotator, and the error threshold of the annotator can be determined as follows:

[0065]

[0066] Where M is the number of annotators for index j, N is the number of source data indexes for i, and l ij are the labels provided by one or more annotators for the source data, x and x i is a source data feature 414 of a piece of source data, w j is the noise feature weight of tagger j, τ j is the error threshold of tagger j, γ and α are w j and τ j The Gaussian prior used during inference, z is the ground truth of a piece of source data, β is the Bernoulli prior over z with standard deviation θz. In various embodiments, other priors may be used as required by a particular application. In several embodiments, β = 0.5 and θx = 0.8.

[0067] In many embodiments, x, w, and τ are solved by finding the maximum a posteriori estimate of the optimal parameters x*, w*, and τ* given by the equations, where the optimal parameters are expressed as:

[0068]

[0069] m(x,w,τ) uses the following formula to obtain the maximum value:

[0070]

[0071] Wherein, Φ(·) is a cumulative normalized normal distribution. In various embodiments, Φ(·) is any sigmoid function. In other embodiments, any function suitable for the requirements of a particular application may be utilized. In many embodiments, maximizing m(x, w, τ) is performed using gradient ascent, alternating between fixing parameters associated with a piece of source data and parameters associated with the annotator. Fixing the parameters in this manner corresponds to initially determining an estimate of the ground truth and the difficulty of describing the source data, using the initial estimate of the ground truth and the difficulty of characterizing the source data to characterize the ability of the annotator, and then updating the estimate of the ground truth and the difficulty of describing the source data by weighting the annotations provided by the annotators. The process of determining the ground truth and the difficulty can be repeated, and these decisions are used to determine the ability of the annotator to update the ground truth until a termination condition is met. In several embodiments, the repetition of gradient ascent is repeated until m(x, w, τ) is determined within a threshold; the threshold can be predetermined or dynamically determined. Other statistical maxima and / or minima may also be used according to implementation forms of the present invention.

[0072] In some embodiments, the sensitivity d′ of the annotator to the noise feature 422 is given by the Mahalanobis distance between μ0 and μ1, normalized by s:

[0073]

[0074] Where d′ is an indicator of the degree of observation of the source data feature 414 by the annotator, μ0 and μ1 are z i =0 and z i = 1 varies greatly:

[0075] P(y ij |z i )=N(y ij |μz,s 2 )

[0076]

[0077] Various other distance determination techniques may be used in accordance with many embodiments of the present invention.

[0078] In some embodiments, the sensitivity of the annotator to the nuisance feature 422 is determined by the equation d′=φ using the false alarm rate f and the hit rate h. -1 (h) = φ -1 (f).

[0079] where Φ -1 is the inverse of the cumulative normal distribution described above, and various distributions may be used. Various other processes may also be used to describe the sensitivity of the tagger to the interference characteristic 422.

[0080] In many embodiments, the error threshold 426 of the annotator is determined by the equation:

[0081]

[0082] Where λ is the error threshold 426. In various embodiments, a large positive λ indicates that the annotator attributes a high cost to false positives, while a large negative λ indicates that the annotator avoids false negatives. In some embodiments, λ is proportional to τ j Regarding, specifically λ=τ j / s.

[0083] The specific process of determining the annotator, source data, and annotation features is described above according to the embodiments of the present invention; however, various processes and statistical techniques may be used according to the embodiments of the present invention. Although the present invention has been described in certain specific aspects, many additional modifications and variations are apparent to those skilled in the art. Therefore, it is understood that the present invention may be practiced beyond the specific description without departing from the scope and spirit of the present invention. Therefore, the embodiments of the present invention should be considered illustrative rather than restrictive. Therefore, the scope of the present invention should not be determined by the illustrated embodiments, but by the attached claims and their equivalents.

[0084] In an embodiment of the present invention, a distributed data labeling method is also provided. The distributed data labeling method is a target data labeling method used to label different characteristic attributes of industry files in the industry file importance identification method. The details can be referred to the above description.

[0085] In the embodiments of the present invention, a device for identifying the importance of industry documents is also provided, and the device is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0086] The industry document importance identification device includes:

[0087] The file acquisition module is used to acquire the target industry file to be identified. The target industry file adopts semi-structured data or unstructured data.

[0088] The feature extraction module is used to construct a question set Q for target industry files, input the question set Q into the large language model, ask the large language model about the characteristic attributes of the target industry files, and obtain the answer set A output by the large language model in response to the question set Q.

[0089] The category matching module is used to match the answer set A in the target feature library and determine the weights of each feature attribute category of the target industry file. The target feature library has a mapping relationship between different feature attributes and different categories. The mapping relationship between different feature attributes and different categories is obtained through the target data labeling method. Different categories are used to represent different degrees of importance.

[0090] The level determination module is used to determine the importance level of the target industry file based on the category weights of each characteristic attribute of the target industry file.

[0091] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0092] In an embodiment of the present invention, a distributed data annotation system is also provided to implement the industry file importance identification method provided by the present invention.

[0093] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0094] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

Claims

1. A method for identifying the importance of industry documents, characterized in that: include: Acquire a target industry file to be identified, where the target industry file uses semi-structured data or unstructured data; Construct a question set Q for the target industry file, input the question set Q into a large language model, ask the large language model questions about characteristic attributes of the target industry file, and obtain an answer set A output by the large language model in response to the question set Q; Matching the answer set A in a target feature library to determine the weights of each feature attribute category of the target industry file, wherein the target feature library has a mapping relationship between different feature attributes and different categories, and the mapping relationship between different feature attributes and different categories is obtained by a target data annotation method, and different categories are used to represent different degrees of importance; The importance level of the target industry file is determined based on the category weights of the characteristic attributes of the target industry file.

2. The method for identifying the importance of industry documents according to claim 1, characterized in that: The target feature library includes a Chinese feature library and / or an English feature library; In the Chinese feature library, the mapping relationship between different feature attributes and different categories is represented by an inverted index; In the English feature library, the mapping relationship between different feature attributes and different categories is represented by a Trie tree.

3. The method for identifying the importance of industry documents according to claim 2, characterized in that: The target data labeling method is implemented by a distributed data labeling server system, and the target data labeling method includes: Receiving one or more target source data, wherein the target source data includes different characteristic attributes of industry files; selecting one or more annotators for one or more of the target source data; Creating one or more of the annotation tasks for the selected one or more of the annotators; Requesting one or more annotations for one or more of the target source data by creating one or more of the annotation tasks; Receiving one or more of the annotations, wherein the annotations are determined according to an industry data classification and grading standard file; Determine metadata of one or more target source data through one or more annotations, wherein the metadata of the target source data includes target source data features; Determining metadata of one or more of the annotators through one or more of the annotations and one or more of the target source data, wherein the metadata of the annotators includes the identified annotator features; The true value of one or more target source data is determined according to metadata of one or more target source data and metadata of one or more taggers.

4. The method for identifying the importance of industry documents according to claim 3, characterized in that: Determining the true value of one or more target source data according to metadata of one or more target source data and metadata of one or more taggers includes: Iteratively update metadata of one or more target source data and metadata of one or more annotators; When the iterative update is terminated, determining the true value of one or more target source data according to the updated metadata of the one or more target source data and the metadata of one or more of the annotators; The iterative updating step includes: Based on the previous metadata of the one or more target source data and the previous metadata of the one or more target source data, update the current metadata of the one or more target source data; Based on the previous metadata of the one or more target source data and the annotation, the current metadata of the one or more annotators is updated.

5. The method for identifying the importance of industry documents according to claim 4, characterized in that: The iterative updating is implemented based on iterative maximum a posteriori estimation.

6. The method for identifying the importance of industry documents according to claim 3, characterized in that: The metadata of the source data includes a true value, source data features, and a record of a previous annotator of the source data; The metadata of the annotator includes interference features, interference feature proportions and error thresholds; The annotation includes annotation data, source data annotation definitions and task data.

7. A distributed data annotation method, characterized in that: The target data labeling method is implemented by a distributed data labeling server system, and the target data labeling method includes: receiving one or more target source data; selecting one or more annotators for one or more of the target source data; Creating one or more of the annotation tasks for the selected one or more of the annotators; Requesting one or more annotations for one or more of the target source data by creating one or more of the annotation tasks; receiving one or more of the annotations; Determine metadata of one or more target source data through one or more annotations, wherein the metadata of the target source data includes target source data features; Determining metadata of one or more of the annotators through one or more of the annotations and one or more of the target source data, wherein the metadata of the annotators includes the identified annotator features; The true value of one or more target source data is determined according to metadata of one or more target source data and metadata of one or more taggers.

8. The distributed data annotation method according to claim 7, characterized in that: Determining the true value of one or more target source data according to metadata of one or more target source data and metadata of one or more taggers includes: Iteratively update metadata of one or more target source data and metadata of one or more annotators; When the iterative update is terminated, determining the true value of one or more target source data according to the updated metadata of the one or more target source data and the metadata of one or more of the annotators; The iterative updating step includes: Based on the previous metadata of the one or more target source data and the previous metadata of the one or more target source data, update the current metadata of the one or more target source data; Based on the previous metadata of the one or more target source data and the annotation, the current metadata of the one or more annotators is updated.

9. A device for identifying the importance of industry documents, characterized in that: include: A file acquisition module, used to acquire a target industry file to be identified, wherein the target industry file adopts semi-structured data or unstructured data; A feature extraction module is used to construct a question set Q for the target industry file, input the question set Q into a large language model, ask the large language model questions about the characteristic attributes of the target industry file, and obtain an answer set A output by the large language model in response to the question set Q; A category matching module is used to match the answer set A in the target feature library to determine the weight of each feature attribute category of the target industry file, wherein the target feature library has a mapping relationship between different feature attributes and different categories, and the mapping relationship between different feature attributes and different categories is obtained by a target data annotation method, and different categories are used to represent different degrees of importance; The level determination module is used to determine the importance level of the target industry file based on the category weights of the various characteristic attributes of the target industry file.

10. A distributed data annotation system, characterized in that: Used to implement the industry document importance identification method described in any one of claims 6-8.

Images