Label evaluation method and device

Abstract

The invention discloses a label evaluation method and device, and the method comprises the steps: constructing an evaluation data table which comprises work order data, a preset correct label set, and a to-be-evaluated label set; performing multi-dimensional evaluation on a to-be-evaluated label based on the work order data and a preset correct label set; determining a service weight corresponding to each evaluation dimension; and based on the business weight, performing weighted fusion calculation on each evaluation dimension score to obtain a comprehensive evaluation score of the to-be-evaluated tag, and generating each evaluation dimension problem detail. In the embodiment of the invention, the evaluation data table is constructed, the service weight weighted comprehensive score of the to-be-evaluated label is evaluated in a multi-dimensional manner, and the problem details are generated. The method realizes automatic quantitative evaluation, reduces subjective deviation and workload, adapts to a large batch of work orders, ensures the accuracy of evaluation results, and solves the problems of mistaken and missed labeling and difficulty in expanded label evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a label evaluation method and device. BACKGROUND

[0002] Currently, in online consultation scenarios such as customer service, it is often necessary to generate a to-be-evaluated label for an online generated work order to mark the core features of the work order, so as to facilitate subsequent distribution and classification for corresponding departments to perform next step processing and other related work. However, the current label evaluation task mainly relies on manual checking and subjective scoring, and lacks efficient and accurate automatic evaluation tools.

[0003] On the one hand, manual evaluation is highly subjective, and evaluation personnel judge the accuracy and completeness of the label according to their personal understanding of the label semantics and business experience, lacking objective text and label correlation quantitative analysis methods. On the other hand, the label system has dynamic expandability, and the pre-defined label set is dynamically updated with the content of the work order. The expanded label lacks clear definition standards, and different evaluation personnel have different judgment angles for the same expanded label, resulting in different scoring scales and making it difficult to form a unified conclusion.

[0004] Moreover, the existing evaluation methods mostly adopt a mode of combining manual spot checks with simple rule checks, which can reduce some repetitive labor, but when faced with a large number of work orders, manual checking of the accuracy and completeness of the label is still required, which not only has limited efficiency improvement, but also makes it difficult for limited manpower to bear, and subjective bias is always difficult to eliminate. The specific manifestations are as follows: manual checking is time-consuming and labor-intensive, and a large amount of manpower is needed to face a large number of work orders, and it is easy to have the problem of wrong labels in which the business semantics deviates from the literal matching; the missing label problem is difficult to identify in advance by automatic means; there is no objective evaluation basis for the relevance and novelty of the expanded label, which is easy to be out of touch with the core needs. SUMMARY

[0005] Therefore, it is necessary to provide a label evaluation method and device to solve at least one problem in the prior art.

[0006] In a first aspect, a label evaluation method is provided, comprising: constructing an evaluation data table, the evaluation data table comprising work order data, a pre-set correct label set, and a to-be-evaluated label set; performing multi-dimensional evaluation on the to-be-evaluated label based on the work order data and the pre-set correct label set to obtain scores of each evaluation dimension, wherein the evaluation dimensions include at least two of accuracy evaluation, completeness evaluation, and expansion evaluation; determining business weights corresponding to each evaluation dimension; perform a weighted fusion calculation on the scores of the evaluation dimensions based on the service weight, to obtain a comprehensive evaluation score of the to-be-evaluated label, and generate an evaluation dimension problem detail, wherein the evaluation dimension problem detail at least includes at least one of a mislabeled label ID, a missed label name, and an invalid extended label reason.

[0007] In a possible implementation, the multi-dimensional evaluation of the to-be-evaluated label based on the work order data and the preset correct label set to obtain the evaluation dimension score includes: calculating a matching depth score and a keyword matching degree score of the to-be-evaluated label and the labels in the preset correct label set as an accuracy score; calculating a coverage rate score and a label co-occurrence correlation score of the to-be-evaluated label in the preset correct label set as an integrity score; if the to-be-evaluated label is an extended label outside the preset correct label set, calculating a semantic correlation score and an extended label novelty score between the to-be-evaluated label and the work order as an extension score.

[0008] In a possible implementation, the matching depth score is calculated by the following method, including: generating semantic vectors of each to-be-evaluated label in the to-be-evaluated label set and semantic vectors of the labels in the preset correct label set according to a preset vector dimension based on label semantics; calculating attention weights of the to-be-evaluated labels and the labels in the corresponding preset correct label set based on the correlation between the semantic vectors of the to-be-evaluated labels and the semantic vectors of the corresponding preset correct labels; obtaining a bidirectional matching depth score based on the average of the attention-weighted semantic similarity between all to-be-evaluated labels and corresponding preset correct labels.

[0009] In a possible implementation, the keyword matching degree score is calculated by the following method: obtaining text content of the work order corresponding to the to-be-evaluated label; performing word segmentation processing on the text content, and selecting words exceeding a preset threshold to form a keyword set; counting the keyword set commonly contained in the to-be-evaluated label and the preset correct label set, and a total keyword set; calculating a first bidirectional attention weight of each keyword in the keyword set and the corresponding to-be-evaluated label, and a second bidirectional attention weight of the keyword and the corresponding preset correct label; calculating the keyword matching degree score based on the first bidirectional attention weight and the second bidirectional attention weight.

[0010] In a possible implementation, the coverage score is obtained in the following manner: Filtering tags in the to-be-evaluated tags that match tags in the preset correct label set to form a covered correct label set; Based on the relevance between the semantic vector of the work order corresponding to the to-be-evaluated tag and the semantic vector of each tag in the preset correct label set, the importance of each tag in the preset correct label set to the work order corresponding to the to-be-evaluated tag is calculated through an attention mechanism; The sum of the first importance of all tags in the covered correct label set and the sum of the second importance of all tags in the preset correct label set are calculated; Based on the ratio of the sum of the first importance and the sum of the second importance, the coverage score is calculated.

[0011] In a possible implementation, the tag co-occurrence relevance score is calculated in the following manner: Filtering tags in the to-be-evaluated tags that match tags in the preset correct label set to form a covered correct label set; Converting the text content of the work order corresponding to the to-be-evaluated tag into a work order semantic vector, and converting each tag in the preset correct label set into a corresponding tag semantic vector; Fusing the semantic vector of any tag in the preset correct label set and the work order semantic vector as a query vector, and fusing the semantic vector of the remaining tags in the preset correct label set and the work order semantic vector as a key vector, and calculating the relevance weight between tags through a self-attention mechanism; Selecting any one tag from the covered correct label set, and filtering the preset correct labels whose relevance weight exceeds a preset threshold, to form a dynamic associated label set of the work order corresponding to the to-be-evaluated tag; Based on the dynamic associated label set, an attention-weighted co-occurrence relevance score is calculated.

[0012] In a possible implementation, the semantic relevance score is obtained in the following manner: Converting the text content of the work order corresponding to the to-be-evaluated tag into a work order semantic vector, and converting the expanded label into an expanded label semantic vector; Based on the work order semantic vector and the expanded label semantic vector, the attention weight and the cosine similarity of the expanded label to the key information of the work order text are calculated; Based on the attention weight and the cosine similarity, the attention degree of each expanded label to the key information of the work order text is calculated; The individual relevance degrees of all expanded labels are averaged to calculate the overall semantic relevance score.

[0013] In a possible implementation, the novelty score of the extended label is calculated by the following method: cosine similarity between each extended label and the labels in the preset correct label set is calculated; the novelty score of each extended label is obtained based on the cosine similarity; the novelty scores of all the extended labels are averaged to obtain the overall novelty score.

[0014] In a possible implementation, after the comprehensive evaluation score of the label to be evaluated is obtained, the following steps are included: verification is performed on the comprehensive evaluation score by randomly selecting the label to be evaluated and performing manual re-evaluation to obtain a manual re-evaluation score; a correlation coefficient between the manual re-evaluation score and the comprehensive evaluation score is calculated; if the correlation coefficient is greater than or equal to a preset threshold, it is confirmed that the comprehensive evaluation score and each dimension problem detail are valid.

[0015] In a second aspect, a label evaluation device is provided, which includes: an evaluation data set construction unit configured to construct an evaluation data table, the evaluation data table including work order data, a preset correct label set, and a label set to be evaluated; a multi-dimensional evaluation unit configured to perform multi-dimensional evaluation on the label to be evaluated based on the work order data and the preset correct label set to obtain scores of each evaluation dimension, wherein the evaluation dimensions include at least two of accuracy evaluation, integrity evaluation, and expansion evaluation; a business weight determination unit configured to determine business weights corresponding to each evaluation dimension; a comprehensive evaluation result generation unit configured to perform weighted fusion calculation on the scores of each evaluation dimension based on the business weights to obtain a comprehensive evaluation score of the label to be evaluated, and to generate each evaluation dimension problem detail, wherein the each dimension problem detail includes at least one of a wrong label ID, a missing label name, and an invalid extended label reason.

[0016] The label evaluation method and device, the method implementation thereof, include: constructing an evaluation data table including work order data, a preset correct label set, and a set of labels to be evaluated; performing multi-dimensional evaluation on the labels to be evaluated based on the work order data and the preset correct label set, wherein the evaluation dimensions include at least two of accuracy evaluation, integrity evaluation, and expansibility evaluation; determining the business weights corresponding to each evaluation dimension; performing weighted fusion calculation on the scores of the evaluation dimensions based on the business weights to obtain the comprehensive evaluation score of the labels to be evaluated, and generating the dimension-by-dimension problem details, wherein the dimension-by-dimension problem details include at least one of at least one of the wrong label ID, the missing label name, and the invalid expansion label reason. In the embodiment of the present application, the problems of relying on manual work, strong subjectivity, and low efficiency in the existing evaluation are solved, and a large amount of manual checking is replaced by automatic quantitative evaluation, which greatly reduces subjective bias and workload and efficiently adapts to large quantities of work order evaluation requirements. Through multi-dimensional evaluation, the core evaluation requirements of label accuracy, integrity, and expansibility are comprehensively covered, the dynamic adjustment of the business weights makes the comprehensive score more suitable for actual business scenarios, and the problem details can quickly locate label defects. In addition, the manual re-evaluation verification mechanism is used to further ensure the objectivity and accuracy of the evaluation results, effectively make up for the shortcomings of the existing simple rule checking, such as difficult identification of wrong labels and missing labels and lack of basis for expansion label evaluation, and provide scientific and reliable technical support for work order label quality control. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of a label evaluation method in an embodiment of the present application; Figure 2 is a structural diagram of a label evaluation device in an embodiment of the present application; Figure 3 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0020] In an embodiment, as shown in Figure 1 A label evaluation method is provided, comprising the following steps: In step S110, an evaluation data table is constructed, which includes work order data, a preset correct label set, and a label set to be evaluated; Optionally, a work order table is collected, which can include work order number, submission time, appeal text, and region to which the work order belongs, as the core text basis for label evaluation. Similarly, a label data table can be collected, which can include corresponding work order number, system-generated label set to be evaluated, and preset correct label set, as a comparison benchmark for evaluating label quality. Finally, the two tables are associated with the work order number as the primary key, and the core fields such as work order number, appeal text, region to which the work order belongs, label set to be evaluated, and preset correct label set are extracted and integrated to ensure that each work order can be accurately matched with the corresponding label, providing complete and related data support for subsequent multi-dimensional quantitative evaluation.

[0021] Among them, the label set to be evaluated is automatically generated by the system for a specific work order, and is used to mark the label combination of the core features of the work order (such as the type of appeal and the problem area). For example, after a user submits a work order (such as customer service consultation), the label generation system (usually based on natural language processing technology) will automatically analyze the appeal text of the work order, and then extract core information (such as property maintenance and water fee inquiry) from the work order text through keyword matching, semantic vector classification, and pre-trained model prediction, forming a label combination corresponding to the work order, which is the label set to be evaluated.

[0022] Among them, the preset correct label set is a standard label combination pre-set for a certain type of work order based on business rules, historical experience or expert knowledge, and is the core benchmark for judging whether the label to be evaluated is accurate and complete. For example, high-quality labels that are accurately labeled by humans and can effectively support distribution classification can be sorted and counted in historical work orders, classified by work order type (such as consultation and suggestion) or problem area (such as water, electricity, gas, property management, traffic and travel), and a standard label library corresponding to each type of work order is formed. The subset of the label library corresponding to the type to which a work order belongs is the preset correct label set of the work order. Alternatively, business personnel can also develop clear label rules based on business needs, and then have them reviewed and confirmed by domain experts to ensure the accuracy and applicability of the labels, and finally form the preset correct label set.

[0023] In step S120, the label to be evaluated is evaluated in multiple dimensions based on the work order data and the preset correct label set, wherein the evaluation dimensions include at least two of accuracy evaluation, completeness evaluation, and expansibility evaluation; Optionally, the work order data is anchored as a real business scenario, and a preset correct label set is used as a standard reference. The quality of the to-be-evaluated label is quantified from different quality dimensions (at least two dimensions of accuracy, integrity, and expansibility). The multi-dimensional evaluation can comprehensively and accurately reflect the quality status of the to-be-evaluated label, and ensure that it meets the business needs.

[0024] The accuracy evaluation is used to measure the degree of fit between the to-be-evaluated label and the preset correct label. By calculating the matching depth between the to-be-evaluated label and the preset correct label, and the keyword matching degree of the two, two accuracy scores are obtained to determine whether the to-be-evaluated label of the system meets the preset correct label in terms of semantics and key information, and to ensure that the label accurately reflects the corresponding content.

[0025] The integrity evaluation is used to reflect the coverage and association of the to-be-evaluated label to the preset correct label. By calculating the coverage rate of the to-be-evaluated label of the system in the preset correct label and the co-occurrence association score between the labels, two integrity scores are obtained to determine whether the to-be-evaluated label of the system comprehensively covers the key part in the preset correct label and whether the association between the labels meets the expectation, and to ensure that the label can present the relevant information completely.

[0026] The expansibility evaluation is for the expansion label in the to-be-evaluated label that exceeds the range of the preset correct label. The semantic relevance of these expansion labels to the work order and the novelty score of the expansion labels are calculated to obtain two expansion scores to evaluate whether the expansion label is closely related to the work order content and whether it has novel value, and to ensure that the expansion label can bring meaningful supplement to the label system.

[0027] In step S130, the business weight corresponding to each evaluation dimension is determined. Optionally, since the business scenarios and focus points carried by different work orders are different, the attention and requirements for each evaluation dimension (accuracy, integrity, expansibility, etc.) of the label are also different. For example, for some work orders whose core business is to accurately identify known problem types, the importance of the accuracy dimension will be higher, and the corresponding business weight will be larger. For work orders that need to explore new business points and expand service boundaries, the business weight of the expansibility dimension may be more prominent. By combining the specific business attributes and demand characteristics of the to-be-evaluated label corresponding work order, an appropriate business weight is assigned to each evaluation dimension, which can make the subsequent comprehensive evaluation score more in line with the actual business needs and more accurately reflect the value and quality of the label in this business scenario.

[0028] In step S140, a weighted fusion calculation is performed on the scores of the evaluation dimensions based on the business weights to obtain a comprehensive evaluation score of the to-be-evaluated label, and a dimension-by-dimension problem detail is generated, wherein the dimension-by-dimension problem detail at least includes at least one of a mislabeled label ID, a missed label name, and an invalid extended label reason.

[0029] Optionally, a weighted fusion calculation is performed on the scores of the evaluation dimensions (accuracy, completeness, and extension scores) based on the business weights, that is, a score weighted summation is performed according to the importance (business weight) of each dimension in the current business scenario, to obtain a comprehensive evaluation score of the to-be-evaluated label. At the same time, a dimension-by-dimension problem detail is generated synchronously in the calculation process, such as at least one of a mislabeled label ID, a missed label name, and an invalid extended label reason. And the label defects found in the evaluation are accurately positioned: the mislabeled label ID is used to identify the labels that deviate too much from the preset correct labels in the accuracy dimension; the missed label name is used to record the preset correct labels that are not covered by the to-be-evaluated label in the completeness dimension; and the invalid extended label reason is used to explain the new labels that do not meet the requirements in the extension dimension (such as irrelevant to the work order semantics, lack of novelty, etc.).

[0030] In the embodiment of the present application, a label evaluation method is provided, which includes: constructing an evaluation data table, the evaluation data table including work order data, a preset correct label set, and a to-be-evaluated label set; performing multi-dimensional evaluation on the to-be-evaluated label based on the work order data and the preset correct label set, wherein the evaluation dimensions include at least two of accuracy evaluation, completeness evaluation, and extension evaluation; determining the business weights corresponding to the evaluation dimensions; and performing a weighted fusion calculation on the scores of the evaluation dimensions based on the business weights to obtain a comprehensive evaluation score of the to-be-evaluated label, and generating a dimension-by-dimension problem detail, wherein the dimension-by-dimension problem detail at least includes at least one of a mislabeled label ID, a missed label name, and an invalid extended label reason. In the embodiment of the present application, the problems of existing evaluation relying on manual work, strong subjectivity, and low efficiency are solved, a large amount of manual checking is replaced by automatic quantitative evaluation, subjective bias and workload are greatly reduced, and large quantities of work order evaluation requirements are efficiently adapted; through multi-dimensional evaluation, core evaluation requirements such as label accuracy, completeness, and extension are comprehensively covered, the comprehensive score is more consistent with the actual business scenario through dynamic adjustment of the business weights, and the problem details can quickly locate label defects; in addition, in cooperation with an artificial reevaluation verification mechanism, the objectivity and accuracy of the evaluation results are further ensured, the shortcomings of existing simple rule checking, such as difficulty in identifying mislabeled and missed labels and lack of basis for evaluating extended labels, are effectively made up, and scientific and reliable technical support is provided for work order label quality control.

[0031] In an embodiment of the present application, the multi-dimensional evaluation of the to-be-evaluated label based on the work order data and the preset correct label set obtains evaluation dimension scores, including: The matching depth score and the keyword matching degree score of the to-be-evaluated label and the labels in the preset correct label set are calculated as the accuracy score; The coverage score and the label co-occurrence correlation score of the to-be-evaluated label in the preset correct label set are calculated as the integrity score; If the to-be-evaluated label is an extended label outside the preset correct label set, the semantic correlation score and the extended label novelty score between the to-be-evaluated label and the work order are calculated as the expansibility score.

[0032] Optionally, the accuracy score can include the matching depth score and the keyword matching degree score; the integrity score can include the coverage score and the label co-occurrence correlation score; and the expansibility score can include the semantic correlation score and the extended label novelty score. By calculating the matching depth score and the keyword matching degree score, the label accuracy can be evaluated from two aspects of semantic deep correlation and explicit key information overlap, avoiding single dimension misjudgment; by calculating the coverage score and the label co-occurrence correlation score, the core label coverage and the logical rationality between labels can be considered, so that the integrity evaluation is more in-depth; by calculating the semantic correlation score and the extended label novelty score, the relevance and uniqueness of the extended label can be double-checked to filter invalid innovations.

[0033] The matching depth score is obtained by converting the to-be-evaluated label and the preset correct label into semantic vectors, calculating the attention weighted similarity average between the vectors, and measuring the deep matching degree of the label semantic level; the keyword matching degree score is obtained by counting the keywords included in the to-be-evaluated label and the preset correct label after the keyword set of the work order text is obtained by word segmentation, and combining the bidirectional attention weight between the keywords and the labels, which reflects the fitting degree of the label on the key information.

[0034] The coverage score is obtained by screening the part of the to-be-evaluated label matching the preset correct label to form a covered correct label set, calculating the importance of each preset correct label to the work order based on the semantic vector association degree of the work order and the preset correct label through an attention mechanism, and finally determining the ratio of the sum of the importance of the covered correct label to the sum of the importance of all preset correct labels to reflect the coverage of the to-be-evaluated label to the core correct label. The label co-occurrence association score is obtained by fusing the semantic vectors of the work order and the preset correct label after obtaining the covered correct label set, calculating the association weight between labels using a self-attention mechanism, screening the preset correct label with an association weight with the covered correct label exceeding a preset threshold to form a dynamic association label set, and then calculating the attention-weighted co-occurrence association score for evaluating the rationality and closeness of the association between labels.

[0035] The semantic relevance score is obtained by converting the work order text and the extended label into semantic vectors, calculating the attention weight and cosine similarity of the extended label to the key information of the work order text, combining the two to obtain the attention degree of each extended label to the key information of the work order, and finally taking the average of the attention degrees of all extended labels to determine the association degree of the extended label to the core content of the work order. The novelty score of the extended label is obtained by calculating the cosine similarity of each extended label to the labels in the preset correct label set, taking the maximum cosine similarity of each extended label to the preset correct label, subtracting the maximum value from 1 to obtain the novelty score of each extended label, and finally taking the average of the novelty scores of all extended labels to evaluate the novelty and value of the extended label.

[0036] In an embodiment of the present application, the matching depth score is calculated by the following method, comprising: generating semantic vectors of each to-be-evaluated label in the to-be-evaluated label set and semantic vectors of labels in the preset correct label set according to a preset vector dimension based on label semantics; calculating attention weights of each to-be-evaluated label and labels in the corresponding preset correct label set based on the association degree between the semantic vector of each to-be-evaluated label and the semantic vector of the corresponding preset correct label; obtaining a bidirectional matching depth score based on the average of the attention-weighted semantic similarity of all to-be-evaluated labels and corresponding preset correct labels.

[0037] It should be noted that the matching depth score is calculated by the bidirectional depth matching strategy. Since the preset correct label may contain the abstract classification specified by the business, and the label to be evaluated is mostly a specific scene description. One-way calculation may over-rely on the specific features of the label to be evaluated and ignore the abstract business connotation of the correct label; bidirectional matching allows both parties to pay attention to each other, considering both the scene details of the label to be evaluated and the business classification logic of the correct label, avoiding the deviation caused by single perspective, and making the accuracy score more in line with the dual standards of business needs and scene reality.

[0038] Optionally, by using a pre-trained model such as BERT, the label to be evaluated and the preset correct label are both converted into semantic vectors according to a preset vector dimension, so that the text label is changed into a calculable mathematical form; then the correlation (such as cosine similarity) of the semantic vectors of the two types of labels is calculated, and the correlation can be converted into an attention weight, according to which a higher attention weight is assigned to a closely related label pair (such as "cell water pipe burst"), reducing the interference of weakly related labels; finally, the semantic similarity of all label pairs is weighted and calculated using the attention weight, and a bidirectional matching depth score is obtained, so as to reflect the deep matching situation of the overall label combination, rather than the performance of a single label.

[0039] Specifically, d can be a vector dimension, the set of labels to be evaluated is defined as , There are n labels, and the semantic vector of each label to be evaluated is ; the set of preset correct labels is , and there are m labels, and the semantic vector of each preset correct label is .

[0040] Then, the attention weight of the label to be evaluated to the preset correct label is calculated, as shown in the following formula: ; Wherein, exp represents an exponential function, T represents a transpose, , which is a common operation in attention mechanism to prevent gradient vanishing or explosion. The above formula measures the attention degree of each label to be evaluated to each correct label , and the symmetry measures the attention degree of each correct label to each label to be evaluated .

[0041] The attention weights in both directions are averaged to obtain the final bidirectional attention weight, as shown in the following formula: ; Finally, the average of the attention-weighted semantic similarities between all the to-be-evaluated labels and the corresponding correct labels is taken to obtain a bidirectional matching depth score, as shown in the following formula: .

[0042] In an embodiment of the present application, the keyword matching degree score is obtained by the following method: Obtaining the text content of the work order corresponding to the to-be-evaluated label; Performing word segmentation processing on the text content, and selecting words exceeding a preset threshold to form a keyword set; Counting the keyword set commonly contained in the to-be-evaluated label and the preset correct label set, and the total keyword set; Calculating the first bidirectional attention weight of each keyword in the keyword set and the corresponding to-be-evaluated label, and the second bidirectional attention weight of each keyword and the corresponding preset correct label; Based on the first bidirectional attention weight and the second bidirectional attention weight, the keyword matching degree score is calculated.

[0043] It should be noted that for accuracy evaluation, in addition to the overall similarity between labels for accuracy calculation, the ability of the system to capture accurate keywords of the to-be-evaluated label also needs to be quantified, that is, whether the system to-be-evaluated label captures the key information in the text of appeal, and whether the capture of these key information relies on brute force enumeration and other inappropriate methods. For the former, the common ability of the system to-be-evaluated label and the preset correct label to capture keywords in the text of appeal is reflected; for the latter, the penalty for the system to-be-evaluated label capturing too many keywords is reflected.

[0044] The keywords exceeding the preset threshold can be extracted from the text of the work order corresponding to the to-be-evaluated label to form a set, and then the common keywords of the to-be-evaluated label and the preset correct label and the total keyword set are counted. By calculating the bidirectional attention weight of each keyword and the two types of labels (reflecting the importance of the keyword to the label and the dependence of the label on the keyword), the keyword matching degree score is finally obtained by weighted fusion based on these weights. Both the core information of the work order is anchored by keyword extraction, and the key association is highlighted by the bidirectional attention mechanism, which accurately quantifies the degree of fit between the to-be-evaluated label and the preset correct label in the explicit key information layer.

[0045] Specifically, the text content of the appeal of the work order corresponding to the to-be-evaluated label is extracted, and after word segmentation, the keywords exceeding the threshold are selected according to the size of the TF-IDF value to form a keyword set .

[0046] Then, based on the keyword set, the keyword set commonly contained in the to-be-evaluated label and the correct label is counted and the total keyword set of both It should be noted that when judging whether to contain keywords, a soft intersection calculation based on semantic similarity can be performed, that is, the candidate words and the label words are converted into semantic vectors of the same dimension through a pre-trained model, the similarity between each pair of vectors is calculated, and the keywords that exceed the threshold are determined as semantically related keywords, and then the common related keyword set and the total keyword set of the to-be-evaluated label and the preset correct label are counted. In addition, the field dictionary matching method, the TF-IDF statistical method, etc. can also be used to judge the keyword containing relationship.

[0047] Then, the bidirectional attention weight of each keyword with respect to the to-be-evaluated label corresponding to the keyword and the bidirectional attention weight of each keyword with respect to the correct label corresponding to the keyword

[0048] Finally, the keyword matching degree score is calculated, which is shown in the following formula: .

[0049] In an embodiment of the present application, the coverage score is obtained in the following manner: Filtering the labels in the to-be-evaluated label set that match the labels in the preset correct label set to form a covered correct label set; Based on the correlation between the semantic vector of the work order corresponding to the to-be-evaluated label and the semantic vector of each label in the preset correct label set, the importance of each label in the preset correct label set to the work order corresponding to the to-be-evaluated label is calculated through an attention mechanism; Calculating the sum of the first importance of all labels in the covered correct label set and the sum of the second importance of all labels in the preset correct label set; Based on the ratio of the sum of the first importance to the sum of the second importance, the coverage score is calculated.

[0050] It should be noted that when performing integrity evaluation, the coverage degree of the to-be-evaluated label of the system to the preset correct label set is quantified to reflect whether the core mandatory label is missed. It should be noted that some labels in the preset correct label set are more critical to the semantics of the current work order, and higher weight should be given through attention to avoid the problem that covering secondary labels but missing core labels still gets a high score.

[0051] ​​​In the calculation of the coverage score, first, the labels matching the preset correct label set are screened out from the to-be-evaluated labels to form the covered correct label set; then, by means of the semantic vectors of the to-be-evaluated label corresponding work order and the preset correct label, the importance of each label in the preset correct label set to the to-be-evaluated label corresponding work order is calculated by using the attention mechanism; then, the sum of the first importance of all labels in the covered correct label set and the sum of the second importance of all labels in the preset correct label set are calculated respectively; finally, the coverage score is obtained by the ratio of the sum of the first importance and the sum of the second importance, so as to measure the coverage level of the to-be-evaluated label to the key part in the preset correct label, and the importance difference of each label in the work order scene is considered, so that the evaluation is more accurate and closer to the actual business.

[0052] Specifically, first, the labels belonging to the preset correct label set can be screened out from the to-be-evaluated labels by string matching or semantic similarity, and is obtained, the size of which is . Among them, string matching refers to consistency comparison of the text content of the label, for example, if the to-be-evaluated label contains “water pressure failure”, and the correct label also contains “water pressure failure”, then the label will be screened into , if the label in the correct label is “water pressure problem”, it will not be screened into . When the label texts are not completely consistent but the semantics are similar, (such as “water supply anomaly” in the to-be-evaluated label and “water pressure failure” in the correct label are semantically related), the labels are converted into semantic vectors by a pre-trained language model (such as BERT), and the similarity (such as cosine similarity) between the vectors is calculated. If the similarity exceeds a preset threshold (such as 0.8), it is determined that the to-be-evaluated label belongs to the preset correct label set and is included in .

[0053] Then, the importance of each preset label to the current work order can be calculated by the attention mechanism, as shown in the following formula: ; Among them, is the semantic vector of the to-be-evaluated label corresponding work order text, is the semantic vector of the preset correct label.

[0054] Finally, the weighted coverage score can be calculated by the following formula: ; The formula shows that the coverage score depends not only on whether it is covered, but also on the importance of the covered label.

[0055] In an embodiment of the present application, the label co-occurrence association score is calculated by the following method: Filtering labels matching the labels in the preset correct label set from the to-be-evaluated labels to form a covered correct label set; Converting the text content of the work order corresponding to the to-be-evaluated label into a work order semantic vector, and converting each label in the preset correct label set into a corresponding label semantic vector; Fusing the semantic vector of any label in the preset correct label set with the work order semantic vector as a query vector, fusing the semantic vectors of the remaining labels in the preset correct label set with the work order semantic vector as a key vector, and calculating the association weight between the labels by self-attention mechanism; Selecting any one label from the covered correct label set, filtering the preset correct labels with an association weight exceeding a preset threshold, and grouping all the filtered preset correct labels into a dynamic association label set of the work order corresponding to the to-be-evaluated label; Based on the dynamic association label set, an attention-weighted co-occurrence association score is calculated.

[0056] It should be noted that when performing integrity evaluation, in addition to the coverage score described above, the co-occurrence association score is also included. Since the association strength between the preset correct labels is not fixed, the label association strength in the current work order scenario needs to be dynamically calculated by self-attention, and the missing associated labels are considered when calculating the score. That is, if the system to-be-evaluated label captures a correct label, but does not capture other correct labels strongly associated with the correct label, the co-occurrence association score will be reduced.

[0057] When calculating the label co-occurrence association score, first, filter the labels matching the preset correct label set from the to-be-evaluated labels to form a covered correct label set; then convert the text content of the work order corresponding to the to-be-evaluated label into a work order semantic vector, and convert each label in the preset correct label set into a corresponding label semantic vector; subsequently, fuse the semantic vector of any label in the preset correct label set with the work order semantic vector as a query vector, and fuse the semantic vectors of the remaining labels with the work order semantic vector as a key vector, and calculate the association weight between the labels by self-attention mechanism; then, select any one label from the covered correct label set, filter the preset correct labels with an association weight exceeding a preset threshold, and group all the filtered preset correct labels into a dynamic association label set of the work order corresponding to the to-be-evaluated label; finally, based on the dynamic association label set, an attention-weighted co-occurrence association score is calculated to evaluate the association closeness of the covered correct label and other related preset correct labels, and to reflect the rationality of the logical association between the labels.

[0058] Specifically, first, take the preset correct label as the query, and take the other preset correct labels is a key, the self-attention association weight between the labels is calculated, which can be represented by the following formula: ; wherein the text vector of the work order appeal corresponding to the to-be-evaluated label is integrated .

[0059] For the covered label , all labels in the preset correct label set are traversed , and the association weight calculated by the self-attention mechanism is viewed . If it exceeds the threshold, the preset correct label exceeding the pre-set threshold is screened out , and a dynamic association label set under the current work order is constituted . In this way, other preset correct labels closely associated with the covered correct label are mined.

[0060] Finally, the co-occurrence association score weighted by attention can be calculated by the following formula: ; Therefore, the association score depends not only on whether the associated label is covered, but also on the importance and association strength of the associated label.

[0061] In an embodiment of the present application, the semantic relevance score is obtained in the following manner: The text content of the work order corresponding to the to-be-evaluated label is converted into a work order semantic vector, and the expansion label is converted into an expansion label semantic vector; Based on the work order semantic vector and the expansion label semantic vector, the attention weight and the cosine similarity of the expansion label to the key information of the work order text are calculated; Based on the attention weight and the cosine similarity, the attention degree of each expansion label to the key information of the work order text is calculated; The overall semantic relevance score is calculated by averaging the single relevance of all expansion labels.

[0062] It should be noted that when performing expansion evaluation, the semantic relevance between the expansion label and the work order can be calculated to determine whether the expansion label is related to the core semantics of the work order text, thereby avoiding generating irrelevant labels. For example, the label generated by the system is expanded to a greening problem when the work order corresponding to the to-be-evaluated label is around elevator failure.

[0063] In the calculation of the semantic correlation score, the text content of the work order corresponding to the to-be-evaluated label can be first converted into a work order semantic vector, and the extended label is converted into an extended label semantic vector; then, based on the two types of semantic vectors, the attention weight of the extended label to the key information of the work order text and the cosine similarity between the two are calculated; the attention weight and the cosine similarity are combined to further calculate the attention degree of each extended label to the key information of the work order text (i.e., the correlation degree of a single extended label); finally, the correlation degrees of all the extended labels are averaged to obtain the overall semantic correlation score, so as to quantify the close degree of the extended label and the core content of the work order, and ensure that the extended label does not deviate from the actual scene of the work order.

[0064] Specifically, first, for the extended label set generated by the system and not in the preset correct label set , the association between the extended label and the core semantics of the work order is calculated, which can be calculated by the following formula: ; The text attention weight can be calculated by the following formula: ; which is used to measure the attention degree of the extended label to the key information of the work order text.

[0065] Then, the correlation degrees of all the extended labels are averaged to calculate the overall semantic correlation score, which can be calculated by the following formula: ; Since contains , this item is the attention weighted average of the similarity between the essential extended label and the work order complaint text corresponding to the to-be-evaluated label.

[0066] In an embodiment of the present application, the novelty score of the extended label is calculated by the following method: Calculate the cosine similarity between each extended label and the labels in the preset correct label set; Based on the cosine similarity, obtain the novelty score of each extended label; Take the average of the novelty scores of all the extended labels to obtain the overall novelty score.

[0067] It should be noted that in the extension evaluation, in addition to the above-mentioned semantic correlation score, the novelty score also needs to be calculated. By calculating the novelty score, it can be determined whether the extended label is an effective supplement to the preset correct label set, so as to avoid repeated preset labels or high semantic coincidence with the preset labels.

[0068] ​The calculation of the novelty score of the extended label is performed. First, the cosine similarity between each extended label and all labels in the preset correct label set is calculated to measure the semantic coincidence degree of the extended label and the existing correct label. Then, the novelty score of the single extended label is determined based on the cosine similarity. Finally, the average value of the single novelty score of all extended labels is taken to obtain the overall novelty score. The score can intuitively reflect the performance of the extended label in terms of departing from the preset correct label system and having uniqueness, and avoid meaningless extension that is repeated with the existing correct label.

[0069] Specifically, first, the novelty of the extended label is calculated. The novelty calculation is performed based on the cosine similarity. The maximum cosine similarity between each extended label and all labels in the preset correct label set is taken to obtain the novelty score of each extended label (the lower the similarity, the higher the novelty), which can be specifically represented by the following formula: ; Wherein the operator takes the maximum cosine similarity between the extended label and all preset correct labels .

[0070] Finally, the novelty of all extended labels is averaged, which can be specifically represented by the following formula: ; Wherein, if there is no extended label, the default score is 0.

[0071] In an embodiment of the present application, after obtaining the comprehensive evaluation score of the label to be evaluated, the following steps are included: Performing verification on the comprehensive evaluation score, obtaining an artificial reevaluation score by randomly extracting the label to be evaluated and performing artificial reevaluation; Calculating the correlation coefficient between the artificial reevaluation score and the comprehensive evaluation score; If the correlation coefficient is greater than or equal to a preset threshold, it is confirmed that the comprehensive evaluation score and each dimension problem detail are valid.

[0072] Optionally, after obtaining the comprehensive evaluation score of the tag to be evaluated, the validity of the score needs to be verified by manual review. The specific process can be as follows: First, randomly select samples from the tags to be evaluated, and have people re-evaluate the quality (such as accuracy, completeness, etc.) of these tags and give a manual review score; then, by calculating the correlation coefficient (such as Pearson correlation coefficient) between the manual review score and the comprehensive evaluation score generated by the system, the consistency between the two is quantified. The higher the correlation coefficient, the higher the degree of fit between the system score and the human subjective judgment; if the calculated correlation coefficient is greater than or equal to a preset threshold (such as 0.8), it can be confirmed that the current comprehensive evaluation score can accurately reflect the true quality of the tag to be evaluated, and also prove that the problem details (such as missing tags, incorrect tag positioning results, etc.) of each dimension (such as matching depth, keyword matching degree, etc.) are valid and can be used as a reliable basis for subsequent tag optimization.

[0073] This application addresses the problems of existing assessments being reliant on manual labor, highly subjective, and inefficient. It replaces extensive manual verification with automated quantitative assessment, significantly reducing subjective bias and workload, and efficiently adapting to the assessment needs of large-volume work orders. Furthermore, it comprehensively covers core assessment requirements such as label accuracy, completeness, and scalability through multi-dimensional assessment. Combined with dynamic adjustments to business weights, the overall score more closely reflects actual business scenarios, and the problem details can quickly pinpoint label defects. In addition, the inclusion of a manual review and verification mechanism further ensures the objectivity and accuracy of the assessment results, effectively compensating for the shortcomings of existing simple rule-based verification, such as the difficulty in identifying mislabeled or missing labels and the lack of basis for assessing extended labels. This provides scientific and reliable technical support for work order label quality control.

[0074] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0075] In one embodiment, a label evaluation apparatus is provided, which corresponds one-to-one with the label evaluation methods described in the above embodiments. For example... Figure 2 As shown, the label evaluation device includes an evaluation dataset construction unit 10, a multi-dimensional evaluation unit 20, a business weight determination unit 30, and a comprehensive evaluation result generation unit 40. Detailed descriptions of each functional module are as follows: The evaluation dataset construction unit 10 is used to construct an evaluation data table, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated. The multi-dimensional evaluation unit 20 is used to perform multi-dimensional evaluation on the tags to be evaluated based on the work order data and the preset set of correct tags, and obtain scores for each evaluation dimension. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation. The service weight determination unit 30 is configured to determine service weights corresponding to the evaluation dimensions; The comprehensive evaluation result generation unit 40 is configured to perform weighted fusion calculation on the scores of the evaluation dimensions based on the service weights, to obtain a comprehensive evaluation score of the to-be-evaluated label, and to generate a dimension-by-dimension problem detail of the to-be-evaluated label, wherein the dimension-by-dimension problem detail at least includes at least one of a mislabeled label ID, a missed label name, and a reason for invalid expansion label.

[0076] In an embodiment of the present application, the multi-dimensional evaluation unit 20 is further configured to: calculate a matching depth score and a keyword matching degree score of the to-be-evaluated label and the labels in the preset correct label set as an accuracy score; calculate a coverage rate score and a label co-occurrence correlation score of the to-be-evaluated label in the preset correct label set as an integrity score; if the to-be-evaluated label is an expansion label outside the preset correct label set, calculate a semantic correlation score and an expansion label novelty score between the to-be-evaluated label and the work order as an expansion score.

[0077] In an embodiment of the present application, the matching depth score is calculated by the following method, comprising: generating semantic vectors of each to-be-evaluated label in the to-be-evaluated label set and semantic vectors of the labels in the preset correct label set according to a preset vector dimension based on label semantics; calculating attention weights of the to-be-evaluated labels and the labels in the corresponding preset correct label set based on the correlation between the semantic vectors of the to-be-evaluated labels and the semantic vectors of the corresponding preset correct labels; obtaining a bidirectional matching depth score based on the average of the attention-weighted semantic similarity between all to-be-evaluated labels and corresponding preset correct labels.

[0078] In an embodiment of the present application, the keyword matching degree score is calculated by the following method: obtaining text content of the work order corresponding to the to-be-evaluated label; performing word segmentation processing on the text content, selecting words exceeding a preset threshold to form a keyword set; counting the keyword set commonly contained in the to-be-evaluated label and the preset correct label set, and the total keyword set; calculating a first bidirectional attention weight of each keyword in the keyword set and the corresponding to-be-evaluated label, and a second bidirectional attention weight of the keyword and the corresponding preset correct label; calculating the keyword matching degree score based on the first bidirectional attention weight and the second bidirectional attention weight.

[0079] In an embodiment of the present application, the coverage score is obtained by the following way: Filtering the tags in the to-be-evaluated tags that match the tags in the preset correct label set to form a covered correct label set; Based on the relevance between the semantic vector of the work order corresponding to the to-be-evaluated tag and the semantic vector of each tag in the preset correct label set, the importance of each tag in the preset correct label set to the work order corresponding to the to-be-evaluated tag is calculated through an attention mechanism; The sum of the first importance of all tags in the covered correct label set and the sum of the second importance of all tags in the preset correct label set are calculated; Based on the ratio of the sum of the first importance and the sum of the second importance, the coverage score is calculated.

[0080] In an embodiment of the present application, the tag co-occurrence relevance score is calculated by the following way: Filtering the tags in the to-be-evaluated tags that match the tags in the preset correct label set to form a covered correct label set; Converting the text content of the work order corresponding to the to-be-evaluated tag into a work order semantic vector, and converting each tag in the preset correct label set into a corresponding tag semantic vector; After fusing the semantic vector of any tag in the preset correct label set and the work order semantic vector as a query vector, and fusing the semantic vector of the remaining tags in the preset correct label set and the work order semantic vector as a key vector, the relevance weight between the tags is calculated through a self-attention mechanism; Selecting any one tag from the covered correct label set, filtering the preset correct labels whose relevance weight exceeds a preset threshold, and grouping all the filtered preset correct labels into a dynamic associated label set of the work order corresponding to the to-be-evaluated tag; Based on the dynamic associated label set, the co-occurrence relevance score weighted by attention is calculated.

[0081] In an embodiment of the present application, the semantic relevance score is obtained by the following way: Converting the text content of the work order corresponding to the to-be-evaluated tag into a work order semantic vector, and converting the expanded label into an expanded label semantic vector; Based on the work order semantic vector and the expanded label semantic vector, the attention weight and the cosine similarity of the expanded label to the key information of the work order text are calculated; Based on the attention weight and the cosine similarity, the attention degree of each expanded label to the key information of the work order text is calculated; The overall semantic correlation score is calculated by averaging the single correlation of all the expansion tags.

[0082] In an embodiment of the present application, the expansion tag novelty score is calculated by the following method: The cosine similarity between each expansion tag and the tags in the preset correct tag set is calculated. The novelty score of each expansion tag is obtained based on the cosine similarity. The overall novelty score is obtained by averaging the novelty scores of all the expansion tags.

[0083] In an embodiment of the present application, the device further comprises a verification unit configured to: The verification is performed on the comprehensive evaluation score by randomly extracting the tags to be evaluated and performing artificial re-evaluation to obtain an artificial re-evaluation score. The correlation coefficient between the artificial re-evaluation score and the comprehensive evaluation score is calculated. If the correlation coefficient is greater than or equal to a preset threshold, it is confirmed that the comprehensive evaluation score and the problem details of each dimension are valid.

[0084] In the embodiments of the present application, the problems of the existing evaluation, such as strong subjectivity, low efficiency and dependence on manual work, are solved. The large amount of manual checking is replaced by automatic quantitative evaluation, which greatly reduces the subjective bias and workload and efficiently adapts to the large batch of work order evaluation requirements. The multi-dimensional evaluation comprehensively covers the core evaluation requirements of label accuracy, integrity and expansion, and the dynamic adjustment of business weight makes the comprehensive score more suitable for the actual business scenario, and the problem details can quickly locate the label defects. In addition, the artificial re-evaluation verification mechanism further guarantees the objectivity and accuracy of the evaluation results, effectively makes up for the shortcomings of the existing simple rule checking, such as incorrect label identification, missing label identification and lack of basis for expansion label evaluation, and provides scientific and reliable technical support for work order label quality control.

[0085] The specific limitations of the label evaluation device can be referred to the limitations of the label evaluation method in the above, which will not be repeated here. Each module in the above label evaluation device can be realized by software, hardware and their combinations in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0086] In one embodiment, a computer device is provided, which can be a terminal device, and its internal structure diagram can be as shown in Figure 3The computer device shown in the figure includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer readable instructions. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer readable instructions are executed by the processor to implement a label evaluation method. The readable storage medium provided in the embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0087] In the embodiment of the present application, a computer device is provided, which includes a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, and the processor executes the computer readable instructions to implement the steps of the label evaluation method described above.

[0088] In the embodiment of the present application, a readable storage medium is provided, which stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the label evaluation method described above.

[0089] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0091] The above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A label evaluation method, characterized in that, The method includes: Construct an evaluation data table, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated; Based on the work order data and the preset set of correct labels, the labels to be evaluated are evaluated in multiple dimensions to obtain scores for each evaluation dimension. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation. Determine the business weights corresponding to each evaluation dimension; Based on the business weight, a weighted fusion calculation is performed on the scores of each evaluation dimension to obtain the comprehensive evaluation score of the label to be evaluated, and a problem detail for each evaluation dimension is generated. The problem detail for each dimension includes at least one of the following: mislabeled label ID, missing label name, and invalid extended label reason.

2. The label evaluation method as described in claim 1, characterized in that, The process involves multi-dimensional evaluation of the tags to be evaluated based on the work order data and a preset set of correct tags, yielding scores for each evaluation dimension, including: Calculate the matching depth score and keyword matching score between the tag to be evaluated and the tags in the preset set of correct tags, and use them as the accuracy score; Calculate the coverage score and the tag co-occurrence association score of the tag to be evaluated in the preset set of correct tags, and use them as the integrity score; If the tag to be evaluated is an extended tag outside the preset set of correct tags, then the semantic relevance score between the tag to be evaluated and the work order, as well as the novelty score of the extended tag, are calculated as the extendedness score.

3. The label evaluation method as described in claim 2, characterized in that, The matching depth score is calculated in the following ways: Based on the semantics of the tags, and according to the preset vector dimensions, the semantic vectors of each tag to be evaluated in the set of tags to be evaluated and the semantic vectors of the tags in the preset set of correct tags are generated. Based on the correlation between the semantic vector of each tag to be evaluated and the semantic vector of the corresponding preset correct tag, the attention weight of each tag to be evaluated and the tag in the set of the corresponding preset correct tags is calculated. Based on the attention-weighted semantic similarity mean between all tags to be evaluated and their corresponding preset correct tags, a bidirectional matching depth score is obtained.

4. The label evaluation method as described in claim 2, characterized in that, The keyword matching score is calculated as follows: Obtain the text content of the work order corresponding to the label to be evaluated; The text content is segmented into words, and words exceeding a preset threshold are selected to form a keyword set; The set of keywords commonly contained in the tags to be evaluated and the preset set of correct tags, as well as the total set of keywords; Calculate the first bidirectional attention weight between each keyword in the keyword set and its corresponding tag to be evaluated, and the second bidirectional attention weight between each keyword and its corresponding preset correct tag; The keyword matching score is calculated based on the first bidirectional attention weight and the second bidirectional attention weight.

5. The label evaluation method as described in claim 2, characterized in that, The coverage score is obtained in the following way: Select tags from the tags to be evaluated that match the tags in the preset set of correct tags to form a set of correctly covered tags; Based on the correlation between the semantic vector of the work order corresponding to the label to be evaluated and the semantic vector of each label in the preset set of correct labels, the importance of each label in the preset set of correct labels to the work order corresponding to the label to be evaluated is calculated through an attention mechanism. Calculate the sum of the first importance of all tags in the already covered correct tag set, and the sum of the second importance of all tags in the preset correct tag set; The coverage score is calculated based on the ratio of the sum of the first importance scores to the sum of the second importance scores.

6. The label evaluation method as described in claim 2, characterized in that, The tag co-occurrence association score is calculated as follows: The tags that match the tags in the preset set of correct tags are selected from the tags to be evaluated, forming a set of correctly covered tags; The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and each tag in the preset set of correct tags is converted into a corresponding tag semantic vector. The semantic vector of any tag in the preset correct tag set is fused with the work order semantic vector to form a query vector. The semantic vector of the remaining tags in the preset correct tag set is fused with the work order semantic vector to form a key vector. The association weight between tags is calculated through a self-attention mechanism. Select any tag from the already covered correct tag set, filter out the preset correct tags whose association weight exceeds a preset threshold, and form a dynamic association tag set of all the filtered preset correct tags for the work order corresponding to the tag to be evaluated. Based on the dynamically associated label set, an attention-weighted co-occurrence association score is calculated.

7. The label evaluation method as described in claim 2, characterized in that, The semantic relevance score is obtained in the following way: The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and the extended tag is converted into an extended tag semantic vector. Based on the work order semantic vector and the extended tag semantic vector, the attention weight and cosine similarity of the extended tag to the key information of the work order text are calculated. Based on the attention weights and cosine similarity, the attention level of each extended tag to the key information of the work order text is calculated. The overall semantic relevance score is calculated by averaging the individual relevance scores of all extended tags.

8. The label evaluation method as described in claim 2, characterized in that, The novelty score of the extended tag is calculated as follows: Calculate the cosine similarity between each extended label and the labels in the preset set of correct labels; Based on the cosine similarity, the novelty score of each extended tag is obtained; The overall novelty score is obtained by averaging the novelty scores of all extended tags.

9. The label evaluation method according to any one of claims 1-8, characterized in that, After obtaining the comprehensive evaluation score of the label to be evaluated, the process includes: The comprehensive evaluation score is verified by randomly selecting labels to be evaluated and conducting a manual review to obtain the manual review score. Calculate the correlation coefficient between the manual review score and the comprehensive evaluation score; If the correlation coefficient is greater than or equal to the preset threshold, then the comprehensive evaluation score and the details of problems in each dimension are confirmed to be valid.

10. A label evaluation device, characterized in that, The device includes: An evaluation dataset construction unit is used to construct an evaluation data table, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated. The multi-dimensional evaluation unit is used to perform multi-dimensional evaluation on the tags to be evaluated based on the work order data and a preset set of correct tags, and obtain scores for each evaluation dimension. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation. The business weight determination unit is used to determine the business weights corresponding to each evaluation dimension; The comprehensive evaluation result generation unit is used to perform weighted fusion calculation on the scores of each evaluation dimension based on the business weight, to obtain the comprehensive evaluation score of the label to be evaluated, and to generate a list of issues for each evaluation dimension. The list of issues for each dimension includes at least one of the following: mislabeled label ID, missing label name, and reason for invalid extended label.

Images