Label evaluation method and apparatus

Abstract

The application discloses a label evaluation method and device, and a method thereof, comprising: constructing an evaluation data table, wherein the evaluation data table comprises work order data, a preset correct label set and a to-be-evaluated label set; performing multi-dimensional evaluation on the to-be-evaluated label based on the work order data and the preset correct label set; determining a business weight corresponding to each evaluation dimension; performing weighted fusion calculation on scores of the evaluation dimensions based on the business weight to obtain a comprehensive evaluation score of the to-be-evaluated label, and generating a problem detail of each evaluation dimension. In the embodiment of the application, the evaluation data table is constructed, the business weight weighted comprehensive score of the to-be-evaluated label is evaluated in multiple dimensions, and the problem detail is generated. The automatic quantitative evaluation is realized, the subjective bias and workload are reduced, a large number of work orders are adapted, the accuracy of the evaluation result is ensured, and the problems of wrong and missed labels and the difficult label evaluation are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a label evaluation method and device. BACKGROUND

[0002] Currently, in online consultation scenarios such as customer service, it is often necessary to generate a to-be-evaluated label for an online generated work order to mark the core features of the work order, so as to facilitate subsequent distribution and classification for corresponding departments to perform next step processing and other related work. However, the current label evaluation task mainly relies on manual checking and subjective scoring, and lacks efficient and accurate automatic evaluation tools.

[0003] On the one hand, manual evaluation is highly subjective, and evaluation personnel judge the accuracy and completeness of the label according to their personal understanding of the label semantics and business experience, lacking objective text and label correlation quantitative analysis methods. On the other hand, the label system has dynamic expandability, and the pre-defined label set is dynamically updated with the content of the work order. The expanded label lacks clear definition standards, and different evaluation personnel have different judgment angles for the same expanded label, resulting in different scoring scales and making it difficult to form a unified conclusion.

[0004] Moreover, the existing evaluation methods mostly adopt a mode of combining manual spot checks with simple rule verification, which can reduce some repetitive labor, but when faced with a large number of work orders, manual checking of the accuracy and completeness of the label is still required, which not only has limited efficiency improvement, but also makes it difficult for limited manpower to bear, and subjective bias is always difficult to eliminate. The specific manifestations are as follows: manual checking is time-consuming and labor-intensive, and a large amount of manpower is needed to face a large number of work orders, and it is easy to have wrong label problems of literal matching but business semantics deviation; missing label problems are difficult to identify in advance by automatic means; there is no objective evaluation basis for the relevance and novelty of the expanded label, which is easy to be out of touch with the core demand. SUMMARY

[0005] Therefore, it is necessary to provide a label evaluation method and device to solve at least one problem in the prior art.

[0006] In a first aspect, a label evaluation method is provided, comprising:

[0007] constructing an evaluation data table, the evaluation data table comprising work order data, a pre-set correct label set and a to-be-evaluated label set;

[0008] performing multi-dimensional evaluation on the to-be-evaluated label based on the work order data and the pre-set correct label set to obtain scores of each evaluation dimension, wherein the evaluation dimension comprises at least two of accuracy evaluation, completeness evaluation and expansion evaluation;

[0009] determining the business weight corresponding to each evaluation dimension;

[0010] Based on the business weight, a weighted fusion calculation is performed on the scores of each evaluation dimension to obtain the comprehensive evaluation score of the label to be evaluated, and a problem detail for each evaluation dimension is generated. The problem detail for each dimension includes at least one of the following: mislabeled label ID, missing label name, and invalid extended label reason.

[0011] In one possible implementation, the multi-dimensional evaluation of the tags to be evaluated based on the work order data and a preset set of correct tags is performed to obtain scores for each evaluation dimension, including:

[0012] Calculate the matching depth score and keyword matching score between the tag to be evaluated and the tags in the preset set of correct tags, and use them as the accuracy score;

[0013] Calculate the coverage score and the tag co-occurrence association score of the tag to be evaluated in the preset set of correct tags, and use them as the integrity score;

[0014] If the tag to be evaluated is an extended tag outside the preset set of correct tags, then the semantic relevance score between the tag to be evaluated and the work order, as well as the novelty score of the extended tag, are calculated as the extendedness score.

[0015] In one possible implementation, the matching depth score is calculated in the following manner:

[0016] Based on the semantics of the tags, and according to the preset vector dimensions, the semantic vectors of each tag to be evaluated in the set of tags to be evaluated and the semantic vectors of the tags in the preset set of correct tags are generated.

[0017] Based on the correlation between the semantic vector of each tag to be evaluated and the semantic vector of the corresponding preset correct tag, the attention weight of each tag to be evaluated and the tag in the set of the corresponding preset correct tags is calculated.

[0018] Based on the attention-weighted semantic similarity mean between all tags to be evaluated and their corresponding preset correct tags, a bidirectional matching depth score is obtained.

[0019] In one possible implementation, the keyword matching score is calculated as follows:

[0020] Obtain the text content of the work order corresponding to the label to be evaluated;

[0021] The text content is segmented into words, and words exceeding a preset threshold are selected to form a keyword set;

[0022] The set of keywords commonly contained in the tags to be evaluated and the preset set of correct tags, as well as the total set of keywords;

[0023] Calculate the first bidirectional attention weight between each keyword in the keyword set and its corresponding tag to be evaluated, and the second bidirectional attention weight between each keyword and its corresponding preset correct tag;

[0024] The keyword matching score is calculated based on the first bidirectional attention weight and the second bidirectional attention weight.

[0025] In one possible implementation, the coverage score is obtained as follows:

[0026] Select tags from the tags to be evaluated that match the tags in the preset set of correct tags to form a set of correctly covered tags;

[0027] Based on the correlation between the semantic vector of the work order corresponding to the label to be evaluated and the semantic vector of each label in the preset set of correct labels, the importance of each label in the preset set of correct labels to the work order corresponding to the label to be evaluated is calculated through an attention mechanism.

[0028] Calculate the sum of the first importance of all tags in the already covered correct tag set, and the sum of the second importance of all tags in the preset correct tag set;

[0029] The coverage score is calculated based on the ratio of the sum of the first importance scores to the sum of the second importance scores.

[0030] In one possible implementation, the tag co-occurrence association score is calculated as follows:

[0031] The tags that match the tags in the preset set of correct tags are selected from the tags to be evaluated, forming a set of correctly covered tags;

[0032] The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and each tag in the preset set of correct tags is converted into a corresponding tag semantic vector.

[0033] The semantic vector of any tag in the preset correct tag set is fused with the work order semantic vector to form a query vector. The semantic vector of the remaining tags in the preset correct tag set is fused with the work order semantic vector to form a key vector. The association weight between tags is calculated through a self-attention mechanism.

[0034] Select any tag from the already covered correct tag set, filter out the preset correct tags whose association weight exceeds a preset threshold, and form a dynamic association tag set of all the filtered preset correct tags for the work order corresponding to the tag to be evaluated.

[0035] Based on the dynamically associated label set, an attention-weighted co-occurrence association score is calculated.

[0036] In one possible implementation, the semantic relevance score is obtained as follows:

[0037] The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and the extended tag is converted into an extended tag semantic vector.

[0038] Based on the work order semantic vector and the extended tag semantic vector, the attention weight and cosine similarity of the extended tag to the key information of the work order text are calculated.

[0039] Based on the attention weights and cosine similarity, the attention level of each extended tag to the key information of the work order text is calculated.

[0040] The overall semantic relevance score is calculated by averaging the individual relevance scores of all extended tags.

[0041] In one possible implementation, the novelty score of the extended tag is calculated as follows:

[0042] Calculate the cosine similarity between each extended label and the labels in the preset set of correct labels;

[0043] Based on the cosine similarity, the novelty score of each extended tag is obtained;

[0044] The overall novelty score is obtained by averaging the novelty scores of all extended tags.

[0045] In one possible implementation, obtaining the comprehensive evaluation score of the label to be evaluated includes:

[0046] The comprehensive evaluation score is verified by randomly selecting labels to be evaluated and conducting a manual review to obtain the manual review score.

[0047] Calculate the correlation coefficient between the manual review score and the comprehensive evaluation score;

[0048] If the correlation coefficient is greater than or equal to the preset threshold, then the comprehensive evaluation score and the details of problems in each dimension are confirmed to be valid.

[0049] Secondly, a label evaluation device is provided, comprising:

[0050] An evaluation dataset construction unit is used to construct an evaluation data table, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated.

[0051] The multi-dimensional evaluation unit is used to perform multi-dimensional evaluation on the tags to be evaluated based on the work order data and a preset set of correct tags, and obtain scores for each evaluation dimension. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation.

[0052] The business weight determination unit is used to determine the business weights corresponding to each evaluation dimension;

[0053] The comprehensive evaluation result generation unit is used to perform weighted fusion calculation on the scores of each evaluation dimension based on the business weight, to obtain the comprehensive evaluation score of the label to be evaluated, and to generate a list of issues for each evaluation dimension. The list of issues for each dimension includes at least one of the following: mislabeled label ID, missing label name, and reason for invalid extended label.

[0054] The aforementioned label evaluation method and apparatus, the method comprising: constructing an evaluation data table, the evaluation data table including work order data, a preset set of correct labels, and a set of labels to be evaluated; performing multi-dimensional evaluation of the labels to be evaluated based on the work order data and the preset set of correct labels, wherein the evaluation dimensions include at least two of accuracy evaluation, completeness evaluation, and extensibility evaluation; determining the business weight corresponding to each evaluation dimension; performing a weighted fusion calculation on the scores of each evaluation dimension based on the business weight to obtain a comprehensive evaluation score for the labels to be evaluated, and generating a problem detail for each evaluation dimension, wherein the problem detail for each dimension includes at least one of incorrectly labeled label ID, missing label name, and invalid extended label reason. This application addresses the problems of existing assessments being reliant on manual labor, highly subjective, and inefficient. It replaces extensive manual verification with automated quantitative assessment, significantly reducing subjective bias and workload, and efficiently adapting to the assessment needs of large-volume work orders. Furthermore, it comprehensively covers core assessment requirements such as label accuracy, completeness, and scalability through multi-dimensional assessment. Combined with dynamic adjustments to business weights, the overall score more closely reflects actual business scenarios, and the problem details can quickly pinpoint label defects. In addition, the inclusion of a manual review and verification mechanism further ensures the objectivity and accuracy of the assessment results, effectively compensating for the shortcomings of existing simple rule-based verification, such as the difficulty in identifying mislabeled or missing labels and the lack of basis for assessing extended labels. This provides scientific and reliable technical support for work order label quality control. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1This is a flowchart illustrating a label evaluation method in one embodiment of this application;

[0057] Figure 2 This is a schematic diagram of the structure of a label evaluation device in one embodiment of this application;

[0058] Figure 3 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] In one embodiment, such as Figure 1 As shown, a label evaluation method is provided, including the following steps:

[0061] In step S110, an evaluation data table is constructed, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated.

[0062] Optionally, a work order table can be collected, which may include basic work order information such as work order number, submission time, request text, and region, serving as the core textual basis for tag evaluation. Similarly, a tag data table can be collected, which may include tag-related content such as the corresponding work order number, the system-generated set of tags to be evaluated, and the preset set of correct tags, serving as a benchmark for evaluating tag quality. Finally, the two tables are joined using the work order number as the primary key, and core fields such as work order number, request text, region, set of tags to be evaluated, and preset set of correct tags are extracted and integrated to ensure that each work order can be accurately matched with the corresponding tag, providing complete and relevant data support for subsequent multi-dimensional quantitative evaluation.

[0063] The set of tags to be evaluated is a combination of tags automatically generated by the system for a specific work order, used to mark the core features of the work order (such as the type of request and the domain of the problem). For example, when a user submits a work order (such as a customer service inquiry), the tag generation system (usually based on natural language processing technology) will automatically parse the request text of the work order, and then extract core information (such as property maintenance or water bill inquiry) from the work order text through keyword matching, semantic vector classification, and pre-trained model prediction, forming a tag combination corresponding to the work order, which is the set of tags to be evaluated.

[0064] The preset correct label set, based on business rules, historical experience, or expert knowledge, is a set of standard labels pre-defined for a certain type of work order. It serves as the core benchmark for judging the accuracy and completeness of the labels to be evaluated. For example, high-quality labels that are accurately marked manually and effectively support allocation and classification from historical work orders can be compiled and statistically analyzed. These labels can be categorized by work order type (e.g., consultation and advice) or problem area (e.g., water, electricity, gas, property management, transportation) to form a standard label library corresponding to each type of work order. The subset of the label library corresponding to the type of a particular work order is the preset correct label set for that work order. Alternatively, business personnel can formulate clear labeling rules based on business needs, which are then reviewed and confirmed by domain experts to ensure the accuracy and applicability of the labels, ultimately forming the preset correct label set.

[0065] In step S120, the work order data and the preset set of correct labels are used to evaluate the labels to be evaluated in a multi-dimensional way. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation.

[0066] Optionally, work order data can be used as the anchor point for real business scenarios, and a preset set of correct labels can be used as the standard reference. The quality of the labels to be evaluated can be quantified from different quality dimensions (accuracy, completeness, and scalability). Through multi-dimensional evaluation, the quality status of the labels to be evaluated can be comprehensively and accurately reflected, ensuring that they meet business needs.

[0067] The accuracy assessment measures the degree of fit between the tag to be evaluated and the preset correct tag. By calculating the matching depth between the tag to be evaluated and the preset correct tag, as well as their keyword matching degree, two accuracy scores are obtained. These scores determine how well the tag to be evaluated matches the preset correct tag in terms of semantics and key information, ensuring that the tag accurately reflects the corresponding content.

[0068] The integrity assessment reflects the coverage and association of the tags to be evaluated with the preset correct tags. By calculating the coverage rate of the tags to be evaluated in the preset correct tags, and the co-occurrence association score between tags, two integrity scores are obtained. These scores are used to determine whether the tags to be evaluated in the system fully cover the key parts of the preset correct tags, and whether the association between tags meets expectations, ensuring that the tags can completely present relevant information.

[0069] The expansion assessment targets expansion tags that exceed the preset range of correct tags. It calculates the semantic relevance of these expansion tags to the work order, as well as the novelty score of the expansion tags themselves, resulting in two expansion scores. These scores assess whether the expansion tags are closely related to the work order content and whether they possess novel value, ensuring that the expansion tags provide a meaningful supplement to the tagging system.

[0070] In step S130, the business weights corresponding to each evaluation dimension are determined;

[0071] Optionally, due to the differences in business scenarios and key demands embodied in different work orders, the level of attention and requirements for each evaluation dimension of the tag (accuracy, completeness, scalability, etc.) will also vary. For example, for work orders whose core business lies in accurately identifying known problem types, the accuracy dimension will be more important, and the corresponding business weight will be greater; while for work orders that need to explore new business points and expand service boundaries, the business weight of the scalability dimension may be more prominent. By combining the specific business attributes and demand characteristics of the work orders corresponding to the tags to be evaluated, and assigning appropriate business weights to each evaluation dimension, the subsequent comprehensive evaluation score can better fit the actual business needs and more accurately reflect the value and quality of the tag in the business scenario.

[0072] In step S140, a weighted fusion calculation is performed on the scores of each evaluation dimension based on the business weight to obtain the comprehensive evaluation score of the label to be evaluated, and a problem detail for each evaluation dimension is generated. The problem detail for each dimension includes at least one of the following: mislabeled label ID, missing label name, and invalid extended label reason.

[0073] Optionally, a weighted fusion calculation is performed on the scores of each evaluation dimension (accuracy, completeness, extensibility, etc.) based on business weights. That is, the scores of each dimension are weighted and summed according to their importance (business weight) in the current business scenario to obtain the comprehensive evaluation score of the label to be evaluated. At the same time, a detailed list of problems for each dimension is generated simultaneously during the calculation process, such as at least one of the following: mislabeled label ID, missing label name, and invalid extended label reason. And the label defects found in the evaluation are precisely located: mislabeled label ID is used to identify labels in the accuracy dimension that deviate too much from the preset correct label; missing label name is used to record the preset correct label in the completeness dimension that is not covered by the label to be evaluated; invalid extended label reason is used to explain the new label in the extensibility dimension that does not meet the requirements (such as being irrelevant to the work order semantics, lacking novelty, etc.).

[0074] This application provides a tag evaluation method, comprising: constructing an evaluation data table, the evaluation data table including work order data, a preset set of correct tags, and a set of tags to be evaluated; performing multi-dimensional evaluation on the tags to be evaluated based on the work order data and the preset set of correct tags, wherein the evaluation dimensions include at least two of accuracy evaluation, completeness evaluation, and extensibility evaluation; determining the business weight corresponding to each evaluation dimension; performing a weighted fusion calculation on the scores of each evaluation dimension based on the business weight to obtain a comprehensive evaluation score for the tags to be evaluated, and generating a problem detail for each evaluation dimension, wherein the problem detail for each dimension includes at least one of incorrectly labeled tag ID, missing label name, and invalid extended tag reason. This application addresses the problems of existing assessments being reliant on manual labor, highly subjective, and inefficient. It replaces extensive manual verification with automated quantitative assessment, significantly reducing subjective bias and workload, and efficiently adapting to the assessment needs of large-volume work orders. Furthermore, it comprehensively covers core assessment requirements such as label accuracy, completeness, and scalability through multi-dimensional assessment. Combined with dynamic adjustments to business weights, the overall score more closely reflects actual business scenarios, and the problem details can quickly pinpoint label defects. In addition, the inclusion of a manual review and verification mechanism further ensures the objectivity and accuracy of the assessment results, effectively compensating for the shortcomings of existing simple rule-based verification, such as the difficulty in identifying mislabeled or missing labels and the lack of basis for assessing extended labels. This provides scientific and reliable technical support for work order label quality control.

[0075] In one embodiment of this application, the multi-dimensional evaluation of the tags to be evaluated based on the work order data and a preset set of correct tags to obtain scores for each evaluation dimension includes:

[0076] Calculate the matching depth score and keyword matching score between the tag to be evaluated and the tags in the preset set of correct tags, and use them as the accuracy score;

[0077] Calculate the coverage score and the tag co-occurrence association score of the tag to be evaluated in the preset set of correct tags, and use them as the integrity score;

[0078] If the tag to be evaluated is an extended tag outside the preset set of correct tags, then the semantic relevance score between the tag to be evaluated and the work order, as well as the novelty score of the extended tag, are calculated as the extendedness score.

[0079] Optionally, the accuracy score may include a match depth score and a keyword matching score; the completeness score may include a coverage score and a tag co-occurrence association score; and the extensibility score may include a semantic relevance score and an extended tag novelty score. By calculating the match depth score and the keyword matching score, the accuracy of tags can be comprehensively evaluated from two aspects: deep semantic association and overlap of explicit key information, avoiding misjudgment from a single dimension; calculating the coverage score and the tag co-occurrence association score can take into account both the coverage of core tags and the logical rationality between tags, making the completeness assessment more in-depth; and calculating the semantic relevance score and the extended tag novelty score can double-verify the relevance and uniqueness of extended tags, filtering out invalid innovations.

[0080] Among them, the matching depth score is obtained by converting the tag to be evaluated and the preset correct tag into semantic vectors, and calculating the average attention-weighted similarity between the vectors, thereby measuring the deep matching degree of the tag at the semantic level; the keyword matching score is obtained by segmenting the work order text to obtain a set of keywords, counting the keywords commonly contained in the tag to be evaluated and the preset correct tag, and combining the bidirectional attention weight between keywords and tags, reflecting the fit of the tag in key information.

[0081] The coverage score is calculated by first selecting the tags to be evaluated that match the preset correct tags, forming a set of covered correct tags. Then, based on the semantic vector correlation between the work order and the preset correct tags, an attention mechanism is used to calculate the importance of each preset correct tag to the work order. Finally, the score is determined by the ratio of the sum of the importance of the covered correct tags to the sum of the importance of all preset correct tags, reflecting the coverage of the tag to be evaluated by the core correct tag. The tag co-occurrence association score is calculated after obtaining the covered correct tag set by fusing the semantic vectors of the work order and the preset correct tags, using a self-attention mechanism to calculate the association weight between tags, and selecting the preset correct tags whose association weight with the covered correct tags exceeds a preset threshold to form a dynamic association tag set. Then, an attention-weighted co-occurrence association score is calculated to evaluate the rationality and tightness of the association between tags.

[0082] The semantic relevance score is calculated by converting the work order text and extended tags into semantic vectors, calculating the attention weight and cosine similarity of the extended tags to the key information in the work order text, and combining these two factors to obtain the attention level of each extended tag to the key information in the work order. The average of all extended tag attention scores is then used to determine the degree of relevance between the extended tags and the core content of the work order. The extended tag novelty score is calculated by calculating the cosine similarity of each extended tag to tags in a preset set of correct tags, taking the maximum cosine similarity between each extended tag and the preset set of correct tags, and subtracting this maximum value from 1 to obtain the novelty score for each extended tag. Finally, the average of all extended tag novelty scores is used to evaluate the novelty and value of the extended tags.

[0083] In one embodiment of this application, the matching depth score is calculated in the following manner, including:

[0084] Based on the semantics of the tags, and according to the preset vector dimensions, the semantic vectors of each tag to be evaluated in the set of tags to be evaluated and the semantic vectors of the tags in the preset set of correct tags are generated.

[0085] Based on the correlation between the semantic vector of each tag to be evaluated and the semantic vector of the corresponding preset correct tag, the attention weight of each tag to be evaluated and the tag in the set of the corresponding preset correct tags is calculated.

[0086] Based on the attention-weighted semantic similarity mean between all tags to be evaluated and their corresponding preset correct tags, a bidirectional matching depth score is obtained.

[0087] It should be noted that the matching depth score is calculated using a two-way deep matching strategy. Since the preset correct labels may contain abstract categories defined by business rules, while the labels to be evaluated are mostly descriptions of specific scenarios, one-way calculation may overly rely on the specific features of the labels to be evaluated, ignoring the abstract business implications of the correct labels. Two-way matching allows both sides to consider each other, taking into account both the scenario details of the labels to be evaluated and the business classification logic of the correct labels. This avoids bias caused by a single perspective, making the accuracy score more aligned with the dual standards of business needs and actual scenario conditions.

[0088] Optionally, by using pre-trained models such as BERT, both the labels to be evaluated and the preset correct labels are converted into semantic vectors according to a preset vector dimension, thus transforming the text labels into a computable mathematical form. Then, the correlation between the semantic vectors of the two types of labels (such as cosine similarity) is calculated, and the correlation can be converted into attention weights. Based on this, higher attention weights are assigned to closely related label pairs (such as "the water pipe in the community burst") to reduce the interference of weakly related labels. Finally, the semantic similarity of all label pairs is calculated by weighting the attention weights to obtain the bidirectional matching depth score, which reflects the deep matching of the overall label combination rather than the performance of a single label.

[0089] Specifically, d can be the vector dimension, and the set of labels to be evaluated can be defined as follows:

[0090] ,

[0091] There are n labels in total, and the semantic vector of each label to be evaluated is... Preset correct tag set There are m tags in total, and the semantic vector of each preset correct tag is: .

[0092] Then, the attention weight of the label to be evaluated to the preset correct label is calculated, as shown in the following formula:

[0093] ;

[0094] Where exp represents the exponential function, and T represents the transpose. This represents scaling the vector dot product, a common operation in attention mechanisms to prevent vanishing or exploding gradients. The above formula measures each label to be evaluated. For each correct label The level of attention, symmetrical. Measure each correct label For each label to be evaluated The level of attention.

[0095] The final bidirectional attention weight is obtained by averaging the attention weights from both directions, as shown in the following formula:

[0096] ;

[0097] Finally, the attention-weighted semantic similarity average between all the tags to be evaluated and their corresponding correct tags is taken to obtain the bidirectional matching depth score, as shown in the following formula:

[0098] .

[0099] In one embodiment of this application, the keyword matching score is calculated as follows:

[0100] Obtain the text content of the work order corresponding to the label to be evaluated;

[0101] The text content is segmented into words, and words exceeding a preset threshold are selected to form a keyword set;

[0102] The set of keywords commonly contained in the tags to be evaluated and the preset set of correct tags, as well as the total set of keywords;

[0103] Calculate the first bidirectional attention weight between each keyword in the keyword set and its corresponding tag to be evaluated, and the second bidirectional attention weight between each keyword and its corresponding preset correct tag;

[0104] The keyword matching score is calculated based on the first bidirectional attention weight and the second bidirectional attention weight.

[0105] It should be noted that, regarding accuracy assessment, in addition to calculating the overall similarity between tags, it is also necessary to quantify the system's ability to capture accurate keywords for the tags to be evaluated. This means assessing whether the system's tags captured key information in the appeal text, and whether this capture relied on inappropriate methods such as brute-force search. For the former, this is demonstrated by characterizing the common abilities of the system's tags to be evaluated and the preset correct tags in capturing keywords from the appeal text; for the latter, it is demonstrated by penalizing the system's tags for capturing an excessive number of keywords.

[0106] First, keywords exceeding a preset threshold can be extracted from the work order text corresponding to the tag to be evaluated to form a set. Then, the common keywords of the tag to be evaluated and the preset correct tag, as well as the total keyword set, are counted. By calculating the bidirectional attention weight of each keyword and the two types of tags (reflecting the importance of keywords to tags and the dependence of tags on keywords), the keyword matching score is finally obtained by weighted fusion based on these weights. This not only anchors the core information of the work order through keyword extraction, but also highlights key associations with the help of the bidirectional attention mechanism, accurately quantifying the fit between the tag to be evaluated and the preset correct tag at the explicit key information level.

[0107] Specifically, the text content of the work order request corresponding to the tag to be evaluated is extracted, and after word segmentation, words exceeding the threshold are selected according to their TF-IDF values ​​to form a keyword set. .

[0108] Then, based on the keyword set, the tags to be evaluated were statistically analyzed. With the correct label Commonly contained set of keywords and the total set of keywords for both. It should be noted that when determining whether keywords are contained, soft intersection based on semantic similarity can be used. This involves using a pre-trained model to convert candidate words and tag words into semantic vectors of the same dimension, calculating the similarity between each pair of vectors, and identifying keywords exceeding a threshold as semantically related keywords. This process then involves statistically analyzing the set of common related keywords between the tag to be evaluated and the preset correct tag, as well as the total keyword set. Alternatively, domain dictionary matching and TF-IDF statistical methods can also be used to determine keyword inclusion relationships.

[0109] Next, the corresponding tag to be evaluated for each keyword can be calculated. Bidirectional attention weights and the correct tag corresponding to the word. Bidirectional attention weights .

[0110] Finally, the keyword matching score is calculated, as shown in the following formula:

[0111] .

[0112] In one embodiment of this application, the coverage score is obtained in the following manner:

[0113] Select tags from the tags to be evaluated that match the tags in the preset set of correct tags to form a set of correctly covered tags;

[0114] Based on the correlation between the semantic vector of the work order corresponding to the label to be evaluated and the semantic vector of each label in the preset set of correct labels, the importance of each label in the preset set of correct labels to the work order corresponding to the label to be evaluated is calculated through an attention mechanism.

[0115] Calculate the sum of the first importance of all tags in the already covered correct tag set, and the sum of the second importance of all tags in the preset correct tag set;

[0116] The coverage score is calculated based on the ratio of the sum of the first importance scores to the sum of the second importance scores.

[0117] It should be noted that during the completeness assessment, the degree to which the system's evaluated tags cover the preset set of correct tags is quantified to reflect whether any core required tags have been omitted. It should be observed that among the preset correct tags, some tags are more semantically critical to the current work order and should be given higher weight through attention to avoid the problem of covering secondary tags while omitting core tags and still receiving a high score.

[0118] When calculating the coverage score, firstly, tags that match the preset set of correct tags are selected from the tags to be evaluated, forming a set of correctly covered tags. Then, using the semantic vectors of the work orders corresponding to the tags to be evaluated and the preset correct tags, an attention mechanism is used to calculate the importance of each tag in the preset correct tag set to the work orders corresponding to the tags to be evaluated. Next, the sum of the first importance of all tags in the set of correctly covered tags and the sum of the second importance of all tags in the preset correct tag set are calculated. Finally, the coverage score is obtained by the ratio of the sum of the first importance to the sum of the second importance. This measures the coverage level of the tags to be evaluated to the key parts of the preset correct tags, and takes into account the differences in importance of each tag in the work order scenario, making the evaluation more accurate and in line with the actual business.

[0119] Specifically, firstly, tags belonging to a pre-defined set of correct tags can be filtered from the tags to be evaluated using string matching or semantic similarity. Its size is String matching refers to directly comparing the text content of the tags for consistency, such as the tag to be evaluated. There is a "water pressure failure" in the label, but the correct label is incorrect. If "water pressure failure" also exists, then this tag will be filtered out. In the middle, if the label is correct If the tag is "water pressure problem", it will not be filtered. Semantic similarity is used when label texts are not completely identical but semantically similar (e.g., the label "water supply anomaly" in the evaluation set is semantically related to the correct label "water pressure failure"). A pre-trained language model (e.g., BERT) converts the labels into semantic vectors, and the similarity between these vectors is calculated (e.g., cosine similarity). If the similarity exceeds a preset threshold (e.g., 0.8), the label to be evaluated is determined to belong to the preset set of correct labels and is included. middle.

[0120] Then, the importance of each preset label to the current work order can be calculated using an attention mechanism, as shown in the following formula:

[0121] ;

[0122] in, It is the semantic vector of the work order text corresponding to the tag to be evaluated. It is a semantic vector of the correctly preset label.

[0123] Finally, the weighted coverage score can be calculated using the following formula:

[0124] ;

[0125] This formula demonstrates that the coverage score depends not only on whether there is coverage, but also on the importance of the covered labels.

[0126] In one embodiment of this application, the tag co-occurrence association score is calculated as follows:

[0127] The tags that match the tags in the preset set of correct tags are selected from the tags to be evaluated, forming a set of correctly covered tags;

[0128] The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and each tag in the preset set of correct tags is converted into a corresponding tag semantic vector.

[0129] The semantic vector of any tag in the preset correct tag set is fused with the work order semantic vector to form a query vector. The semantic vector of the remaining tags in the preset correct tag set is fused with the work order semantic vector to form a key vector. The association weight between tags is calculated through a self-attention mechanism.

[0130] Select any tag from the already covered correct tag set, filter out the preset correct tags whose association weight exceeds a preset threshold, and form a dynamic association tag set of all the filtered preset correct tags for the work order corresponding to the tag to be evaluated.

[0131] Based on the dynamically associated label set, an attention-weighted co-occurrence association score is calculated.

[0132] It should be noted that, in addition to the coverage score mentioned above, the co-occurrence association score is also included when performing the integrity assessment. Since the association strength between the preset correct labels is not fixed, the label association strength in the current work order scenario needs to be dynamically calculated through self-attention. When calculating the score, the missing associated labels are taken into account. That is, if the system captures a correct label for the label to be evaluated, but does not capture other correct labels that are strongly associated with that correct label, the co-occurrence association score will be reduced.

[0133] When calculating the tag co-occurrence association score, firstly, tags matching the preset correct tag set are selected from the tags to be evaluated, forming the covered correct tag set. Next, the work order text content corresponding to the tag to be evaluated is converted into a work order semantic vector, and each tag in the preset correct tag set is converted into a corresponding tag semantic vector. Then, the semantic vector of any tag in the preset correct tag set is fused with the work order semantic vector as a query vector, and the semantic vectors of the remaining tags are fused with the work order semantic vector as key vectors. The association weight between tags is calculated using a self-attention mechanism. Then, a tag is randomly selected from the covered correct tag set, and preset correct tags with association weights exceeding a preset threshold are selected to form the dynamic association tag set of the work order corresponding to the tag to be evaluated. Finally, based on the dynamic association tag set, the attention-weighted co-occurrence association score is calculated to evaluate the closeness of association between the covered correct tags and other relevant preset correct tags, reflecting the rationality of the logical association between tags.

[0134] Specifically, firstly, using the preset correct labels For querying other preset correct tags Using the key, calculate the self-attention association weights between tags, which can be expressed by the following formula:

[0135] ;

[0136] This includes incorporating the text vector of the work order request corresponding to the label to be evaluated. .

[0137] For the covered labels It will iterate through all tags in the preset set of correct tags. Examine the correlation weights calculated by the self-attention mechanism between the two. If the value exceeds the threshold, then preset correct labels that exceed the preset threshold will be filtered out. This constitutes the dynamic associated tag set under the current work order. This is used to discover other preset correct tags that are closely related to the already covered correct tags.

[0138] Finally, the attention-weighted co-occurrence correlation score can be calculated using the following formula:

[0139] ;

[0140] Therefore, the correlation score depends not only on whether the correlation tags are covered, but also on the importance and correlation strength of the correlation tags.

[0141] In one embodiment of this application, the semantic relevance score is obtained in the following manner:

[0142] The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and the extended tag is converted into an extended tag semantic vector.

[0143] Based on the work order semantic vector and the extended tag semantic vector, the attention weight and cosine similarity of the extended tag to the key information of the work order text are calculated.

[0144] Based on the attention weights and cosine similarity, the attention level of each extended tag to the key information of the work order text is calculated.

[0145] The overall semantic relevance score is calculated by averaging the individual relevance scores of all extended tags.

[0146] It should be noted that when conducting extended evaluation, the semantic relevance between extended tags and work orders can be calculated to assess whether extended tags other than the preset correct tags are related to the core semantics of the work order text, thus avoiding the generation of irrelevant tags. For example, if the work order to be evaluated is about elevator malfunction, the system may generate extended tags for greening issues.

[0147] When calculating the semantic relevance score, the text content of the work order corresponding to the tag to be evaluated can be converted into a work order semantic vector first, and the extended tags can be converted into extended tag semantic vectors. Then, based on these two types of semantic vectors, the attention weight of the extended tags to the key information of the work order text and the cosine similarity between the two can be calculated. Then, combining the attention weight and cosine similarity, the attention of each extended tag to the key information of the work order text (i.e., the relevance of a single extended tag) can be calculated. Finally, the average of the relevance of all extended tags is taken to obtain the overall semantic relevance score, thereby quantifying the closeness of the connection between the extended tags and the core content of the work order, and ensuring that the extended tags do not deviate from the actual scenario of the work order.

[0148] Specifically, firstly, for system-generated tags that are not in the preset correct tag set... Extended tag set in The association between extended tags and the core semantics of the work order is calculated using the following formula:

[0149] ;

[0150] The text attention weight can be calculated using the following formula:

[0151] ;

[0152] Used to measure extended labels Attention to key information in work order texts.

[0153] Then, the relevance of all extended tags is averaged to calculate the overall semantic relevance score, which can be calculated using the following formula:

[0154] ;

[0155] because contain This is an attention-weighted average of the similarity between the extended tag and the work order request text corresponding to the tag to be evaluated.

[0156] In one embodiment of this application, the novelty score of the extended tag is calculated as follows:

[0157] Calculate the cosine similarity between each extended label and the labels in the preset set of correct labels;

[0158] Based on the cosine similarity, the novelty score of each extended tag is obtained;

[0159] The overall novelty score is obtained by averaging the novelty scores of all extended tags.

[0160] It should be noted that, in addition to the semantic relevance score mentioned above, a novelty score also needs to be calculated when conducting the expansion assessment. By calculating the novelty score, it is possible to clearly assess whether the expanded tags are an effective supplement to the pre-set correct tag set, avoiding duplication of pre-set tags or high semantic overlap with pre-set tags.

[0161] The calculation of the novelty score for expanded tags involves several steps. First, the cosine similarity between each expanded tag and all tags in the pre-defined set of correct tags is calculated to measure the semantic overlap between the expanded tag and existing correct tags. Next, the novelty score for each expanded tag is determined based on these cosine similarities. Finally, the average of the individual novelty scores for all expanded tags is used to obtain the overall novelty score. This score directly reflects the uniqueness of the expanded tags in departing from the pre-defined set of correct tags, avoiding meaningless expansions that duplicate existing correct tags.

[0162] Specifically, firstly, regarding extended tags The novelty is calculated by taking the maximum cosine similarity between each extended tag and all tags in the preset set of correct tags, and then subtracting the maximum cosine similarity from 1 to obtain the novelty score of each extended tag (the lower the similarity, the higher the novelty). This can be expressed by the following formula:

[0163] ;

[0164] in Operators retrieve extended tags With all preset correct labels The maximum cosine similarity.

[0165] Finally, the novelty of all extended tags is averaged, which can be expressed by the following formula:

[0166] ;

[0167] If no extended tags are provided, the default score is 0.

[0168] In one embodiment of this application, obtaining the comprehensive evaluation score of the label to be evaluated includes:

[0169] The comprehensive evaluation score is verified by randomly selecting labels to be evaluated and conducting a manual review to obtain the manual review score.

[0170] Calculate the correlation coefficient between the manual review score and the comprehensive evaluation score;

[0171] If the correlation coefficient is greater than or equal to the preset threshold, then the comprehensive evaluation score and the details of problems in each dimension are confirmed to be valid.

[0172] Optionally, after obtaining the comprehensive evaluation score of the tag to be evaluated, the validity of the score needs to be verified by manual review. The specific process can be as follows: First, randomly select samples from the tags to be evaluated, and have people re-evaluate the quality (such as accuracy, completeness, etc.) of these tags and give a manual review score; then, by calculating the correlation coefficient (such as Pearson correlation coefficient) between the manual review score and the comprehensive evaluation score generated by the system, the consistency between the two is quantified. The higher the correlation coefficient, the higher the degree of fit between the system score and the human subjective judgment; if the calculated correlation coefficient is greater than or equal to a preset threshold (such as 0.8), it can be confirmed that the current comprehensive evaluation score can accurately reflect the true quality of the tag to be evaluated, and also prove that the problem details (such as missing tags, incorrect tag positioning results, etc.) of each dimension (such as matching depth, keyword matching degree, etc.) are valid and can be used as a reliable basis for subsequent tag optimization.

[0173] This application addresses the problems of existing assessments being reliant on manual labor, highly subjective, and inefficient. It replaces extensive manual verification with automated quantitative assessment, significantly reducing subjective bias and workload, and efficiently adapting to the assessment needs of large-volume work orders. Furthermore, it comprehensively covers core assessment requirements such as label accuracy, completeness, and scalability through multi-dimensional assessment. Combined with dynamic adjustments to business weights, the overall score more closely reflects actual business scenarios, and the problem details can quickly pinpoint label defects. In addition, the inclusion of a manual review and verification mechanism further ensures the objectivity and accuracy of the assessment results, effectively compensating for the shortcomings of existing simple rule-based verification, such as the difficulty in identifying mislabeled or missing labels and the lack of basis for assessing extended labels. This provides scientific and reliable technical support for work order label quality control.

[0174] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0175] In one embodiment, a label evaluation apparatus is provided, which corresponds one-to-one with the label evaluation methods described in the above embodiments. For example... Figure 2 As shown, the label evaluation device includes an evaluation dataset construction unit 10, a multi-dimensional evaluation unit 20, a business weight determination unit 30, and a comprehensive evaluation result generation unit 40. Detailed descriptions of each functional module are as follows:

[0176] The evaluation dataset construction unit 10 is used to construct an evaluation data table, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated.

[0177] The multi-dimensional evaluation unit 20 is used to perform multi-dimensional evaluation on the tags to be evaluated based on the work order data and the preset set of correct tags, and obtain scores for each evaluation dimension. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation.

[0178] Business weight determination unit 30 is used to determine the business weights corresponding to each evaluation dimension;

[0179] The comprehensive evaluation result generation unit 40 is used to perform weighted fusion calculation on the scores of each evaluation dimension based on the business weight, to obtain the comprehensive evaluation score of the label to be evaluated, and to generate a problem detail for each evaluation dimension. The problem detail for each dimension includes at least one of the following: mislabeled label ID, missing label name, and invalid extended label reason.

[0180] In one embodiment of this application, the multi-dimensional evaluation unit 20 is further configured to:

[0181] Calculate the matching depth score and keyword matching score between the tag to be evaluated and the tags in the preset set of correct tags, and use them as the accuracy score;

[0182] Calculate the coverage score and the tag co-occurrence association score of the tag to be evaluated in the preset set of correct tags, and use them as the integrity score;

[0183] If the tag to be evaluated is an extended tag outside the preset set of correct tags, then the semantic relevance score between the tag to be evaluated and the work order, as well as the novelty score of the extended tag, are calculated as the extendedness score.

[0184] In one embodiment of this application, the matching depth score is calculated in the following manner, including:

[0185] Based on the semantics of the tags, and according to the preset vector dimensions, the semantic vectors of each tag to be evaluated in the set of tags to be evaluated and the semantic vectors of the tags in the preset set of correct tags are generated.

[0186] Based on the correlation between the semantic vector of each tag to be evaluated and the semantic vector of the corresponding preset correct tag, the attention weight of each tag to be evaluated and the tag in the set of the corresponding preset correct tags is calculated.

[0187] Based on the attention-weighted semantic similarity mean between all tags to be evaluated and their corresponding preset correct tags, a bidirectional matching depth score is obtained.

[0188] In one embodiment of this application, the keyword matching score is calculated as follows:

[0189] Obtain the text content of the work order corresponding to the label to be evaluated;

[0190] The text content is segmented into words, and words exceeding a preset threshold are selected to form a keyword set;

[0191] The set of keywords commonly contained in the tags to be evaluated and the preset set of correct tags, as well as the total set of keywords;

[0192] Calculate the first bidirectional attention weight between each keyword in the keyword set and its corresponding tag to be evaluated, and the second bidirectional attention weight between each keyword and its corresponding preset correct tag;

[0193] The keyword matching score is calculated based on the first bidirectional attention weight and the second bidirectional attention weight.

[0194] In one embodiment of this application, the coverage score is obtained in the following manner:

[0195] Select tags from the tags to be evaluated that match the tags in the preset set of correct tags to form a set of correctly covered tags;

[0196] Based on the correlation between the semantic vector of the work order corresponding to the label to be evaluated and the semantic vector of each label in the preset set of correct labels, the importance of each label in the preset set of correct labels to the work order corresponding to the label to be evaluated is calculated through an attention mechanism.

[0197] Calculate the sum of the first importance of all tags in the already covered correct tag set, and the sum of the second importance of all tags in the preset correct tag set;

[0198] The coverage score is calculated based on the ratio of the sum of the first importance scores to the sum of the second importance scores.

[0199] In one embodiment of this application, the tag co-occurrence association score is calculated as follows:

[0200] The tags that match the tags in the preset set of correct tags are selected from the tags to be evaluated, forming a set of correctly covered tags;

[0201] The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and each tag in the preset set of correct tags is converted into a corresponding tag semantic vector.

[0202] The semantic vector of any tag in the preset correct tag set is fused with the work order semantic vector to form a query vector. The semantic vector of the remaining tags in the preset correct tag set is fused with the work order semantic vector to form a key vector. The association weight between tags is calculated through a self-attention mechanism.

[0203] Select any tag from the already covered correct tag set, filter out the preset correct tags whose association weight exceeds a preset threshold, and form a dynamic association tag set of all the filtered preset correct tags for the work order corresponding to the tag to be evaluated.

[0204] Based on the dynamically associated label set, an attention-weighted co-occurrence association score is calculated.

[0205] In one embodiment of this application, the semantic relevance score is obtained in the following manner:

[0206] The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and the extended tag is converted into an extended tag semantic vector.

[0207] Based on the work order semantic vector and the extended tag semantic vector, the attention weight and cosine similarity of the extended tag to the key information of the work order text are calculated.

[0208] Based on the attention weights and cosine similarity, the attention level of each extended tag to the key information of the work order text is calculated.

[0209] The overall semantic relevance score is calculated by averaging the individual relevance scores of all extended tags.

[0210] In one embodiment of this application, the novelty score of the extended tag is calculated as follows:

[0211] Calculate the cosine similarity between each extended label and the labels in the preset set of correct labels;

[0212] Based on the cosine similarity, the novelty score of each extended tag is obtained;

[0213] The overall novelty score is obtained by averaging the novelty scores of all extended tags.

[0214] In one embodiment of this application, the device further includes a verification unit, used for:

[0215] The comprehensive evaluation score is verified by randomly selecting labels to be evaluated and conducting a manual review to obtain the manual review score.

[0216] Calculate the correlation coefficient between the manual review score and the comprehensive evaluation score;

[0217] If the correlation coefficient is greater than or equal to the preset threshold, then the comprehensive evaluation score and the details of problems in each dimension are confirmed to be valid.

[0218] This application addresses the problems of existing assessments being reliant on manual labor, highly subjective, and inefficient. It replaces extensive manual verification with automated quantitative assessment, significantly reducing subjective bias and workload, and efficiently adapting to the assessment needs of large-volume work orders. Furthermore, it comprehensively covers core assessment requirements such as label accuracy, completeness, and scalability through multi-dimensional assessment. Combined with dynamic adjustments to business weights, the overall score more closely reflects actual business scenarios, and the problem details can quickly pinpoint label defects. In addition, the inclusion of a manual review and verification mechanism further ensures the objectivity and accuracy of the assessment results, effectively compensating for the shortcomings of existing simple rule-based verification, such as the difficulty in identifying mislabeled or missing labels and the lack of basis for assessing extended labels. This provides scientific and reliable technical support for work order label quality control.

[0219] Specific limitations regarding the label evaluation device can be found in the limitations of the label evaluation method described above, and will not be repeated here. Each module in the aforementioned label evaluation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0220] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a tag evaluation method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0221] In this application embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the tag evaluation method described above.

[0222] In this embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, they implement the steps of the tag evaluation method described above.

[0223] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0224] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0225] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A label evaluation method, characterized in that, The method includes: Construct an evaluation data table, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated; Based on the work order data and the preset set of correct labels, the labels to be evaluated are evaluated in multiple dimensions to obtain scores for each evaluation dimension. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation. Determine the business weights corresponding to each evaluation dimension; Based on the business weights, a weighted fusion calculation is performed on the scores of each evaluation dimension to obtain the comprehensive evaluation score of the tag to be evaluated, including: calculating the matching depth score and keyword matching score between the tag to be evaluated and the tags in the preset correct tag set, as the accuracy score; calculating the coverage score and tag co-occurrence association score of the tag to be evaluated in the preset correct tag set, as the completeness score; if the tag to be evaluated is an extended tag outside the preset correct tag set, then the semantic relevance score between the tag to be evaluated and the work order and the novelty score of the extended tag are calculated as the extensibility score; wherein, the tag co-occurrence association score is calculated as follows: selecting tags that match the tags in the preset correct tag set from the tags to be evaluated to form a covered correct tag set; converting the text content of the work order corresponding to the tag to be evaluated... The process involves: generating a work order semantic vector; converting each tag in a preset set of correct tags into its corresponding semantic vector; fusing the semantic vector of any tag in the preset set of correct tags with the work order semantic vector to create a query vector; fusing the semantic vectors of the remaining tags in the preset set of correct tags with the work order semantic vector to create a key vector; calculating the association weight between tags using a self-attention mechanism; selecting any tag from the already covered set of correct tags; filtering out preset correct tags whose association weight exceeds a preset threshold; forming a dynamic association tag set for the work order corresponding to the tag to be evaluated by all the filtered preset correct tags; calculating the attention-weighted co-occurrence association score based on the dynamic association tag set; and generating a problem detail for each evaluation dimension, wherein the problem detail for each evaluation dimension includes at least one of the following: mislabeled tag ID, missing tag name, and invalid extended tag reason.

2. The label evaluation method as described in claim 1, characterized in that, The matching depth score is calculated in the following ways: Based on the semantics of the tags, and according to the preset vector dimensions, the semantic vectors of each tag to be evaluated in the set of tags to be evaluated and the semantic vectors of the tags in the preset set of correct tags are generated. Based on the correlation between the semantic vector of each tag to be evaluated and the semantic vector of the corresponding preset correct tag, the attention weight of each tag to be evaluated and the tag in the set of the corresponding preset correct tags is calculated. Based on the attention-weighted semantic similarity mean between all tags to be evaluated and their corresponding preset correct tags, a bidirectional matching depth score is obtained.

3. The label evaluation method as described in claim 1, characterized in that, The keyword matching score is calculated as follows: Obtain the text content of the work order corresponding to the label to be evaluated; The text content is segmented into words, and words exceeding a preset threshold are selected to form a keyword set; The set of keywords commonly contained in the tags to be evaluated and the preset set of correct tags, as well as the total set of keywords; Calculate the first bidirectional attention weight between each keyword in the keyword set and its corresponding tag to be evaluated, and the second bidirectional attention weight between each keyword and its corresponding preset correct tag; The keyword matching score is calculated based on the first bidirectional attention weight and the second bidirectional attention weight.

4. The label evaluation method as described in claim 1, characterized in that, The coverage score is obtained in the following way: Select tags from the tags to be evaluated that match the tags in the preset set of correct tags to form a set of correctly covered tags; Based on the correlation between the semantic vector of the work order corresponding to the label to be evaluated and the semantic vector of each label in the preset set of correct labels, the importance of each label in the preset set of correct labels to the work order corresponding to the label to be evaluated is calculated through an attention mechanism. Calculate the sum of the first importance of all tags in the already covered correct tag set, and the sum of the second importance of all tags in the preset correct tag set; The coverage score is calculated based on the ratio of the sum of the first importance scores to the sum of the second importance scores.

5. The label evaluation method as described in claim 1, characterized in that, The semantic relevance score is obtained in the following way: The text content of the work order corresponding to the tag to be evaluated is converted into a work order semantic vector, and the extended tag is converted into an extended tag semantic vector. Based on the work order semantic vector and the extended tag semantic vector, the attention weight and cosine similarity of the extended tag to the key information of the work order text are calculated. Based on the attention weights and cosine similarity, the attention level of each extended tag to the key information of the work order text is calculated. The overall semantic relevance score is calculated by averaging the individual relevance scores of all extended tags.

6. The label evaluation method as described in claim 1, characterized in that, The novelty score of the extended tag is calculated as follows: Calculate the cosine similarity between each extended label and the labels in the preset set of correct labels; Based on the cosine similarity, the novelty score of each extended tag is obtained; The overall novelty score is obtained by averaging the novelty scores of all extended tags.

7. The label evaluation method according to any one of claims 1-6, characterized in that, After obtaining the comprehensive evaluation score of the label to be evaluated, the process includes: The comprehensive evaluation score is verified by randomly selecting labels to be evaluated and conducting a manual review to obtain the manual review score. Calculate the correlation coefficient between the manual review score and the comprehensive evaluation score; If the correlation coefficient is greater than or equal to the preset threshold, then the comprehensive evaluation score and the details of problems in each dimension are confirmed to be valid.

8. A label evaluation device, characterized in that, The device includes: An evaluation dataset construction unit is used to construct an evaluation data table, which includes work order data, a preset set of correct labels, and a set of labels to be evaluated. The multi-dimensional evaluation unit is used to perform multi-dimensional evaluation on the tags to be evaluated based on the work order data and a preset set of correct tags, and obtain scores for each evaluation dimension. The evaluation dimensions include at least two of the following: accuracy evaluation, completeness evaluation, and extensibility evaluation. The business weight determination unit is used to determine the business weights corresponding to each evaluation dimension; The comprehensive evaluation result generation unit is used to perform weighted fusion calculation on the scores of each evaluation dimension based on the business weight to obtain the comprehensive evaluation score of the tag to be evaluated, including: calculating the matching depth score and keyword matching score between the tag to be evaluated and the tags in the preset correct tag set as the accuracy score; calculating the coverage score and tag co-occurrence association score of the tag to be evaluated in the preset correct tag set as the completeness score; if the tag to be evaluated is an extended tag outside the preset correct tag set, then calculating the semantic relevance score between the tag to be evaluated and the work order and the novelty score of the extended tag as the extensibility score; wherein, the tag co-occurrence association score is calculated as follows: selecting tags that match the tags in the preset correct tag set from the tags to be evaluated to form a covered correct tag set; and assigning the work order corresponding to the tag to be evaluated... The text content is converted into a work order semantic vector, and each tag in the preset correct tag set is converted into a corresponding tag semantic vector; the semantic vector of any tag in the preset correct tag set is fused with the work order semantic vector to form a query vector, and the semantic vector of the remaining tags in the preset correct tag set is fused with the work order semantic vector to form a key vector. The association weight between tags is calculated through a self-attention mechanism; any tag is selected from the already covered correct tag set, and preset correct tags with association weights exceeding a preset threshold are filtered out. All the filtered preset correct tags are combined into a dynamic association tag set for the work order corresponding to the tag to be evaluated; based on the dynamic association tag set, the attention-weighted co-occurrence association score is calculated; and a problem detail for each evaluation dimension is generated, wherein the problem detail for each evaluation dimension includes at least one of the following: mislabeled tag ID, missing tag name, and invalid extended tag reason.

Images