A telecommunications complaint work order classification method based on a hierarchical label fusion model

Abstract

The application discloses a kind of based on hierarchical label fusion model's telecommunication complaint work order classification method.The method is first to work order data is washed and standardized, constructs and includes first, second, third level label hierarchical label system, and based on telecommunication service real-time load coefficient dynamically adjusts data enhancement intensity, to alleviate data imbalance problem.Core model generates semantic vector of text and label by semantic coding module respectively;Semantic matching module calculates cosine similarity and outputs probability distribution;Hierarchical weighted FocalLoss module enhances attention to bottom long tail label;Hierarchical constraint reasoning module implements "parent label must be selected" rule by loss mask and mutual exclusion penalty.The method can effectively fuse hierarchical label semantics, improve classification accuracy, hierarchical consistency and the recognition ability to minority class label, realize the automation, fine classification of telecommunication complaint work order.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and natural language processing technology, and in particular to a method for classifying telecommunications complaint work orders based on a hierarchical label fusion model. Background Technology

[0002] In today's digital age, the telecommunications industry plays a crucial role globally, providing people with communication, internet access, and other essential services. However, with the rapid increase in the number of users, telecommunications companies face increasingly complex tasks in processing user complaint tickets. User complaints can involve a variety of issues, including network failures, billing errors, and service quality problems, making manual processing of these complaints extremely tedious and time-consuming. Traditional complaint ticket classification methods rely primarily on manual processing, which presents several challenges. First, due to the increasing diversity and volume of complaints, manual classification is susceptible to subjective factors, leading to inconsistencies. Second, manual processing cannot meet users' expectations for immediate responses, delaying problem resolution. To address these challenges, the introduction of Natural Language Processing (NLP) technology has become a cutting-edge approach to solving the telecommunications complaint ticket classification problem.

[0003] The existing methods for classifying telecommunications complaint tickets have the following main problems and shortcomings:

[0004] Limitations of single-label classification methods: Traditional single-label classification methods cannot effectively handle the complex hierarchical structure of telecommunications complaint tickets. Since complaints often involve multiple levels of categories, single-label classification methods can only categorize tickets into a fixed category, failing to capture the multi-layered semantic information of the complaint content and thus reducing classification accuracy.

[0005] Ignoring hierarchical relationships: Existing methods typically ignore the hierarchical relationships between labels, treating them as independent categories. This approach fails to fully utilize the hierarchical structure of labels, causing the model to misunderstand the relationships between labels during prediction and affecting classification performance.

[0006] Imbalanced data is a common problem in telecommunications complaint work orders, where the distribution of labels is often imbalanced, with some categories having a significantly larger sample size than others. Traditional methods, when faced with imbalanced data, tend to favor predicting the category with the larger sample size, leading to poor classification performance for minority categories and impacting the model's generalization ability.

[0007] Insufficient generalization ability: Existing classification models often lack generalization ability when faced with the diverse content of telecommunications complaint tickets. The models rely too heavily on a large number of samples of a specific category during training, making it difficult to accurately classify new problems in real-world scenarios.

[0008] Performance evaluation is singular:

[0009] Traditional methods typically rely on single metrics such as accuracy for performance evaluation, making it difficult to comprehensively assess a model's performance across different classes, especially minority classes. This lack of comprehensive evaluation of classification models within hierarchical label structures can lead to poor results in practical applications.

[0010] Lack of adaptability: As telecommunications services and user needs change, the content and categories of complaint tickets are also constantly being updated. The existing model is not adaptable enough to face new categories and problems, and cannot update and adjust the classification strategy in a timely manner.

[0011] Therefore, there is an urgent need for an intelligent classification method that can effectively integrate multi-level business tag information, deeply understand the semantics of complex complaint texts, and accurately capture the hierarchical relationships between tags, so as to overcome the limitations of existing technologies, realize the automated, refined, and highly accurate classification of telecommunications complaint work orders, and provide a solid foundation for subsequent efficient processing and root cause analysis. Summary of the Invention

[0012] To address the aforementioned technical problems, the present invention provides a telecommunications complaint ticket classification method based on a hierarchical label fusion model, which includes the following steps:

[0013] Hierarchical Tagging System Construction: Telecom complaint work order data is cleaned, removing meaningless words, special symbols, and invalid text, and business entities are standardized and mapped. A hierarchical tagging system containing first-level, second-level, and third-level tags is constructed, clarifying the hierarchical relationships between tags. A stratified sampling method is used to divide the dataset into training, validation, and test sets. The data augmentation intensity is dynamically adjusted based on the real-time load factor of telecommunications services. This real-time load factor is a dynamic parameter reflecting the system's processing capacity, introduced from the telecommunications network operation and maintenance system. When the load factor is higher than a threshold, the augmentation factor is reduced to maintain processing efficiency; when the load factor is lower than a threshold, the augmentation factor is increased to improve data diversity.

[0014] Hierarchical label fusion model construction: A classification model based on a pre-trained language model is constructed, including a semantic encoding module, a semantic matching module, and a hierarchical constraint module. The semantic encoding module generates semantic vectors for the work order text and hierarchical labels, respectively. The semantic matching module calculates the similarity between the text and the label semantic vectors and outputs the probability distribution. The hierarchical constraint module implements hierarchical logical constraints through a loss adjustment mechanism to ensure the parent label must be selected and handle label mutual exclusion relationships.

[0015] Model training: Configure model training parameters, including optimizer, learning rate strategy and batch settings, and use early stopping mechanism and cross-validation process to optimize the training process, prevent overfitting and improve the model's generalization ability;

[0016] The preprocessed work order text is input into the trained model, and through text semantic encoding, label semantic matching and hierarchical constraint reasoning, the model outputs classification results and predicted probabilities containing first-level, second-level and third-level labels.

[0017] As a further aspect of the present invention, the specific steps for cleaning the telecommunications complaint work order data are as follows:

[0018] Meaningless words are removed based on a professional stop word list in the telecommunications field. Special symbols, system-generated number prefixes, and garbled characters in the text are matched and deleted using regular expressions. Business entities are standardized and mapped. A text length threshold of 8 characters is set, and invalid work orders with a length lower than this threshold or a keyword matching score lower than 0.3 are removed.

[0019] The hierarchical labeling system is as follows: it divides the relationship between first-level, second-level, and third-level labels. There are 12 first-level labels, including "network quality", "business rules", "fee issues", "service attitude", "package content", "equipment failure", "roaming issues", "data services", "voice services", "value-added services", "complaint handling" and "other issues". Each first-level label corresponds to several second-level labels, and each second-level label is further subdivided into third-level labels. Each second-level label contains at least 3 third-level labels, and the total number of samples of all third-level labels accounts for no less than 85% of the total sample.

[0020] As a further aspect of the present invention, the step of dynamically adjusting the data enhancement strength based on the real-time load factor of telecommunications services specifically includes:

[0021] Obtain the real-time load coefficient of the current telecommunications network operation and maintenance system, and set a first load threshold and a second load threshold, wherein the first load threshold is less than the second load threshold;

[0022] When the real-time load coefficient is lower than the first load threshold, high-intensity data augmentation is applied to the long-tail tags to generate synthetic data that is 5 times the original data.

[0023] When the real-time load coefficient is between the first load threshold and the second load threshold, medium-intensity data augmentation is used to generate synthetic data that is 3 times the original data.

[0024] When the real-time load coefficient is higher than the second load threshold, low-intensity data augmentation is used to generate synthetic data that is 1 times the original data.

[0025] The data augmentation is achieved using text enhancement techniques, including one or more of synonym replacement, random insertion, and sentence rearrangement.

[0026] As a further aspect of the present invention, the semantic encoding module in the hierarchical tag fusion model specifically comprises:

[0027] A pre-trained model is used as the basic framework. The complete hierarchical path of the label is taken as the input text. The path format is "first-level label##second-level label##third-level label". When inputting, separators are inserted between each level of label. After model encoding, the output vector of the CLS position is extracted as the label semantic vector. The vector contains complete hierarchical structure information.

[0028] As a further aspect of the present invention, the semantic matching module in the hierarchical tag fusion model comprises the following specific steps:

[0029] Input the work order text into the model, extract the CLS position vector as the text semantic vector; calculate the cosine similarity between the text semantic vector and the semantic vector of each tag, using the following formula:

[0030] Here, x and y are two semantic vectors whose similarity is to be calculated. The semantic similarity is represented by the cosine distance between the two vectors. The similarity score is obtained by performing similarity analysis between the semantic vector Text_CLS of the input text and the semantic Label_CLS[i] of each label. Finally, the obtained similarity score is taken as the softmax value and then passed to the loss function for loss analysis. The specific formula is as follows: , among which d It is the vector dimension. This is the semantic hidden layer vector obtained after the work order text has been processed by the model. The semantic vector is obtained after the i-th level label is processed by the model. Calculate the semantic similarity score between the text and the i-th label;

[0031] The formula for calculating the softmax function is as follows:

[0032] The output is the probability value after softmax processing, ranging from [0, 1], and the sum of the probabilities of all tags is 1; the similarity scores of multiple tags are converted into a probability distribution, so that the probability value of each tag is distinguishable from other tags;

[0033] After the input work order text (Text) is semantically trained on the model, the text semantic hidden vector (Text_CLS) is extracted. Simultaneously, the hierarchical label semantics (Label_1, Label_2, Label_3) are trained on the model to obtain the hierarchical label semantics. , , Text_CLS and Semantic similarity analysis was performed on (i=1, 2, 3) respectively, and similarity scores were obtained. , , Then, the similarity score is calculated using the softmax function, and the result is used as the prediction result for each label.

[0034] Hierarchical weighted FocalLoss module: Employs an improved FocalLoss loss function, calculated as follows: ,in To predict the probability, when the true label is 1, ,otherwise ; As a balancing factor, it is dynamically adjusted based on the sample percentage; labels with a sample percentage <5% are included. Labels with a sample size of 5%-20% Labels with a sample share > 20% ; This represents the tag level, with values ​​of 1, 2, and 3. The focus parameter is set to 2.0; this is achieved through the hierarchical coefficient. Increase the loss weight of the underlying labels to enhance the model's attention to long-tail labels.

[0035] As a further aspect of the present invention, the hierarchical constraint reasoning module in the hierarchical label fusion model includes:

[0036] The above modules follow the "parent label must be selected" rule. If the model predicts the third-level label L3, then its corresponding second-level label L2 and first-level label L1 must be selected simultaneously. This constraint is implemented through a loss mask mechanism. When L3 is predicted but L1 or L2 is not predicted, the loss value of L3 is weighted and penalized to maintain hierarchical logical consistency. At the same time, a label co-occurrence constraint matrix is ​​established, and mutual exclusion penalty terms are set for mutually exclusive labels. If mutually exclusive labels are predicted simultaneously, an additional 5 times loss is added.

[0037] As a further aspect of the present invention, the semantic matching module for calculating the similarity between text and tag semantic vectors also includes a semantic correction mechanism: when the similarity between the text semantic vector and a certain tag semantic vector is >0.8 but the similarity between the tag's parent tag is <0.3, the weight of the current tag is automatically reduced, and a second matching of the parent tag is triggered to recalculate the similarity between the text and all child tags under the parent tag, thereby avoiding cross-level mismatches.

[0038] Beneficial effects: 1) Improved loss function: An improvement on the classic cross-entropy loss function has been made, making it particularly suitable for handling the problem of imbalanced labels. By attenuating the loss of easily classified samples, the model focuses more on samples that are difficult to classify, thereby effectively improving the model's learning ability for minority class samples.

[0039] 2) Improved Label Prediction Method: By performing multi-label prediction on three levels of labels—"Channel Service," "Channel Service ##Physical Channel," and "Channel Service ##Physical Channel ##Acceptance Error"—the model can capture the hierarchical relationships between labels. This improvement structurally aligns more closely with actual business scenarios and better reflects the multi-layered attributes of complaint work orders. The multi-label prediction method allows the model to refine predictions layer by layer, improving the understanding and classification accuracy of complex work orders. This not only enhances the model's predictive capabilities but also makes the classification results more consistent with actual business logic.

[0040] 3) Propose fusing label semantics: By performing similarity analysis between the semantic vector of the text input and the semantic vector of the label, the model considers not only the semantic information of the text but also the semantic information of the label. This makes the model more accurate in processing semantically similar labels, significantly improving the model's semantic understanding ability. At the same time, it enables the model to more accurately identify the correspondence between text and labels, especially when facing complex or similar labels, providing more accurate classification results. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the steps of a telecommunications complaint work order classification method based on a hierarchical label fusion model according to the present invention.

[0042] Figure 2 This is a schematic diagram of the architecture of the hierarchical label fusion model. Detailed Implementation

[0043] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Example 1

[0044] Please see Figure 1 The diagram shows the steps of a telecommunications complaint ticket classification method based on a hierarchical label fusion model according to the present invention. The specific steps are as follows:

[0045] S100 Hierarchical Label System Construction: The construction of the hierarchical label system is the foundation of the entire classification method. Its core lies in eliminating noise through data preprocessing, constructing a hierarchical label structure that conforms to business logic, rationally partitioning the dataset, and specifically addressing the data imbalance problem to provide high-quality data support for subsequent model training. Specific steps include:

[0046] S110 Data Cleaning: This involves refined preprocessing of raw telecommunications complaint ticket data to eliminate noise, redundancy, and invalid information. First, a professional stop word list for the telecommunications field is used for filtering. This list, compiled from a massive amount of telecommunications complaint tickets, summarizes high-frequency words lacking business semantics, including communication terms, modal particles, and filler words without substantial meaning. During the filtering process, precise string matching and semantic similarity are used to ensure that only truly meaningless words are removed, without overlooking business-related expressions.

[0047] Next, special symbols and invalid characters are cleaned up using regular expressions for global matching. The regular expression pattern is designed to target common invalid character types in telecom work orders, including punctuation marks, system-generated number prefixes, and garbled characters. After matching, these characters are deleted uniformly using a batch replacement function to prevent them from interfering with the semantic encoding process.

[0048] Secondly, standardized mapping of business entities is implemented: a standardized dictionary of telecommunications business entities is established. This dictionary covers common expressions in core business areas such as network terminology, service names, fee types, and equipment types, uniformly mapping non-standard expressions in work orders to standard terms. For example, "5G network," "5G network," and "fifth-generation mobile communication" are uniformly mapped to "fifth-generation mobile communication network"; "call charges," "communication fees," and "mobile phone fees" are uniformly mapped to "communication fees"; and "data usage exceeded," "data usage exceeded," and "data usage quota exceeded" are uniformly mapped to "data traffic exceeded." The mapping process is achieved through dictionary matching and semantic similarity verification to ensure that the mapped terms accurately reflect the business meaning of the original work order.

[0049] Finally, invalid work orders are filtered: a dual filtering condition is set, consisting of a text length threshold and a keyword matching score threshold. The text length threshold is set because, based on extensive business statistical analysis, the core information of valid telecommunications complaint work orders typically requires a certain length of text. Text that is too short often fails to clearly express the complaint's intent; therefore, a text length threshold of 8 characters is set, and work orders shorter than this threshold are removed. The keyword matching score threshold is calculated using the TF-IDF algorithm, matching the work order text against a core terminology list of telecommunications services. Work orders with a score below 0.3 are considered invalid. This threshold is based on multiple experimental verifications; when the score is below 0.3, the work order has extremely low relevance to telecommunications services, and is often a mistaken submission or meaningless content. Removing such work orders significantly improves data quality and avoids the accidental deletion of valid work orders.

[0050] S120 constructs a hierarchical tagging system: defining a three-level tag structure and clarifying the hierarchical relationships and logic between tags. The first-level tags focus on the core areas of telecommunications services, comprising 12 tags: "Network Quality," "Business Rules," "Cost Issues," "Service Attitude," "Package Content," "Equipment Failure," "Roaming Issues," "Data Services," "Voice Services," "Value-Added Services," "Complaint Handling," and "Other Issues." Each first-level tag corresponds to a core complaint scenario. Second-level tags are further subdivisions of the first-level tags, focusing on the main problem types within that area. For example, under "Network Quality," second-level tags include "Signal Coverage," "Network Speed ​​Performance," "Network Interruption," and "Network Latency," each corresponding to different dimensions of network quality-related issues. Third-level tags are further refinements of the second-level tags, focusing on specific problem manifestations or scenarios. For example, under "Signal Coverage," third-level tags include "No Signal Indoors," "Weak Signal in Remote Areas," "Signal Obstruction by Buildings," "Poor Signal Due to Base Station Failure," and "Frequent Signal Fluctuations," enabling precise identification of complaint issues.

[0051] Furthermore, the requirements for the number and coverage of tags are as follows: each secondary tag should have at least 3 tertiary tags to ensure the fine granularity of the tagging system and avoid overly general problem classification under secondary tags, which would affect the accuracy of classification; at the same time, the total number of samples of all tertiary tags should account for no less than 85% of the total sample size, to ensure that the vast majority of complaint work orders can be accurately classified under specific tertiary tags, reduce the proportion of "other problems" tags, and improve the practicality and relevance of classification.

[0052] Furthermore, label definition and ambiguity resolution: Detailed definition documents are written for each label, clearly defining its scope and exclusion criteria to avoid ambiguity. Precise definition ensures consistency in label annotation during model training, improving model classification accuracy.

[0053] S130 Dataset Partitioning and Data Augmentation: The dataset is partitioned using stratified sampling, allocating samples according to the proportion of each primary label in the total sample to ensure that the label distribution of each subset is consistent with the overall distribution. A typical partition ratio is 80% training set, 10% validation set, and 10% test set. The test set contains a separate number of samples of newly added complaint types to verify the model's generalization ability.

[0054] Furthermore, data augmentation is implemented for long-tail tags to alleviate data imbalance. The augmentation intensity is dynamically adjusted based on the real-time load factor of telecommunications services. This factor is obtained in real time from the telecommunications network operation and maintenance system and reflects the dynamic changes in system processing capacity, such as the normalized values ​​of indicators like CPU utilization and memory usage. Two load thresholds are set: a first load threshold (low load) and a second load threshold (high load). When the real-time load factor is lower than the first load threshold, system resources are sufficient, and high-intensity data augmentation is used, generating synthetic data that is many times larger than the original long-tail tag sample size to maximize data diversity. When the real-time load factor is between the first and second load thresholds, medium-intensity data augmentation is used to generate an appropriate amount of synthetic data, achieving a balance between improving data diversity and ensuring system performance. When the real-time load factor is higher than the second load threshold, the system load is high. To avoid consuming too many system resources and affecting normal business processing, low-intensity data augmentation is used, generating only synthetic data equivalent to the original sample size, or no additional augmentation is performed. Data augmentation techniques include text enhancement methods such as synonym replacement (replacing the original words with domain thesaurus), random insertion (inserting relevant words at random positions), and sentence rearrangement (adjusting the order of sentences). All generated synthetic data must undergo semantic consistency verification to ensure that it conforms to the expression habits of telecommunications complaint work orders and does not contain semantic confusion or content unrelated to the tags.

[0055] S200 hierarchical label fusion model construction: The model includes the following modules:

[0056] Semantic Encoding Module: Employing the pre-trained language model ERNIE as the basic framework, this module generates semantic vectors for both work order text and hierarchical labels. For work order text, it is directly input into the model, where a multi-layer Transformer encoder extracts contextual semantics, ultimately using the output vector at the CLS (classification) position as the text semantic vector Text_CLS. For label encoding, the complete hierarchical path of the label is used as input text, formatted as "Level 1 Label ## Level 2 Label ## Level 3 Label", where "##" serves as a separator. After inputting into the model, it is also encoded using Transformer layers, extracting the output vector at the CLS position as the label semantic vector Label_CLS. This vector captures the hierarchical structure information of the label; for example, the label "Network Quality ## Signal Coverage ## No Signal Indoors" is encoded as a fixed-dimensional semantic vector. The encoding process includes word segmentation, word vector conversion, multi-layer self-attention mechanism, and feedforward neural network processing. A dynamic masking strategy is used, randomly replacing or deleting 50% of the characters in the input text to enhance model robustness. This ratio, chosen experimentally, effectively prevents overfitting without sacrificing semantic integrity.

[0057] Semantic matching module: This module calculates the similarity between the semantic vector of the work order text and the semantic vectors of each label, and outputs the probability distribution. The specific steps are as follows: Input the work order text into the pre-trained model, extract the CLS position vector as the text semantic vector Text_CLS. Then, calculate the cosine similarity between Text_CLS and each label semantic vector Label_CLS[i]. The cosine similarity formula is defined as:

[0058] ,

[0059] Where x and y are two semantic vectors whose similarity is to be calculated, and n is the vector dimension. and It is the i-th element of the vector. The cosine similarity value ranges from -1 to 1, with values ​​closer to 1 indicating greater semantic similarity. In actual calculations, the similarity score (similarity_score) is achieved through vector dot product normalization, as shown in the formula:

[0060] ;in d It is the vector dimension. and It is the i-th element of the vector. The score is converted into a probability distribution using the softmax function, ensuring that the output value is in the range [0,1] and sums to 1.

[0061] The formula for the softmax function is: Where K is the total number of tags, This is the predicted probability of the i-th label. This transformation makes the probability value of each label distinguishable from other labels.

[0062] For further details, please refer to Figure 2 The diagram shows the architecture of the hierarchical label fusion model. The input work order text (Text) is trained on the model with contextual semantics, and the hidden semantic vector (Text_CLS) is extracted. Simultaneously, the hierarchical label semantics (Label_1, Label_2, Label_3) are trained on the ERNIE model to obtain the hierarchical label semantics (label_1_CLS, label_2_CLS, Label_3_CLS). Then, semantic similarity analysis is performed between Text_CLS and Label_i_CLS (i=1, 2, 3) to obtain similarity scores (similarity_score_1, similarity_score_2, similarity_score_3). Finally, a softmax operation is performed on the similarity scores, and the final result is used as the prediction result for each label.

[0063] In addition, a semantic correction mechanism is included: when a tag has a similarity score higher than 0.8 with the work order text (indicating a high semantic similarity between the text and the tag), but the similarity score between the corresponding parent tag and the text is lower than 0.3 (indicating a very low semantic association between the text and the parent tag), it is determined that there may be a cross-level mismatch. At this time, the weight of the current tag is automatically reduced, and a secondary matching process for the parent tag is triggered to recalculate the similarity between the work order text and all child tags under the parent tag, ensuring that the matching results conform to the hierarchical logic. The reason for setting the threshold for this mechanism is that a high similarity threshold of 0.8 ensures a strong association between the text and the tag, while a low similarity threshold of 0.3 ensures a weak association between the text and the parent tag. The combination of the two can effectively identify cross-level mismatch scenarios and improve the accuracy of classification.

[0064] Hierarchical Weighted FocalLoss Module: Employs an improved hierarchical weighted FocalLoss to address data imbalance and hierarchical concerns. The formula is defined as: ;in, It is the predicted probability, when the true label is 1. When the true label is 0, . It is a balancing factor that is dynamically adjusted based on the sample percentage: when the sample percentage of labels is less than 5%, =0.8; when the sample proportion of labels is 5% to 20%, =0.5; When the sample percentage of the label is greater than 20%, =0.2. This setting assigns higher weights to the minority class, mitigating sample imbalance. `level` represents the label hierarchy, with values ​​of 1, 2, and 3, corresponding to first-level, second-level, and third-level labels, respectively. Hierarchy coefficient. To enhance the loss weights of the lower-level labels, the hierarchy coefficients are 1.0 for the third-level labels, 0.5 for the second-level labels, and 0.0 for the first-level labels. The rationale for this design is that the lower-level labels (second and third levels) are fine-grained classification labels, which are more critical for locating the complaint issues, and the sample size is often smaller, requiring higher loss weights to enhance the model's learning ability for them. This is the focus parameter, set to 2.0.

[0065] Hierarchical Constraint Module: This module implements hierarchical logical constraints to ensure that the prediction results conform to the label tree structure. The main rule is "parent label must be selected": If the model predicts a third-level label as positive, then its corresponding second-level and first-level labels must also be predicted as positive; similarly, if a second-level label is predicted as positive, then its corresponding first-level label must be predicted as positive. This rule is implemented through a loss mask mechanism. When calculating the loss value, if a third-level label is predicted as positive but its corresponding second-level or first-level label is negative, or a second-level label is predicted as positive but its corresponding first-level label is negative, a weighted penalty is applied to the loss value of that third-level or second-level label. The penalty coefficient is set to a reasonable value based on experimental verification to ensure that the model can quickly learn the hierarchical logical relationships. For example, if the model predicts "Network quality ## Signal coverage ## No signal indoors" as a positive class, but does not predict "Network quality ## Signal coverage" or "Network quality" as a positive class, then the loss value of "No signal indoors" will be additionally weighted, forcing the model to predict the corresponding upper-level label at the same time when predicting the lower-level label.

[0066] Furthermore, the mutual exclusion label constraint is implemented as follows: First, a label co-occurrence constraint matrix is ​​established. Based on the telecommunications service logic, mutually exclusive label pairs are defined, meaning that the complaint issues represented by two labels cannot occur simultaneously. For example, the "network quality" label and the "equipment failure" label are mutually exclusive because "network quality problems" are caused by network-side issues, while "equipment failure problems" are caused by user terminal equipment issues. Their causes are completely different and cannot coexist in the same complaint ticket. Similarly, the "voice service" label and the "data service" label are mutually exclusive in some scenarios. If a complaint ticket clearly reflects a voice call problem, it cannot simultaneously reflect a data traffic problem. During model prediction, if a pair of mutually exclusive labels is predicted to be positive simultaneously, an additional 5-fold penalty is added to the loss calculation. This penalty coefficient has been verified through multiple experiments; the 5-fold penalty effectively suppresses the simultaneous prediction of mutually exclusive labels without excessively affecting the normal convergence of the model.

[0067] S300 Model Training: Model training involves parameter configuration, optimization strategies, loss function design, and evaluation mechanisms to ensure efficient model learning and generalization. The main steps are as follows:

[0068] S310 parameter configuration: The AdamW optimizer is used, with parameters set to β1=0.9 to control the decay rate of first-order momentum, enabling the model to quickly track the latest changes in gradient; β2=0.98 to control the decay rate of second-order momentum, making the model less sensitive to gradient fluctuations; and weight decay=0.01 to suppress excessive growth of model parameters and avoid overfitting.

[0069] Furthermore, a Warmup strategy is employed for the learning rate, linearly increasing from a very low value of 1e-6 to the target learning rate of 5e-5 over the first 10,000 steps, and then maintaining the target learning rate until the end of training. This strategy stabilizes the gradient in the early stages of training and avoids oscillations. Batch processing is configured with a single GPU batch size of 16, with 4 GPUs training in parallel, using cumulative gradient optimization to adapt to hardware limitations. Mixed precision training is enabled to accelerate the process and reduce memory usage. Dropout is set to 0.15 and LayerNormeps to 1e-6 to prevent overfitting. These values ​​are based on common practices and experimental tuning, balancing model capacity and regularization.

[0070] S320 Training Strategy: An early stopping mechanism is employed, monitoring the macro-F1 score on the validation set. Training is halted if the increase is less than 0.3% for 15 consecutive epochs. This threshold is based on training curve analysis, avoiding overfitting while ensuring convergence. Simultaneously, five-fold cross-validation is used, randomly dividing the training set into five subsets. Four subsets are used for training and one subset for validation each time, repeated five times, and the average performance metric is taken. Cross-validation reduces data partitioning bias and ensures model generalization ability. During training, the model is evaluated on the validation set every two epochs, recording the precision, recall, and F1 score for each label level.

[0071] Preferably, the model's parameter configuration is shown in the table below:

[0072]

[0073] S330 Model Evaluation Metrics: The model's evaluation metrics employ a multi-dimensional indicator system to avoid the limitations of a single indicator and comprehensively reflect the model's classification performance. Specifically, these include:

[0074] Precision, Recall, and F1 Score for each level of label: Calculate the average precision, average recall, and average F1 score for level 1, level 2, and level 3 labels respectively. Focus on the performance of level 3 labels, as they are fine-grained classification results and directly affect the efficiency of subsequent processing of complaint tickets.

[0075] Overall accuracy: Reflects the proportion of work orders correctly classified by the model, and is the most intuitive evaluation indicator.

[0076] Macro-F1 and micro-F1: Macro-F1 treats all labels equally, with each label having the same weight, reflecting the model's classification performance for minority class labels. Micro-F1, on the other hand, is calculated based on the prediction results of all samples, with weights proportional to the sample size, reflecting the model's classification performance for majority class labels. By comprehensively analyzing macro-F1 and micro-F1, a complete understanding of the model's performance on labels with different sample sizes can be obtained.

[0077] S400 Work Order Classification Application: The pre-processed work order text is input into a trained model, which sequentially undergoes semantic encoding, semantic matching, and hierarchical constraint inference. The output is a complete classification result containing first-level, second-level, and third-level labels, each with a corresponding predicted probability. A prediction probability threshold of 0.5 is set; only labels with a prediction probability ≥ 0.5 are considered the final classification result. A classification confidence report is generated, including the similarity score for each level of label, the predicted probability, the hierarchical consistency check result (whether the "parent label must be selected" rule is satisfied), and the mutual exclusion label check result (whether mutually exclusive labels are predicted simultaneously), providing a reference for subsequent complaint work order processing and root cause analysis.

[0078] Through the detailed implementation steps described above, this method can fully utilize the semantic information and structural relationships of hierarchical tags to effectively solve problems such as data imbalance, neglect of hierarchical relationships, and insufficient generalization ability in telecommunications complaint ticket classification. It achieves automated, refined, and highly accurate ticket classification, providing strong support for telecommunications companies to improve complaint handling efficiency and optimize service quality. Example 2

[0079] This embodiment takes the complaint ticket processing scenario of a provincial telecommunications operator (hereinafter referred to as "Operator A") as the application object, selects 482,000 historical tickets from 6 months, covering 12 core complaint scenarios such as network, fees, and services, and retains 457,000 valid tickets after cleaning.

[0080] First, complaint work orders underwent data cleaning, hierarchical tagging system construction, and dynamic data augmentation. Meaningless expressions (such as "Hello" or "Please handle this") were removed using a stop word list in the telecommunications field. Regular expressions were used to clean system numbers and garbled characters. Non-standard expressions such as "5G network unstable" and "overcharged call charges" were standardized as business entities (accuracy rate reached 98.2% in sample verification). Finally, invalid work orders were filtered out, retaining 94.8% of valid data. Based on 12 primary tags, 45 secondary tags and 182 tertiary tags were refined, ensuring that each secondary tag has at least 3 tertiary tags, and that the tertiary tag samples cover 90.1% of the total data (meeting the "≥85%" requirement). Based on the real-time load coefficient of the operator's network operation and maintenance system (reflecting CPU / memory usage), the augmentation intensity was adjusted according to low, medium, and high loads. Under low load, long-tail tags generated 5 times more synthetic data, while under high load, only the original data was retained. After augmentation, the average sample size of long-tail tags increased by 3.2 times, and the degree of data imbalance was significantly reduced.

[0081] The processed data labels are then input into the hierarchical label fusion model in the format "Level 1## Level 2## Level 3". CLS vectors are extracted as label semantic vectors, and the work order text is encoded to generate text semantic vectors. A 15% character mask is dynamically applied to enhance model robustness. Semantic matching and correction are then performed, calculating the cosine similarity between the label and the actual label. If a label's similarity is >0.8 but its parent label's similarity is <0.3, the label's weight is automatically reduced, and a second matching is performed to avoid cross-level misjudgments. Next, hierarchical constraints and loss optimization are implemented, using a loss mask to ensure that "the parent label is always selected".

[0082] Subsequently, the model was trained according to the preset parameter configuration, outputting results containing first to third-level labels and predicted probabilities. The effectiveness was then validated using a test set of 45,700 work orders. Key performance indicators of this invention were compared with traditional methods (SVM, ordinary BERT), and the results are shown in the table below:

[0083]

[0084] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A method for classifying telecommunications complaint work orders based on a hierarchical label fusion model, characterized in that, Includes the following steps: Hierarchical Tagging System Construction: Telecom complaint work order data is cleaned, removing meaningless words, special symbols, and invalid text, and business entities are standardized and mapped. A hierarchical tagging system containing first-level, second-level, and third-level tags is constructed, clarifying the hierarchical relationships between tags. A stratified sampling method is used to divide the dataset into training, validation, and test sets. The data augmentation intensity is dynamically adjusted based on the real-time load factor of telecommunications services. This real-time load factor is a dynamic parameter reflecting the system's processing capacity, introduced from the telecommunications network operation and maintenance system. When the load factor is higher than a threshold, the augmentation factor is reduced to maintain processing efficiency; when the load factor is lower than a threshold, the augmentation factor is increased to improve data diversity. Hierarchical label fusion model construction: A classification model based on a pre-trained language model is constructed, including a semantic encoding module, a semantic matching module, and a hierarchical constraint module. The semantic encoding module generates semantic vectors for the work order text and hierarchical labels, respectively. The semantic matching module calculates the similarity between the text and the label semantic vectors and outputs the probability distribution. The hierarchical constraint module implements hierarchical logical constraints through a loss adjustment mechanism to ensure the parent label must be selected and handle label mutual exclusion relationships. The hierarchical constraint module follows the "parent label must be selected" rule. If the model predicts the third-level label L3, then its corresponding second-level label L2 and first-level label L1 must be selected simultaneously. This constraint is implemented through a loss mask mechanism. When L3 is predicted but L1 or L2 is not predicted, a weighted penalty is applied to the loss value of L3 to maintain hierarchical logical consistency. At the same time, a label co-occurrence constraint matrix is ​​established, and a mutual exclusion penalty term is set for mutually exclusive labels. If mutually exclusive labels are predicted simultaneously, an additional 5 times the loss is added. Hierarchical weighted FocalLoss module: Employs an improved FocalLoss loss function, calculated as follows: ,in To predict the probability, when the true label is 1, ,otherwise =; As a balancing factor, it is dynamically adjusted according to the sample proportion; labels with a sample proportion of <5% are included. Labels with a sample size of 5%-20% Labels with a sample share > 20% ; This represents the tag level, with values ​​of 1, 2, and 3. The focus parameter is set to 2.0; this is achieved through the hierarchical coefficient. Increase the loss weight of the underlying labels to enhance the model's attention to long-tail labels; Model training: Configure model training parameters, including optimizer, learning rate strategy and batch settings, and use early stopping mechanism and cross-validation process to optimize the training process, prevent overfitting and improve the model's generalization ability; The preprocessed work order text is input into the trained model, and through text semantic encoding, label semantic matching and hierarchical constraint reasoning, the model outputs classification results and predicted probabilities containing first-level, second-level and third-level labels.

2. The telecommunications complaint ticket classification method based on a hierarchical label fusion model according to claim 1, characterized in that, The specific steps for cleaning telecommunications complaint ticket data are as follows: Meaningless words are removed based on a professional stop word list in the telecommunications field. Special symbols, system-generated number prefixes, and garbled characters in the text are matched and deleted using regular expressions. Business entities are standardized and mapped. A text length threshold of 8 characters is set, and invalid work orders with a length lower than this threshold or a keyword matching score lower than 0.3 are removed. The hierarchical labeling system is as follows: It divides the relationships between first-level, second-level, and third-level labels. There are 12 first-level labels, including "Network Quality," "Business Rules," "Cost Issues," "Service Attitude," "Package Content," "Equipment Failure," "Roaming Issues," "Data Services," "Voice Services," "Value-Added Services," "Complaint Handling," and "Other Issues." Each first-level label corresponds to several second-level labels, and each second-level label is further subdivided into third-level labels. Each second-level label contains at least three third-level labels, and the total number of samples for all third-level labels accounts for no less than 85% of the total sample size.

3. The telecommunications complaint ticket classification method based on a hierarchical label fusion model according to claim 1, characterized in that, The steps of dynamically adjusting the data enhancement strength based on the real-time load factor of telecommunications services specifically include: Obtain the real-time load coefficient of the current telecommunications network operation and maintenance system, and set a first load threshold and a second load threshold, wherein the first load threshold is less than the second load threshold; When the real-time load coefficient is lower than the first load threshold, high-intensity data augmentation is applied to the long-tail tags to generate synthetic data that is 5 times the original data. When the real-time load coefficient is between the first load threshold and the second load threshold, medium-intensity data augmentation is used to generate synthetic data that is 3 times the original data. When the real-time load coefficient is higher than the second load threshold, low-intensity data augmentation is used to generate synthetic data that is 1 times the original data. The data augmentation is achieved using text enhancement techniques, including one or more of synonym replacement, random insertion, and sentence rearrangement.

4. The telecommunications complaint ticket classification method based on a hierarchical label fusion model according to claim 1, characterized in that, The semantic encoding module in the hierarchical label fusion model contains the following specific content: A pre-trained model is used as the basic framework. The complete hierarchical path of the label is taken as the input text. The path format is "first-level label##second-level label##third-level label". Separators are inserted between each level of label during input. After model encoding, the output vector of the CLS position is extracted as the label semantic vector. The vector contains complete hierarchical structure information.

5. The telecommunications complaint ticket classification method based on a hierarchical label fusion model according to claim 1, characterized in that, The semantic matching module also includes a semantic correction mechanism for calculating the similarity between text and tag semantic vectors: when the similarity between the text semantic vector and a certain tag semantic vector is >0.8 but the similarity between the tag's parent tag is <0.3, the weight of the current tag is automatically reduced, and the secondary matching of the parent tag is triggered to recalculate the similarity between the text and all child tags under the parent tag, thus avoiding cross-level mismatches.

Images