A MULTI-CLASS AND UNBALANCED TEXT CLASSIFICATION SYSTEM AND METHOD BASED ON THE INTEGRATION OF UNCERTAINTY AND MODEL MISMATCH.

Abstract

The invention relates to an integrated system (10) and method developed to address the class imbalance and prediction uncertainty problems that arise in multi-class text classification processes. The invention presents a two-level meta-learning architecture that operates in a hybrid meta-feature space with class probabilities obtained from models with different architectures, uncertainty measures derived from these outputs, and inter-model discrepancy indicators. This invention integrates prediction methods that prevent data leakage, nonlinear meta-learning processes, and class-based decision threshold optimization stages that enable the reshaping of decision boundaries according to data distribution, thereby producing more reliable, balanced, and stable classification results, especially in underrepresented classes.

Description

1 TARIFF UNCERTAINTY AND MODEL MISMATCH BASED ON INTEGRATION CLASSIFICATION SYSTEM AND METHOD FOR CLASSIFIED AND UNBALANCED TEXT TECHNICAL FIELD This invention has implications for natural language processing and machine learning, particularly for multi-class text. Class imbalance and estimation uncertainty arising in classification processes It relates to a method for resolving problems. 10 The invention enables the simultaneous extraction of different model outputs from a multi-class and unbalanced dataset. by evaluating not only individual estimates, but also the estimates themselves a decision-making process that takes into account confidence levels and inconsistencies between models It presents an approach. In this context, the method is derived from 15 multiple model outputs. using probability distributions, particularly for underrepresented and marginalized classes restructuring the decision-making process in a way that will allow for more accurate determination and a method that produces more balanced and stable results in multi-class, unbalanced data structures It provides classification infrastructure. STATE OF THE ART Today, the volume of text-based data produced in digital environments across various fields is... This data is increasing rapidly, and it needs to be automatically analyzed and classified. It has become a technical necessity. In the technical field, user feedback, emotions 25 For the purpose of analyzing their situations and complex text structures; classical machine learning methods, deep learning-based networks, and context-aware transformers Their architectures are widely used. In recent years, the outputs of different models have been... combining "ensemble" and "stacking" to improve performance 30-based approaches have been developed; studies shared on platforms such as SemEval Several strategies have been proposed for combining multiple classifiers. However, current text classification approaches, especially multi-class ones, Certain technical limitations when it comes to problems and unbalanced data distributions. 2 and includes reliability problems. The fundamental aspect of the current state of the technique. The shortcomings and technical problems are detailed below:  Class Imbalance and Majority Bias: In datasets, some classes are more likely than others. underrepresentation leads models to tend towards the majority class. and the emergence of high error rates in underrepresented critical classes 5 This is the reason. This situation is particularly relevant for those who remain ambiguous or on the border between classes. This makes it technically difficult to parse the texts correctly.  Uncertainty and Lack of Reliability Measurement: Current technical applications, models sufficient technical mechanisms to evaluate the reliability of predictions It does not include how accurate a prediction is or what degree of uncertainty there is. Because the uncertainty involved cannot be directly reflected in the decision-making process, systems The margin of error and stability cannot be measurably established.  Inability to Evaluate Model Discrepancy: Different models using the same data To what extent are the predictions on the model consistent or contradictory? Disagreement is not effectively handled in existing systems. 15 Outputs from multiple models are often combined directly; treating information on inter-model discrepancies as a separate technical layer Integrated structures are not offered.  Deficiencies in the Decision-Making Process: The literature addresses the problem of unbalanced data. Proposed rebalancing, data boosting, or class-weighted loss functions 20 Solutions like focal-loss (etc.) address the problem primarily during the training phase. However, multiple model uncertainty and inter-class differences can be considered during the decision-making process. structures that consider the difficulty of differentiation together in a hybrid meta-feature space It remains quite limited.  Independent Module Structure: Most existing solutions either only have 25 individual modules. to improve model performance or simpler combining methods It focuses on underrepresentation in multi-class and unbalanced text data. For more reliable classification of classes; model uncertainty, inter-model analysis combining conflict resolution and decision threshold adaptation under a single system. There is no systematic approach. 30 In conclusion, the reliability of model predictions and inter-model diversity are crucial factors in making a decision. an adaptive process that integrates uncertainty and conflict awareness. A technical solution involving the mechanism is needed. This situation is described above. 3 the development of an integrated system and method to address the aforementioned problems It has made it mandatory. A BRIEF DESCRIPTION OF THE INVENTION The present invention eliminates the aforementioned disadvantages and the related technical "Uncertainty and Model Discrepancy" aims to bring new advantages to the field. Multi-class and Unbalanced Text Classification System Based on Integration and It is related to the "method". The invention is not merely a classification approach based on the use of multiple models, hybrid model probabilities, uncertainty measures, and inter-model discrepancy indicators combining them in a meta-feature vector and ultimately through class-based decision thresholds. It provides a technical decision-making mechanism that generates the classification decision. The main purpose of the invention is to address the class distinction that arises in multi-class text classification processes. The aim is to address the problems of imbalance and prediction uncertainty. In this context, multi-class and by evaluating different model outputs together in an unbalanced dataset, only not based on singular estimates, but also on the confidence level of the estimates and the models A decision-making approach is used that takes into account the inconsistencies between them. 20 Another aim of the invention is to reduce uncertainty and inter-model discrepancies in the output of each model. The aim is to present a two-level meta-learning architecture that evaluates information together with other factors. To achieve the objective, class probabilities are derived from basic models. The derived maximum probability, uncertainty measures such as entropy and margin, and vote distribution, 25 By calculating indicators of disagreement such as the number of agreements and statistical spread, a combined analysis is performed. A hybrid meta-feature space is being created. Another aim of the invention is to increase the diversity between models and to improve error correlation. The goal is to provide a stronger learning infrastructure by reducing [the number of students]. This objective is achieved by 30. so; transformer-based models with different learning paradigms (ELECTRA, BERTurk, ALBERT, DistilBERT) and hybrid neural networks (CNN+BiLSTM, ELECTRA-AWP) It is used within the same system in a combination of heterogeneous models. 4 Another purpose of the invention is to ensure the reliability of model predictions and prevent data leakage. to prevent and enable interactive learning of uncertainty components. The aim is to achieve this goal; out-of-fold estimation The prediction (OOF) approach is used, and the resulting meta-features are linear. 5 XGBoost-based systems optimized with GridSearchCV that can model non-existent relationships. It is being processed by a meta-learner. Another aim of the invention is to address the problem of underrepresented and ambiguous classes (neutral class) (etc.) by reducing the errors seen, increasing the detectability of these classes and inter-class coordination. The goal is to achieve balance. To accomplish this goal, we have examined 10 meta-learner outcomes. Unlike classical decision rules, decision boundaries are redefined according to the data distribution. Class-based threshold optimization is applied to enable its shaping. Another objective of the invention is to demonstrate that the system is not a random combination of patterns. The aim is to prove and technically verify its structure, which is based on the interaction between its components. 15 To achieve this goal, we will be optimizing for uncertainty, conflict, and thresholds. The contributions of its components to performance are systematically analyzed using component inference analysis. is being evaluated. The ultimate goal of the invention is to create a more stable, reliable and 20-year-old technology, deviating from existing techniques. The goal is to provide a technical solution that produces generalizable results. This objective is achieved through... In this way, integrating model reliability and diversity into the decision-making process, and addressing uncertainty... an integrated system with conflict-aware, adaptive decision-making mechanisms It is presented. The best way to utilize the advantages of the existing invention, together with its structure and additional elements. For understanding, it should be evaluated together with the figures explained below. is necessary. BRIEF DESCRIPTION OF THE FIGURES 30 Figure 1 is a schematic view of the system described in the invention. REFERENCE NUMBERS System 11 Electronic Devices 12 Applications 13 Database 5 14 Servers DETAILED DESCRIPTION OF THE INVENTION This detailed explanation of the invention's innovation not only helps to better understand the subject matter, but also... This is explained with examples that will not create any limiting effect. The system in question (10) is different in the process of multi-model based text classification. Uncertainty and inter-model discrepancies using outputs from architectures calculates the features, processes these features in a unified meta-feature space, and class 15 It is an integrated structure that generates the final decision through threshold-based optimization. The system (10) has the following structural features to perform its technical functions. It consists of the following components:  User-entered text data into the system and classification 20 client-side hardware that enables the results to be presented to the user. an electronic device that forms its infrastructure (11). The mentioned electronic device (11), receiving data input via the user interface, processing the results visualization and data transmission between the system and the server It refers to the terminal systems used for. 25  End-to-end execution of a multi-model-based text classification process. an application that provides (12). The mentioned application (12) provides the text data in front of processing, generating multiple model outputs, the model in question uncertainty measures and inter-model discrepancy indicators from the outputs calculation, combining this data into a hybrid meta-feature vector, 30 by implementing the meta-learning process and applying class-based threshold values It enables the production of the final classification decision.  Raw text data, probability distributions derived from multiple model outputs, derived uncertainty and inconsistency features and in the meta-learning process 6 a database that stores the intermediate data representations used (13). Database (13), meta- consistent feature generation, model comparison, and threshold optimization processes It enables its implementation.  Parallel or sequential basic classification models with different architectures It is run as such, using outer layer prediction, meta-learning and class-based threshold 5 a server where optimization processes are performed (14). In the first stage, within the scope of the system (10) of which the invention is based, text data is electronically processed. Data suitable for the multi-class classification problem is obtained from (11). The structure is created and saved to the database (13). Application (12) converts this data into natural language 10 by processing-based preprocessing steps; cleaning empty content, HTML modeling through removal of elements, character normalization, and filtering processes. It makes it suitable for the input. The system (10) incorporates different learning paradigms. A heterogeneous combination of models has been designed. In this context, on the server (14); ELECTRA, BERTurk, ALBERT and DistilBERT are among the transformer models included in the 15. Powered by CNN+BilSTM architecture and Adversarial Weight Perturbation (AWP). The ELECTRA model is executed simultaneously. This structure allows for sub-word representations, contextual attention mechanisms and sequential patterns at the word level work together. By evaluating these factors, diversity between models is increased and error correlation is analyzed. is reduced. 20 class probability distributions are obtained from each model. Technical measurements are carried out by application (12). Probabilities directly Instead of using; uncertainty measures such as maximum likelihood, entropy, and margin are used. is calculated. At the same time, the mean and standard deviation of class probabilities are also calculated. Variance is a probability-based method that includes at least one of its minimum and maximum values. Indicators of discrepancy, such as statistical dispersion values, are extracted. 25 The calculated probability, uncertainty and conflict information is provided by a single application (12). It is integrated into a combined hybrid meta-feature space. To ensure reliability. Out-of-fold prediction (OOF) approach on server (14) Data leakage is prevented by implementing these meta-features, which are nonlinear 30. It is processed by an XGBoost-based meta-learner that can model relationships, and GridSearchCV optimizes with the most suitable parameters, reducing uncertainty and discrepancies. The components are learned interactively (jointly). 7 A significant technical advantage of the invention is that it addresses the issues arising in multi-class and unbalanced data structures. It offers a direct solution to classification problems, especially for underrepresented and Application to reduce errors seen in ambiguous classes, such as the neutral class. (12) server class-based threshold optimization operation on meta-learner outputs (14) It is carried out on this. This process involves selecting only the highest probability. 5 It offers a different technical approach than the classic decision rules (argmax rule) that are based on. In this context, the application (12) involves determining separate decision thresholds for classes and the decision This allows the boundaries to be reshaped according to the data distribution. This process increases the identifiability of underrepresented classes and improves class balance. It is technically established. 10 The invention enables a systematic analysis of the contributions of the proposed components. technically in the server (14) infrastructure with component ablation study This has been verified. As a result of these analyses, the basic model outputs were used alone. Compared to other approaches, the combined use of uncertainty and conflict components is 15 It has been shown that it significantly improves performance. Furthermore, with threshold optimization... When used together, significant improvements are achieved, especially in underrepresented classes. This has been technically demonstrated. This situation shows that the method is a random combination of models. that it is not, on the contrary, that it contains a systematic structure based on the interaction between its components It reveals. In the final stage; model reliability and model diversity are decided upon. 20 integrating uncertainty-aware, conflict-aware, and adaptive decision-making into the process The system ensures the generation of outputs that have a robust mechanism. This optimized mechanism... The final classification outputs produced using the decision-making mechanism are electronic devices. (11) is presented to the user. The operation of the system (10) which is the subject of the invention is with the following technical steps: is being carried out:  The problem of text data being retrieved by the system and multi-class classification. Creating a suitable data structure.  Text data is processed through natural language processing-based preprocessing steps to create model 30 making it suitable for entry.  Creation of a multi-model structure with different architectures and transformer-based Operating models and hybrid models. 8  Obtaining class probability distributions for each model and extracting uncertainty from these distributions Calculation of criteria.  Using multiple model outputs to identify indicators of inter-model discrepancy removal.  Model-based probability, uncertainty, and discrepancy information in a single combined meta-feature 5 integration in space.  Capable of modeling nonlinear relationships on the generated meta-features. Training the final classification model using a meta-learning algorithm.  Adapting to a multi-class and unbalanced data structure on meta-learner outputs. Optimizing class-based decision thresholds to ensure this. 10  The impact of uncertainty and inconsistency components on classification performance Model behavior is validated by evaluating it using component inference analysis.  By using an optimized decision-making mechanism, especially in underrepresented classes final classification outputs in a way that will produce more balanced results creation and presentation to the user. 15

Claims

9 REQUESTS 1. Class imbalance and estimation in multi-class text classification processes It is a system (10) for solving uncertainty problems, and its feature is;  User-entered text data into the system and classification 5 client-side hardware that enables the results to be presented to the user. an electronic device that forms its infrastructure (11),  Preprocessing text data, generating multiple model outputs, word the subject is the uncertainty measures from model outputs and the discrepancy between models Calculating the indicators, these data are presented as a hybrid meta-feature vector. integration, implementation of meta-learning process and class-based thresholds a system that enables the production of the final classification decision by applying values application (12),  Raw text data, probabilistic outputs derived from multiple models distributions, derived uncertainty and discordance properties, and meta-15 a database that stores intermediate data representations used in the learning process (13),  Parallel or sequential basic classification models with different architectures It is run as, external layer prediction, meta-learning and class-based A server (14) 20 where threshold optimization operations are performed including and the application (12) and the server (14); obtained from different architectures Using the obtained outputs, uncertainty and inter-model discrepancy features can be analyzed. It will calculate, process these features in a unified hybrid meta-feature space, and It is structured to generate the final decision through class-based threshold optimization. that is. 25 2. Class imbalance and estimation in multi-class text classification processes It is a method for resolving uncertainty problems, and its characteristic feature is;  a data structure suitable for the multi-class classification problem of text data creation, 30  text data is processed through natural language processing-based preprocessing steps making it suitable for model input,  Creation of a multi-model structure with different architectures and transformers Running base-based models and hybrid models,  Obtaining class probability distributions for each model and from these distributions Calculation of uncertainty measures,  Indicators of inter-model discrepancy using multiple model outputs removal,  A single unified meta-5 of model-based probability, uncertainty, and discrepancy information. integration in the feature space,  by using a meta-learning algorithm on the generated meta-features training the final classification model,  Adapting to a multi-class and unbalanced data structure on meta-learner outputs. Optimizing class-based decision thresholds to ensure this, 10  The impact of uncertainty and discrepancy components on classification performance Model behavior is evaluated using component inference analysis. verification and  Final classification using an optimized decision mechanism 15. It includes the steps of the process.

3. A method that complies with Claim 2, and its characteristic feature is that the uncertainty criteria in question are: The process involves calculating the maximum likelihood, entropy, and margin values.

4. This is a method consistent with Claim 2, and its characteristic feature is that it includes the indicators of the dispute in question. as; mean, standard deviation, variance, minimum and related to class probabilities. at least one of the maximum values ​​of probability-based statistical dispersion values It is removal.

5. A method that complies with Claim 2, characterized by its transformer-based models; ELECTRA, BERTurk, ALBERT and DistilBERT architectures as hybrid models. with CNN+BiLSTM architecture and Adversarial Weight Perturbation (AWP) It is the use of enhanced models within a heterogeneous combination of models.

6. This is a method that complies with Claim 2, and its characteristic feature is the reliability of the model estimates. To ensure and prevent data leakage, the outer layer on the server (14) It is the application of the forecasting approach. 11 7. This is a method that complies with Claim 2, and its characteristic is that it is used in the meta-learning phase in question. The use of an XGBoost-based algorithm as a meta-learner and this The algorithm is optimized using the GridSearchCV method.

8. This is a method that complies with Claim 2, and its characteristic feature is the threshold optimization in question. In this step, instead of the classic maximum likelihood rule, the data distribution for each class is considered. This involves reshaping decision boundaries by establishing separate decision thresholds.

Images