Text classification method and system based on large model and rule engine

Abstract

The invention relates to the technical field of text classification, and provides a text classification method based on a large model and a rule engine, and the method comprises the steps: S1, storing multi-level rule classification labels, and constructing a classification rule template library; s2, receiving text data from various data sources, and preprocessing the text data; s3, performing rule matching on the text data based on the classification rule through a rule engine, and outputting a rule classification result; and S4, when any one of the following conditions is met, large language model classification is triggered: a, a classification rule is not matched; b, matching a classification rule, wherein the rule confidence is smaller than a rule confidence threshold; c, the text data length exceeds the preset text data length; d, matching a specific business scene label; outputting a model classification result; and S5, when the rule engine classification in the S3 and the large language model classification in the S4 are parallel, executing the strategy. The output reliability and the service continuity are guaranteed, and the method is suitable for scenes with high accuracy requirements such as financial compliance examination and the like.

Description

Technical Field

[0001] The present application relates to the field of text classification technology, and in particular to a text classification method and system based on a large model and a rule engine. Background Art

[0002] Traditional financial text classification relies on manually constructed rules or handcrafted features. While reliable for structured data, it struggles to cope with semantic ambiguity and long-range dependencies when dealing with massive amounts of unstructured text. Furthermore, manual maintenance costs rapidly increase with evolving regulatory policies and market terminology. Machine learning methods (such as SVM and random forests) rely on manual feature engineering, requiring domain experts to refine text features (such as TF-IDF and word embeddings). While reliable for structured or semi-structured data, they are limited by human cognition in the completeness of feature extraction and have weak generalization capabilities when dealing with unstructured financial text. This is particularly true when dealing with long-range semantic dependencies and in scenarios with few samples. With the advancement of deep learning technology, neural network-based models (such as Text CNN and BERT) have significantly improved the classification accuracy of complex text by automatically learning deep semantic features and capturing complex contextual relationships (such as multi-label associations in financial risk reports). However, these methods are limited by their reliance on massive amounts of annotated data, high computational resource requirements, and poor interpretability. Furthermore, the reliability verification of model outputs is costly in scenarios involving dynamic terminology updates and compliance audits. Summary of the Invention

[0003] In order to help solve the above technical problems, the present application provides a text classification method and system based on a large model and a rule engine.

[0004] In the first aspect, the present application provides a text classification method based on a large model and a rule engine, which adopts the following technical solutions:

[0005] A text classification method based on a large model and a rule engine, including:

[0006] S1: Store multi-level rule classification labels, build a classification rule template library, associate rule classification labels with classification rules in the classification rule template library, and set a corresponding rule confidence threshold for each classification rule;

[0007] S2: Receive text data from various data sources and preprocess the text data;

[0008] S3: The rule engine matches the text data based on the classification rules and outputs the rule classification results. The rule classification results include the text data and the corresponding classification rules, the rule classification labels, and the rule confidence level of the matching between the text data and the classification rules.

[0009] S4: Large language model classification is triggered when any of the following conditions are met:

[0010] a. No classification rules are matched;

[0011] b. The classification rule is matched and the rule confidence is less than the rule confidence threshold;

[0012] c. The length of the text data exceeds the preset text data length;

[0013] d. Match to specific business scenario tags;

[0014] The model classification label, label definition, and text to be classified are spliced ​​into the Prompt input large language model, and the model classification results are output, including text data and corresponding model classification labels, model confidence, and classification basis;

[0015] S5: When the rule engine classification of S3 and the large language model classification of S4 are carried out in parallel, the following strategy is implemented: rule score = rule confidence × (priority base score + rule weight parameter) × scenario factor, model score = model confidence × (model reliability factor + model weight parameter) × scenario factor, where the model reliability factor is the average accuracy of the large language model in classifying text data within a preset period; the rule weight parameter, model weight parameter and decision parameter are dynamically adjusted through reinforcement learning, and the decision is executed: if |rule score - model score| ≥ decision parameter, the high score result is used as the final classification result, otherwise manual review is triggered.

[0016] Preferably, S3 includes: setting a corresponding priority factor for each classification rule, and calculating a priority score based on the priority factor, where priority score = ∑ (priority factor × priority weight) + priority base score.

[0017] Preferably, the S4 includes:

[0018] Build text vectors and create text vector indexes;

[0019] Vectorize the input text data and retrieve similar samples and model classification labels based on the text vector index;

[0020] The text data to be classified, the retrieved similar samples and the model classification labels are input into the large language model, and the text data, the model classification labels, the model confidence and the classification basis are output.

[0021] Preferably, the S5 includes:

[0022] When the rule engine classification of S3 is triggered and the large language model classification of S4 is not triggered, the rule classification result is used as the final classification result. When the large language model classification of S4 is triggered based on conditions a and b, the model classification result is used as the final classification result. When the large language model classification of S4 is triggered based on conditions c and d, the strategy of step S5 is executed.

[0023] Preferably, the S5 includes: the reinforcement learning algorithm includes:

[0024] State space definition:

[0025] Construct an n-dimensional vector as the state space, integrating at least the rule confidence, model confidence, model reliability factor, scenario factor, scenario factor, and decision parameter, where n is a positive integer;

[0026] Action Space Design:

[0027] The action space includes threshold adjustment parameters and weight update parameters. The weight update parameters include rule weight update parameters and model weight update parameters. The action vector is constructed as [threshold adjustment parameters, rule weight update parameters, model weight update parameters].

[0028] Outputting the action vector based on the state space through the action space, adjusting the rule weight parameter, the model weight parameter and the decision parameter;

[0029] Reward function design:

[0030] The reward function includes accuracy reward and review rate reward:

[0031] Accuracy Reward: When the rule classification results and model classification results are consistent with the results of proactive manual review, a positive reward is given; when the rule classification results and model classification results are incorrect and no manual review is triggered, a negative reward is given; when manual review is triggered, no reward is given;

[0032] Review rate reward: the inverse of the ratio that triggers manual review;

[0033] Total reward = accuracy reward × first weight + review rate reward × second weight.

[0034] Preferably, the method further comprises S6:

[0035] When a manual review condition is met, a manual review is triggered, and the manual review condition at least includes: the definition of the classification label is the same as the definition of the preset sensitive classification label.

[0036] Preferably, the method further comprises S7:

[0037] The attention mechanism visualization tool is used to visualize the attention weights of each layer in the classification process of a large language model.

[0038] Preferably, S8 is further included:

[0039] A rule effect evaluation model is established through machine learning algorithms, rule structure, business attributes, and performance indicators are extracted as basic features, and rule complexity index and business relevance are constructed to achieve rule attribute vectorization.

[0040] Rule complexity index = number of keywords × (number of logical operators + 1) × (context constraint complexity + 1), used to quantify the logical judgment complexity of the rule;

[0041] Business relevance = sensitivity level × (1 + hit rate for small sample scenarios), which is used to measure the rule's coverage of key business scenarios.

[0042] Preferably, S9 is also included:

[0043] The system automatically screens samples through uncertainty sampling, diversity sampling, and domain knowledge guidance to form a training data set, and regularly fine-tunes the large language model through model fine-tuning technology.

[0044] In a second aspect, the present application also provides a text classification system, which adopts the following technical solutions:

[0045] A text classification system using the text classification method based on a large model and a rule engine as described in any one of the first aspects, wherein the text classification system comprises:

[0046] The input and output layer includes a configuration management module, a data access module, a data processing module, and a visual output module. The configuration management module is used to execute step S1, the data access module and the data processing module are used to execute step S2, and the visual output module is used to execute step S7.

[0047] The model processing layer includes a rule engine module, a large language model classification module, and a hybrid decision module. The rule engine module is used to execute step S3, the large language model classification module is used to execute step S4, and the hybrid decision module is used to execute step S5.

[0048] The tool chain layer includes a manual review module, a data annotation module, a model fine-tuning module and a rule evaluation module. The manual review module is used to execute step S6, the data annotation module and the model fine-tuning module are used to execute step S9, and the rule evaluation module is used to execute step S8.

[0049] In summary, the present invention has the following beneficial effects:

[0050] 1. The integration of rule engines and large language models to jointly classify text data improves the accuracy of text data classification;

[0051] 2. When the classification results of the rule engine conflict with those of the large language model, the present invention uses preset rules to select a classification result as the final classification result or trigger manual review. Furthermore, a reinforcement learning algorithm is introduced to objectively adjust the parameters in the preset rules based on the current system state, thereby further improving the accuracy of the classification results.

[0052] 3. Subsequent visualization, rule effectiveness evaluation models, and regular fine-tuning mechanisms for large models form a closed-loop optimization of "data-model-rules", ensuring output reliability and business continuity. This is particularly suitable for scenarios such as financial compliance reviews that require high interpretability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of an embodiment of a text classification method based on a large model and a rule engine of the present application;

[0054] Figure 2 This is a schematic block diagram of an embodiment of a text classification system based on a large model and a rule engine of the present application. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to the accompanying drawings. The structure and principle of the present invention will be very clear to those skilled in the art. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention.

[0056] Figure 1 This is a flowchart of an embodiment of a text classification method based on a large model and a rule engine of the present application. Figure 2 This is a schematic block diagram of an embodiment of a text classification system based on a large model and a rule engine of the present application.

[0057] Combine Figure 1 and Figure 2 It is understood that the text classification method based on the large model and rule engine of this application includes:

[0058] A text classification method based on a large model and a rule engine, wherein the text classification method based on a large model and a rule engine includes:

[0059] S1: Store multi-level rule classification labels, build a classification rule template library, associate the rule classification labels with the classification rules in the classification rule template library, and set a corresponding rule confidence threshold for each classification rule.

[0060] S2: Receive text data from various data sources and preprocess the text data.

[0061] S3: The rule engine matches the text data based on the classification rules and outputs the rule classification results. The rule classification results include the text data and the corresponding classification rules, the rule classification label, and the rule confidence level of the match between the text data and the classification rules. S3 includes setting a corresponding priority factor for each classification rule and calculating a priority score based on the priority factor. The priority score = ∑(priority factor × priority weight) + priority base score.

[0062] S4: Large language model classification is triggered when any of the following conditions are met:

[0063] a. No classification rules are matched;

[0064] b. The classification rule is matched and the rule confidence is less than the rule confidence threshold;

[0065] c. The length of the text data exceeds the preset text data length;

[0066] d. Match to specific business scenario tags;

[0067] The model classification label, label definition, and text to be classified are spliced ​​into the prompt input large language model, and the model classification results are output, including text data and corresponding model classification labels, model confidence, and classification basis. S4 includes:

[0068] Build text vectors and create text vector indexes;

[0069] Vectorize the input text data and retrieve similar samples and model classification labels based on the text vector index;

[0070] The text data to be classified, the retrieved similar samples and the model classification labels are input into the large language model, and the text data, the model classification labels, the model confidence and the classification basis are output.

[0071] When the rule engine classification of S3 is triggered and the large language model classification of S4 is not triggered, the rule classification result is used as the final classification result. When the large language model classification of S4 is triggered based on conditions a and b, the model classification result is used as the final classification result. When the large language model classification of S4 is triggered based on conditions c and d, the strategy of step S5 is executed.

[0072] S5: When the rule engine classification of S3 and the large language model classification of S4 are carried out in parallel, the following strategy is implemented: rule score = rule confidence × (priority base score + rule weight parameter) × scenario factor, model score = model confidence × (model reliability factor + model weight parameter) × scenario factor, where the model reliability factor is the average accuracy of the large language model in classifying text data within a preset period; rule weight parameters, model weight parameters, and decision parameters are dynamically adjusted through reinforcement learning, and decisions are executed: if |rule score - model score| ≥ decision parameter, the high score result is used as the final classification result, otherwise manual review is triggered. S5 includes:

[0073] Rule score = the rule confidence × (the priority base score + rule weight parameter) × scenario factor,

[0074] Model score = model confidence × (model reliability factor + model weight parameter) × scenario factor, where the model reliability factor is the average accuracy of the large language model in classifying text data within a preset period;

[0075] If the rule score ≥ model score + decision parameter, the rule classification result is used as the final classification result.

[0076] If the model score ≥ rule score + decision parameter, the model classification result is used as the final classification result.

[0077] If the absolute value of the difference between the rule score and the model score is less than the decision parameter, manual review is triggered.

[0078] Reinforcement learning algorithms include:

[0079] State space definition:

[0080] Construct an n-dimensional vector as the state space, integrating at least the rule confidence, model confidence, model reliability factor, scenario factor, scenario factor, and decision parameter, where n is a positive integer;

[0081] Action Space Design:

[0082] The action space includes threshold adjustment parameters and weight update parameters. The weight update parameters include rule weight update parameters and model weight update parameters. The action vector is constructed as [threshold adjustment parameters, rule weight update parameters, model weight update parameters].

[0083] Outputting the action vector based on the state space through the action space, adjusting the rule weight parameter, the model weight parameter and the decision parameter;

[0084] Reward function design:

[0085] The reward function includes accuracy reward and review rate reward:

[0086] Accuracy Reward: When the rule classification results and model classification results are consistent with the results of proactive manual review, a positive reward is given; when the rule classification results and model classification results are incorrect and no manual review is triggered, a negative reward is given; when manual review is triggered, no reward is given;

[0087] Review rate reward: the inverse of the ratio that triggers manual review;

[0088] Total reward = accuracy reward × first weight + review rate reward × second weight.

[0089] S6: When a manual review condition is met, triggering manual review, the manual review condition at least includes: the definition of the classification label is the same as the definition of the preset sensitive classification label.

[0090] S7: Use the attention mechanism visualization tool to visualize the attention weights of each layer in the classification process of the large language model.

[0091] S8: Establish a rule effect evaluation model through machine learning algorithms, extract rule structure, business attributes, and performance indicators as basic features, and construct rule complexity index and business relevance to achieve rule attribute vectorization.

[0092] Rule complexity index = number of keywords × (number of logical operators + 1) × (context constraint complexity + 1), used to quantify the logical judgment complexity of the rule;

[0093] Business relevance = sensitivity level × (1 + hit rate for small sample scenarios), which is used to measure the rule's coverage of key business scenarios.

[0094] S9: The system automatically screens samples through uncertainty sampling, diversity sampling, and domain knowledge guidance to form a training dataset, and regularly fine-tunes the large language model through model fine-tuning technology.

[0095] The following is a detailed description of each step.

[0096] Step S1: Build and dynamically maintain the financial text classification label system and classification rule template library

[0097] A tree or graph structure is used to store multi-level rule classification labels. This structure helps to clearly represent the hierarchical relationship and association between labels.

[0098] Tag attributes: Each tag contains the following attributes:

[0099] Tag Name: The name or identifier of the tag.

[0100] Level: The hierarchical position of the tag in the system.

[0101] Business scenario: The specific business scenario or field to which the tag applies.

[0102] Tag definition: Detailed definition and description of the tag.

[0103] Sensitivity level: The sensitivity of the label, used for subsequent permission management and security control.

[0104] Historical hit rate: The frequency with which a label has been hit in past text classifications, which can be used to evaluate the effectiveness and importance of the label.

[0105] Dynamic maintenance: Supports the addition, merging, deprecation and version management of tags, ensuring that the tag system can be dynamically adjusted as business needs change.

[0106] Design a set of flexible and configurable classification rule templates to guide the text classification process.

[0107] Rule Type:

[0108] Conditional: Direct matching based on a single keyword or phrase or simple conditions.

[0109] Combination: Combine multiple conditions through logical operators (such as AND, OR, NOT), or limit the positional relationship between keywords (such as distance, order).

[0110] Contextual: judge based on the contextual information of the keyword (such as previous and next entities, sentiment, and paragraph structure).

[0111] Rule configuration:

[0112] Rule confidence threshold: Configure a rule confidence threshold for each rule to evaluate the reliability of rule matching.

[0113] Rule priority: In subsequent steps, the priority strategy is used to achieve intelligent tuning of the rule matching order.

[0114] In addition, an integration between the labeling system and the rule template library is set up to ensure that each rule classification label is associated with a set of classification rules, which are used to determine whether the text belongs to the rule classification label.

[0115] Step S2: Support multi-source text input and preprocessing

[0116] The system needs to support receiving text data from a variety of data sources, including APIs, files, and databases, and be able to process unstructured data. Specific implementation methods may include the following:

[0117] API access:

[0118] Standardized interface: Provides a RESTful API interface and supports requests in JSON / XML format.

[0119] Content parsing: Ability to parse plain text content or file binary streams (such as uploaded scanned documents) contained in the request.

[0120] Real-time: Supports real-time reception and processing of text data, suitable for scenarios that require rapid response.

[0121] File upload:

[0122] Local file reading: supports users to directly upload local files.

[0123] Document type support:

[0124] PDF / Word and other documents: Use document parsing libraries (such as Apache PDFBox and Apache POI) to extract text content.

[0125] Image formats such as JPG / PNG or scanned documents: Triggers the OCR (Optical Character Recognition) text recognition process and calls an OCR engine (such as Tesseract or Baidu OCR) to convert the image into editable text.

[0126] Metadata Retention: The metadata of the original file (such as file name and page number) is retained during the conversion process for subsequent tracking and verification.

[0127] Database access:

[0128] Connection configuration: Establish a connection with the database by configuring database connection information (such as database type, host name, port, user name, password, etc.).

[0129] SQL query: supports executing SQL query statements and obtaining text data in batches.

[0130] Data formatting: Format the query results into a text format that can be processed within the system.

[0131] Step S2 also includes data preprocessing, specifically preprocessing the received text data to improve the accuracy and efficiency of subsequent classification.

[0132] The following implementations may be included:

[0133] OCR processing:

[0134] Page-by-page recognition: For non-text data such as images and scans, the OCR engine is called to recognize text page by page.

[0135] Text generation: Generate searchable text content and retain the metadata of the original file (such as file name, page number) for subsequent processing and analysis.

[0136] Noise Removal:

[0137] Regular expression filtering: Use regular expressions to filter out noise data such as HTML tags, special symbols, and garbled characters.

[0138] Space Merging: Merge consecutive spaces and standardize punctuation to make text neater and more standardized.

[0139] Custom rules: Supports users to customize filtering rules to meet the needs of specific business scenarios.

[0140] Text segmentation:

[0141] Segmentation strategy: Split text into semantic paragraphs, sentences, or fixed lengths to suit the needs of different classification tasks.

[0142] Overlap area configuration: Supports configurable overlap areas. Adjacent paragraphs retain overlapping areas to maintain contextual coherence, helping classification models better understand text content.

[0143] Step S3: Initial rule screening and priority tuning

[0144] The preprocessed text is input into the rule engine, matched according to the rules in the classification rule template library, and the successfully matched rules and their corresponding classification labels and confidence levels are recorded.

[0145] The following implementations may be included:

[0146] The rule engine receives text: The master node receives pre-processed text data and prepares for rule matching.

[0147] Rule matching:

[0148] Traversing the rule base: The rule engine traverses each rule in the classification rule template library.

[0149] Conditional judgment: For each rule, perform conditional judgment based on its type (conditional, combinational, contextual) to check whether the text meets the conditions of the rule.

[0150] Record matching results: If the text meets the conditions of a rule, record the rule and its corresponding classification label and confidence level.

[0151] Output matching results: Output the successfully matched rules, their classification labels, and confidence levels for use by subsequent hybrid decision modules.

[0152] In addition, this step also sets a priority tuning strategy, specifically setting a priority strategy to achieve intelligent tuning of the rule matching order and improve classification efficiency and accuracy.

[0153] The following implementations may be included:

[0154] Priority factor determination:

[0155] Business scenario: Assign different business scenario factor values ​​to the rules based on the importance and urgency of the business scenario.

[0156] Label sensitivity level: Assign different sensitivity level factor values ​​to rules based on the sensitivity of the label.

[0157] Time Window: Considers the effective time range of the rule and assigns a time window factor value to the rule (e.g., recent rules are given higher priority).

[0158] Data Source: Assign a data source factor value to the rule based on the reliability and importance of the data source.

[0159] Rule update status: Considers the update time and status of the rule and assigns a higher priority factor value to new rules or rules that are updated frequently.

[0160] Weight Matrix Configuration: Assign weights to each priority factor and construct a weight matrix. Weights can be adjusted and optimized based on actual business needs and historical data.

[0161] Priority score calculation: Rule priority score Score = ∑ (priority factor × priority weight) + priority base score, calculate the priority score of each rule.

[0162] Priority Base Score: The fixed priority set when a rule is created, reflecting the basic importance and applicability of the rule. Factor Value: Mapped to a specific value based on real-time trigger conditions, reflecting the applicability and importance of the rule in the current business scenario.

[0163] Rules are sorted based on the calculated priority scores, with rules with higher priority scores being matched first. During rule engine operation, the weight matrix and priority strategy are dynamically adjusted based on actual matching results and business needs, enabling intelligent tuning.

[0164] Step S4: Set the conditions for triggering large language model classification, such as text that does not trigger any rules after initial rule screening, text that triggers rules but has a confidence level lower than a set threshold, text that exceeds a set length, and labels for special business scenarios.

[0165] Large language models are used to classify qualified texts. Large models in the financial field, such as BloombergGPT and FinGPT, are preferred.

[0166] Large model classification uses a solution that combines vector retrieval with large models:

[0167] First, you can use models such as SimCSE (Simple Contrastive Learning of Sentence Embeddings, sentence encoding model) and BM25 (Best Match25, keyword matching algorithm) and build an index through FAISS (Facebook AI Similarity Search).

[0168] In the online phase, vector retrieval is first performed on the input text to recall similar samples and alternative labels;

[0169] The search results, label definitions, and text to be classified are combined into a specific prompt input model, and the output of the model is finally parsed to determine the category.

[0170] Among them, the assembled Prompt contains alternative labels, label definitions, text to be classified, output requirements and other parts, calling the large model to generate results.

[0171] Step S5: The hybrid decision module integrates the rules and the large model results to generate the final classification conclusion

[0172] The system uses a rule engine and a large model parallel processing mechanism:

[0173] The rule engine quickly matches the text based on the dynamic priority strategy in step S3;

[0174] The large model starts classification synchronously according to the triggering conditions of step S4, and the results of both are generated independently and then enter the hybrid decision-making process.

[0175] The specific decision logic is as follows:

[0176] 5.1 Rule-Priority Decision-Making Path

[0177] If the rule confidence is greater than or equal to the threshold and the large model does not output a valid result:

[0178] Directly adopt the rule engine results, the output includes:

[0179] Category label: the label corresponding to the trigger rule;

[0180] Classification basis: specific triggering rules;

[0181] Confidence: The match confidence calculated by the rule engine.

[0182] Logical basis: In high-frequency standardization scenarios, the deterministic matching results of the rule engine have priority and authority.

[0183] 5.2 Model-led decision path

[0184] When triggering S4's large language model classification based on conditions a and b:

[0185] Using the large model classification results, the output includes:

[0186] Classification label: the label predicted by the large model;

[0187] Classification basis: key features that the model's attention mechanism focuses on; confidence: the predicted probability value output by the large model.

[0188] Logical basis: In complex semantic scenarios, the contextual understanding capabilities of large models can make up for the limitations of rules.

[0189] 5.3 Dynamic Decision-making Strategy for Conflict Scenarios

[0190] Both the rule engine and the large model output valid results (when the large language model classification of S4 is triggered based on conditions c and d), forming parallel results with the rule engine (such as the rule matches the keyword but the results conflict after the model analyzes the context).

[0191] However, the labels are inconsistent. The scores are calculated through a multi-factor weighted model and reinforcement learning optimization is introduced:

[0192] (1) Score calculation model

[0193] Rule score = rule confidence × (rule priority factor + Δw_rule) × scenario factor

[0194] Model score = large model confidence × (model reliability factor + Δw_model) × scenario factor

[0195] The rule priority factor is the basic priority in step 3, the model reliability factor is the average accuracy of the model in the current scenario over the past 7 days, and Δw_rule and Δw_model are weight parameters.

[0196] The scenario factor is the static business attribute weight (preset), and the weight parameter is the dynamic optimization parameter (reinforcement learning output).

[0197] Scenario factors are preset according to different business scenarios. Possible settings are as follows:

[0198] Strong rule scenarios (such as compliance review): rule factor ≥ 1.0, model factor ≤ 1.0

[0199] Weak rule scenarios (such as financial sentiment analysis): rule factor ≤ 1.0, model factor ≥ 1.0

[0200] Balanced scenario (e.g., financial product marketing text classification): Rule factor = Model factor = 1.0

[0201] Decision logic

[0202] If the rule score ≥ model score + decision threshold δ, the rule result is adopted;

[0203] If the model score ≥ rule score + δ, the model result is adopted;

[0204] If the score difference is less than δ, manual review is triggered (step S6).

[0205] The decision threshold δ is an automatically adjusted optimization value, which is optimized together with Δw_rule and Δw_model through the reinforcement learning algorithm.

[0206] Specifically, the system automatically optimizes the decision threshold (δ) and weight parameters (Δw_rule, Δw_model) through deep reinforcement learning (DDPG) to maximize classification accuracy and minimize manual review rates, as follows:

[0207] (1) The state space is defined as a 9-dimensional vector, which integrates the real-time characteristics of the rules and models, decision parameters, scenario attributes and historical performance data to fully describe the decision context. In actual applications, it can be selected according to business requirements and actual conditions, as shown in the following table:

[0208]

[0209] (2) The action space is designed to be continuous and contains two types of optimizable parameters: threshold adjustment (Δδ) and weight update (Δw_rule, Δw_mode). The decision strategy is dynamically adjusted through the action vector A = [Δδ, Δw rule, Δw model].

[0210] Optimizable parameters:

[0211] Δδ: threshold adjustment step (range [-0.05, 0.05], step 0.01). For example, if the current δ = 0.1, it can be adjusted to 0.09 / 0.11;

[0212] Δw_rule: rule weight adjustment value (range [-0.1, 0.1]), used to dynamically increase / decrease the rule score;

[0213] Δw_model: model weight adjustment value (range [-0.1, 0.1]), used to dynamically scale up / down the model score;

[0214] Motion Vector:

[0215] The action space A = [Δδ, Δw_rule, Δw_model] is output by the reinforcement learning policy network to ensure that the parameters are adjusted within a safe range.

[0216] (3) The reward function adopts a multi-objective weighted mechanism, and the design principle is "minimizing the manual review rate and maximizing the classification accuracy":

[0217] Correct decisions directly contribute to positive rewards, while incorrect decisions are severely punished; high review rates increase labor costs, and negative rewards are used to suppress meaningless review triggers.

[0218] The reward calculation details can be set as follows:

[0219] Accuracy reward R_acc:

[0220] Correct decision (automatic decision result is consistent with manual review): +1 point;

[0221] Decision error (automatic decision error and no review is triggered): -0.5 points;

[0222] Triggering manual review (regardless of whether it is ultimately correct): 0 points (neutral, indirect optimization is required through review rate rewards);

[0223] Audit rate reward R_audit:

[0224] Calculate the proportion of manual audits triggered in the current batch, p_audit (e.g., if there are 15 audits in 100 classifications, p_audit = 0.15);

[0225] reward value = -p_audit;

[0226] Total reward R:

[0227] R = α × R_acc + β × R_audit, initially α = 0.6, β = 0.4, and dynamically adjusted through Pareto optimization to balance accuracy and audit efficiency.

[0228] (4) Real-time decision-making process

[0229] First, complete the state construction, integrate the rule engine, large model, and historical log data, and generate a 9-dimensional state vector S;

[0230] Secondly, the action is generated. The policy network outputs the action A = [Δδ, Δw_rule, Δw_model] according to S and updates the decision parameters. Finally, the score and decision are calculated, and the rule / model score and decision result are calculated according to the new parameters.

[0231] (5) Feedback closed-loop mechanism

[0232] After manual review is completed, (state S, action A, reward R, next state S') is stored in the experience buffer;

[0233] Training is performed regularly, sampling 512 data points from the buffer at regular intervals. The action value is evaluated through the value network (Critic), and the policy network (Actor) is updated using gradient descent. The goal is to maximize the cumulative discounted reward (the discount factor γ = 0.95 is an empirical value for balancing short-term and long-term rewards in reinforcement learning, ensuring that the algorithm focuses on cumulative benefits rather than single decisions).

[0234] Parameter validation: After each update, parameter rationality is checked to ensure that δ∈[0,0.5]. If it exceeds the range, it will automatically roll back to the valid value before the update. Second, weight parameters: rule weight (rule priority factor + Δw_rule) and model weight (model reliability factor + Δw_model) must both be ≥ 0. If negative weights occur, recalculate as "Δw = 0". A boundary validation mechanism ensures that decision parameters are within the business safety range, preventing extreme values ​​from causing classification logic failure.

[0235] The method of the present application also includes:

[0236] S6: When a manual review condition is met, triggering manual review, the manual review condition at least includes: the definition of the classification label is the same as the definition of the preset sensitive classification label.

[0237] Specifically, step S6 sets trigger conditions for manual review, including but not limited to the following: classification result confidence falling below a set threshold, conflicting classification results between the rule engine and the large model, and the involvement of sensitive classification labels. The review interface supports batch review, automatic recommendation of review opinions, and retrieval of historical review records. Audit results are stored in a structured manner, supporting various uses such as rule optimization, model fine-tuning, and label system adjustments. Labels that frequently trigger manual review enter the rule optimization or model fine-tuning process, forming a continuous closed loop.

[0238] S7: Use attention mechanism visualization tools to visualize the attention weights of each layer in the large language model during classification. Based on the attention weight matrix output by the large model, use TensorBoard and wandb (Weights & Biases, W&B) visualization tools to show the degree of attention each layer pays to the text segment.

[0239] Specifically, step S7 introduces an attention mechanism visualization tool to visualize the attention weights of each layer of the large model during the classification process, clearly presenting the text regions and key features that the model focuses on. During the rule engine's matching process, the matching path and context of each rule are recorded in detail, generating a rule matching log for user query. The visualization output module provides an interpretable report that comprehensively displays the decision-making basis of the rule engine and the large model, including the rule priority calculation process and the large model's reasoning logic chain. This enhances user confidence in the classification results and is particularly suitable for financial compliance review scenarios where interpretability is a high priority.

[0240] S8: Establish a rule effect evaluation model through machine learning algorithms, extract rule structure, business attributes, and performance indicators as basic features, and construct rule complexity index and business relevance to achieve rule attribute vectorization.

[0241] Rule complexity index = number of keywords × (number of logical operators + 1) × (context constraint complexity + 1), used to quantify the logical judgment complexity of the rule;

[0242] Business relevance = sensitivity level × (1 + hit rate for small sample scenarios), which is used to measure the rule's coverage of key business scenarios.

[0243] Specifically, step S8 uses a machine learning algorithm to establish a rule effectiveness evaluation model. In terms of feature engineering, rule structure (number of keywords, logical operator type, etc.), business attributes (label sensitivity level, historical hit rate, etc.), and performance indicators (accuracy, average matching time, etc.) are extracted as basic features. Derivative features such as rule complexity index and business relevance are constructed to achieve rule attribute vectorization.

[0244] Rule complexity index complexity = number of keywords × (number of logical operators + 1) × (context constraint complexity + 1) is used to quantify the logical judgment complexity of the rule. High-complexity rules may have matching efficiency issues.

[0245] Business Relevance: Relevance = Sensitivity Level × (1 + Hit Rate for Few-Sample Scenario) measures the rule's coverage of key business scenarios. A higher value indicates a more important rule.

[0246] The model is based on the CART (Classification and Regression Tree) algorithm, configuring core parameters and optimizing splitting criteria, such as prioritizing sensitive labels and protecting against low-sample scenarios. During training, a dataset containing historical rule matching logs and manually reviewed data is constructed, with sample balancing and 5-fold stratified cross-validation performed. Inefficient or outdated rules are automatically identified, and recommendations for rule optimization are generated.

[0247] S9: The system automatically screens samples through uncertainty sampling, diversity sampling and domain knowledge guidance to form a training data set, and regularly fine-tunes the large language model through LoRA fine-tuning technology. Use efficient fine-tuning technologies such as LoRA (Low-Rank Adaptation) to regularly fine-tune the large model. Step S9: The system automatically screens high-value samples, including uncertainty sampling (low confidence, conflicting samples), diversity sampling (outliers, few samples), and domain knowledge guidance (highly sensitive labels, etc.) to form a training data set. Use efficient fine-tuning technologies such as LoRA to regularly fine-tune the large model. After fine-tuning, the system automatically evaluates the model performance. If the effect deteriorates, it automatically rolls back to the historical optimal version to ensure business continuity and security.

[0248] In order to further illustrate the method of the present application, the following examples are provided:

[0249] Example: Financial Risk Identification in Listed Company Annual Reports

[0250] Step 1: Build a classification label system and rule base

[0251] 1.1 Build a label system. Some labels are as follows:

[0252] First-level labels: financial risk, operational risk, market risk, and compliance risk

[0253] Secondary label (taking financial risk as an example):

[0254] Liquidity Risk (Tag ID: FR-001, Sensitivity Level: High)

[0255] Solvency Risk (Label ID: FR-002, Sensitivity Level: Medium)

[0256] Revenue Recognition Risk (Tag ID: FR-003, Sensitivity Level: Low)

[0257] 1.2 Rule base configuration:

[0258] Conditional rule: The keyword "current ratio is less than 1.2" directly matches "liquidity risk" with a confidence threshold of 80%.

[0259] Combination rule: "Debt-to-asset ratio > 70%" and "Net cash flow from operating activities is negative" are combined to match "debt repayment ability risk", the logical operator is "and", the keyword distance is ≤ 50 words, and the confidence threshold is 75%.

[0260] Contextual rules: If "bad debt provision ratio adjustment" or "increased customer default rate" appears near "accounts receivable," the "revenue recognition risk" rule is matched, with a confidence threshold of 85%.

[0261] Step 2: Data access and preprocessing

[0262] 2.1 Data Input

[0263] Upload the annual report PDF file and trigger OCR processing (for scanned documents or images) through the data access module to extract the text content and retain the page number metadata (such as P.35 "Notes to the Financial Statements").

[0264] 2.2 Preprocessing

[0265] Noise Removal: Filter invalid characters such as headers and footers in PDFs and standardize punctuation.

[0266] Text segmentation: Split by semantic paragraphs, for example, split the "Management Discussion and Analysis" chapter into multiple paragraphs, with adjacent paragraphs overlapping by 50 words.

[0267] Step 3: Rule Engine Initial Screening

[0268] 3.1 Rule Matching

[0269] Calculate the rule priority score based on the dynamic priority policy and match the rule.

[0270] Scanning the text paragraphs, we found that the "Current Ratio 1.15" in P.42 "Consolidated Balance Sheet" triggered the conditional rule FR-001 with a confidence level of 85% and a record label of "Liquidity Risk".

[0271] Another paragraph, "asset-liability ratio 72%, net cash flow from operating activities -50 million yuan," triggers combined rule FR-002 with a confidence level of 80% and a record label of "debt repayment risk."

[0272] Step 4: Classify the large model

[0273] 4.1 Set the trigger condition as no direct keyword matched in the rule engine

[0274] For example, the paragraph below, "Management Discussion" on page 28 of the annual report, reads: "Despite revenue growth in the current period, overseas customer concentration has increased to 65%, and increased uncertainty in trade policies in some countries may affect future order stability." If the rule engine fails to match direct keywords, it triggers the large language model classification module, entering the vector search-driven intelligent tag screening process.

[0275] 4.2 Vector Retrieval and Alternative Label Screening

[0276] Label vectorization: Generate semantic vectors for all secondary labels in the label system in advance. Generate vectors based on the label definition text through the SimCSE model and store them in the FAISS vector index library.

[0277] Text vectorization: Generate query vectors for the target paragraph using the same SimCSE model.

[0278] Similarity retrieval: Perform cosine similarity retrieval through FAISS to recall historical samples and alternative labels. The results are as follows (sorted by similarity):

[0279] 1. Operational risk - market dependence risk (0.89)

[0280] 2. Market risk-policy compliance risk (0.85)

[0281] 3. Operational risk - customer concentration risk (0.83)

[0282] 4. Market Risk-Geopolitical Risk (0.81)

[0283] 5. Operational risk - income structure risk (0.79)

[0284] 4.3 Dynamic Prompt Construction and Model Reasoning

[0285] Assemble a prompt containing dynamic alternative labels:

[0286] Alternative tags (sorted by similarity):

[0287] 1. Operational risk - market dependence risk (Concept: business risk caused by abnormal customer concentration or changes in the market environment)

[0288] 2. Operational risk - customer concentration risk (Concept: the risk of business fluctuations caused by a single customer accounting for an excessively high proportion of revenue)

[0289] 3. Market Risk - Policy Compliance Risk (Concept: Compliance risk caused by regulatory policy changes or geopolitical factors) .....

[0291] Text to be classified: "Despite revenue growth this period... future orders remain stable."

[0292] Output requirements:

[0293] 1. Determine the best possible tag for the text (only in the list) and give a confidence level (0-100%)

[0294] 2. Extract up to 3 key features as the basis for classification

[0295] 3. If the matching degree is less than 70%, output "no matching label"

[0296] 4.4 Large model output:

[0297] Matching tags: Operational risk-Market dependence risk

[0298] Confidence level: 88%

[0299] Classification by:

[0300] ① Customer concentration increased to 65%

[0301] ②Increased trade policy uncertainty

[0302] ③ Risk scenarios involve both customer structure and market environment factors

[0303] Step 5: Dynamic Policy Module

[0304] Rule results: financial risk-liquidity risk, confidence level 85%, financial risk-solvency risk (confidence level 80%).

[0305] Model results: operating risk - market dependence risk (confidence 88%).

[0306] Decision logic:

[0307] When the rule engine matches a valid rule and the confidence level is greater than the preset threshold, it directly adopts the rule result and outputs "Financial Risk-Liquidity Risk" or "Financial Risk-Debt Solvency Risk" based on the triggered rule keywords.

[0308] Since there is no rule conflict in the model results, the “operating risk-market dependence risk” is output separately based on the key features extracted from the large model.

[0309] Hypothetical conflict scenario:

[0310] When the model output "Market Risk - Policy Compliance Risk (85% confidence)" conflicts with the rule result "Financial Risk - Solvency Risk (80% confidence)":

[0311] Dynamic score calculation:

[0312] The current business scenario is risk identification for listed companies' annual reports. The rule score and model score are calculated based on the preset scenario factors, rule priority basic score, current rule weight, model reliability factor, and model weight, resulting in a rule score of 70.4% and a model score of 76.5%.

[0313] Reinforcement Learning Decision-Making:

[0314] Currently, δ = 0.05 (optimized by the DDPG algorithm, corresponding to the threshold with the lowest manual review rate).

[0315] Score difference = 76.5% - 70.4% = 6.1% > δ × 100% = 5%, adopt the model result.

[0316] Trigger reward feedback: If manual review confirms that the model is correct, the reward function R = 1 (correct classification does not trigger review), and the model reliability factor in the state vector is updated at the same time.

[0317] Step 6: Manual Review

[0318] 6.1 Set trigger conditions: tag sensitivity level is high and confidence level is <60%.

[0319] 6.2 Triggering Scenario:

[0320] The classification results involve "liquidity risk" (highly sensitive label), which automatically triggers manual review.

[0321] The reviewer checked the original text, confirmed that it was true and there was no contradiction in the context, and passed the review.

[0322] Rule optimization and update: If a rule is found to be frequently triggered by mistake during the audit, adjust the rule configuration.

[0323] Step 7: Visualize the Output

[0324] The following table shows an example of structured results:

[0325] Structured results:

[0326] Risk Type Classification basis Confidence Original location Liquidity risk Current ratio 1.15 (triggering rule FR-001) 85％ P.42 Solvency risk Debt-to-asset ratio 72% + cash flow -50 million yuan 80％ P.42 Market dependence risk Customer concentration 65% + trade policy uncertainty 88％ P.28

[0327] Original text highlighting: Automatically highlight key fragments such as "current ratio 1.15", "asset-liability ratio 72%", "customer concentration increased to 65%" in the annual report PDF.

[0328] Step 8: Rule evaluation and optimization

[0329] Quantitative evaluation of rule effectiveness:

[0330] Rule complexity index: FR-002 (combination rule) = number of keywords 2 × (logical operator 1 + 1) × (context constraint 0 + 1) = 4;

[0331] Business relevance: FR-002 = Sensitivity level 3 × (1 + 8% hit rate for small sample scenarios) = 3.24; CART model decision: When complexity > 3 and business relevance < 3.5, mark as "to be optimized."

[0332] Step 9: Model fine-tuning

[0333] 9.1 Sample Screening and Labeling

[0334] The sample selection strategy is to select samples with a large model confidence of less than 60% and samples that trigger the rule less than 50 times for labeling.

[0335] 9.2 Model Update

[0336] Model update: Use LoRA to fine-tune FinGPT and input labeled data to improve accuracy in specific scenarios.

[0337] The present application also provides a text classification system using the above-mentioned text classification method based on a large model and a rule engine, the text classification system comprising:

[0338] The input and output layer includes a configuration management module, a data access module, a data processing module, and a visualization output module. The configuration management module is used to execute step S1, the data access module and the data processing module are used to execute step S2, and the visualization output module is used to execute step S7.

[0339] The model processing layer includes a rule engine module, a large language model classification module and a hybrid decision module. The rule engine module is used to execute step S3, the large language model classification module is used to execute step S4, and the hybrid decision module is used to execute step S5.

[0340] The toolchain layer includes a manual review module, a data annotation module, a model fine-tuning module, and a rule evaluation module. The manual review module is used to execute step S6, the data annotation module and the model fine-tuning module are used to execute step S9, and the rule evaluation module is used to execute step S8. Details are shown in the following table.

[0341]

Claims

1. A text classification method based on a large model and a rule engine, characterized in that: include: S1: Store multi-level rule classification labels, build a classification rule template library, associate rule classification labels with classification rules in the classification rule template library, and set a corresponding rule confidence threshold for each classification rule; S2: Receive text data from various data sources and preprocess the text data; S3: The rule engine matches the text data based on the classification rules and outputs the rule classification results. The rule classification results include the text data and the corresponding classification rules, the rule classification labels, and the rule confidence level of the matching between the text data and the classification rules. S4: Large language model classification is triggered when any of the following conditions are met: a. No classification rules are matched; b. The classification rule is matched and the rule confidence is less than the rule confidence threshold; c. The length of the text data exceeds the preset text data length; d. Match to specific business scenario tags; Input the model classification label, label definition, and text to be classified into the large language model, and output the model classification results, including text data and corresponding model classification labels, model confidence, and classification basis; S5: When the rule engine classification of S3 and the large language model classification of S4 are carried out in parallel, the following strategy is implemented: rule score = rule confidence × (priority base score + rule weight parameter) × scenario factor, model score = model confidence × (model reliability factor + model weight parameter) × scenario factor, where the model reliability factor is the average accuracy of the large language model in classifying text data within a preset period; the rule weight parameter, model weight parameter and decision parameter are dynamically adjusted through reinforcement learning, and the decision is executed: if |rule score - model score| ≥ decision parameter, the high score result is used as the final classification result, otherwise manual review is triggered.

2. The text classification method based on a large model and a rule engine according to claim 1 is characterized in that: The S3 includes: setting a corresponding priority factor for each classification rule, and calculating a priority score based on the priority factor, where the priority score = ∑ (priority factor × priority weight) + priority base score.

3. The text classification method based on a large model and a rule engine according to claim 1, characterized in that: The S4 includes: Build text vectors and create text vector indexes; Vectorize the input text data and retrieve similar samples and model classification labels based on the text vector index; The text data to be classified, the retrieved similar samples and the model classification labels are input into the large language model, and the text data, the model classification labels, the model confidence and the classification basis are output.

4. The text classification method based on a large model and a rule engine according to claim 1, characterized in that: The S5 includes: When the rule engine classification of S3 is triggered and the large language model classification of S4 is not triggered, the rule classification result is used as the final classification result. When the large language model classification of S4 is triggered based on conditions a and b, the model classification result is used as the final classification result. When the large language model classification of S4 is triggered based on conditions c and d, the strategy of step S5 is executed.

5. The text classification method based on a large model and a rule engine according to claim 1 is characterized in that: The S5 includes: the reinforcement learning algorithm includes: state space definition: Construct an n-dimensional vector as the state space, integrating at least the rule confidence, model confidence, model reliability factor, scenario factor, scenario factor, and decision parameter, where n is a positive integer; Action Space Design: The action space includes threshold adjustment parameters and weight update parameters. The weight update parameters include rule weight update parameters and model weight update parameters. The action vector is constructed as [threshold adjustment parameters, rule weight update parameters, model weight update parameters]. Outputting the action vector based on the state space through the action space, adjusting the rule weight parameter, the model weight parameter and the decision parameter; Reward function design: The reward function includes accuracy reward and review rate reward: Accuracy Reward: When the rule classification results and model classification results are consistent with the results of proactive manual review, a positive reward is given; when the rule classification results and model classification results are incorrect and no manual review is triggered, a negative reward is given; when manual review is triggered, no reward is given; Review rate reward: the inverse of the ratio that triggers manual review; Total reward = accuracy reward × first weight + review rate reward × second weight.

6. The text classification method based on a large model and a rule engine according to claim 1, characterized in that: Also includes S6: When a manual review condition is met, a manual review is triggered, and the manual review condition at least includes: the definition of the classification label is the same as the definition of the preset sensitive classification label.

7. The text classification method based on a large model and a rule engine according to claim 1, characterized in that: Also includes S7: The attention mechanism visualization tool is used to visualize the attention weights of each layer in the classification process of a large language model.

8. The text classification method based on a large model and a rule engine according to claim 1, characterized in that: Also includes S8: A rule effect evaluation model is established through machine learning algorithms, rule structure, business attributes, and performance indicators are extracted as basic features, and rule complexity index and business relevance are constructed to achieve rule attribute vectorization. Rule complexity index = number of keywords × (number of logical operators + 1) × (context constraint complexity + 1), used to quantify the logical judgment complexity of the rule; Business relevance = sensitivity level × (1 + hit rate for small sample scenarios), which is used to measure the rule's coverage of key business scenarios.

9. The text classification method based on a large model and a rule engine according to claim 1, characterized in that: Also includes S9: The system automatically screens samples through uncertainty sampling, diversity sampling, and domain knowledge guidance to form a training data set, and regularly fine-tunes the large language model through model fine-tuning technology.

10. A text classification system using the text classification method based on a large model and a rule engine according to any one of claims 1 to 9, characterized in that: The text classification system includes: The input and output layer includes a configuration management module, a data access module, a data processing module, and a visual output module. The configuration management module is used to execute step S1, the data access module and the data processing module are used to execute step S2, and the visual output module is used to execute step S7. The model processing layer includes a rule engine module, a large language model classification module, and a hybrid decision module. The rule engine module is used to execute step S3, the large language model classification module is used to execute step S4, and the hybrid decision module is used to execute step S5. The tool chain layer includes a manual review module, a data annotation module, a model fine-tuning module and a rule evaluation module. The manual review module is used to execute step S6, the data annotation module and the model fine-tuning module are used to execute step S9, and the rule evaluation module is used to execute step S8.

Images