Big language model fused reasoning enhanced text classification method and system
By generating high-quality chained inference data and combining lightweight LoRA fine-tuning and majority voting mechanisms, the problem of insufficient multi-step logical reasoning in large language models in text classification tasks is solved, and the stability and accuracy of the model in complex scenarios are improved.
Patent Information
- Application Number
- CN202511454976.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-13
- Publication Date
- 2026-05-08
AI Technical Summary
Existing large language models lack multi-step logical reasoning capabilities in text classification tasks, resulting in unstable performance in complex scenarios. Existing augmented reasoning methods also suffer from unstable reasoning chain quality and lack targeted fine-tuning in text classification.
By generating high-quality chained inference data, combined with lightweight LoRA fine-tuning and majority voting mechanisms, the model's multi-step inference ability and classification stability are improved.
It significantly improves the performance and stability of the model in complex semantic classification tasks, and enhances classification accuracy and consistency.
Smart Images
Figure CN121996788A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of text classification technology, specifically to a reasoning-enhanced text classification method and system that integrates a large language model. Background Technology
[0002] Text classification is one of the core tasks in natural language processing, aiming to assign input text to predefined categories based on its content. It is widely used in scenarios such as news aggregation, public opinion analysis, medical literature retrieval, and legal document classification. With the development of deep learning and pre-training techniques, methods have gradually evolved from shallow models such as CNN and LSTM to pre-trained language models based on Transformer (such as BERT and RoBERTa), and then to large language models (LLM) represented by GPT and LLaMA.
[0003] While LLMs excel in language understanding and generation, they often employ a "direct mapping" paradigm in text classification tasks, which involves a one-step prediction from input to label. This approach relies on the model's implicit semantic representation and is prone to skipping necessary inference steps in complex scenarios, leading to unstable performance.
[0004] In tasks requiring multi-step logical or causal reasoning, models need to establish logical relationships across multiple semantic segments. For example, when a film review contains both positive and negative comments, relying solely on keyword matching may misjudge the overall sentiment; multiple paragraphs may have topic shifts, requiring cross-segment information integration to make accurate judgments; different pathological states under the same disease require the integration of multiple pieces of evidence for differentiation.
[0005] Existing augmented reasoning methods are mainly Chain-of-Thought (CoT), which guides the model to generate reasoning chains by prompting it to "think step by step," and are effective in tasks such as mathematical reasoning. However, when directly applied to text classification, the quality of the reasoning chains is unstable and lacks targeted fine-tuning. Summary of the Invention
[0006] The purpose of this invention is to provide a reasoning-enhanced text classification method and system that integrates a large language model to solve the problems mentioned in the background art.
[0007] To achieve the above objectives, the present invention provides the following technical solution: A reasoning-enhanced text classification method incorporating a large language model, the method comprising: Using a large language model, chained reasoning data containing a step-by-step reasoning process from input text to classification labels is generated for training samples of a text classification task. Based on the chained inference data, a lightweight fine-tuning method based on LoRA is used to fine-tune the target large language model to obtain a text classification model. For the text to be classified, multiple inference paths are generated using the fine-tuned text classification model, and the final classification label is obtained through a majority voting mechanism based on the classification results output by each inference path.
[0008] As a further aspect of the present invention, the step of generating chained reasoning data containing a step-by-step reasoning process from input text to classification labels for training samples of a text classification task using a large language model specifically includes: In response to receiving a training sample and its correct label, the generation parameters of the large language model are adjusted to generate multiple diverse chain-like inference paths for a single training sample. The generated chain reasoning path is subjected to quality screening, which includes logical consistency testing and / or label correctness testing; The quality-filtered chained inference paths are converted into a structured training data format, generating chained inference data that contains a step-by-step inference process from input text to classification labels.
[0009] As a further embodiment of the present invention, the generation parameters include the sampling temperature.
[0010] As a further embodiment of the present invention, the structured training data format is Alpaca format.
[0011] As a further aspect of the present invention, the step of generating multiple inference paths using the fine-tuned text classification model for the text to be classified, and obtaining the final classification label based on the classification results output by each inference path through a majority voting mechanism, specifically includes: Using a preset chain-like reasoning prompt template, the text classification model is guided to generate K reasoning paths for the text to be classified, where K is an integer greater than 1; Analyze each reasoning path and extract its corresponding category label; Count the frequency of each category label and use the label with the highest frequency as the final category label.
[0012] As a further embodiment of the present invention, the K inference paths are obtained by adjusting the generation parameters of the text classification model.
[0013] This invention also provides a reasoning-enhanced text classification system that integrates large language models, used to implement the aforementioned reasoning-enhanced text classification method that integrates large language models. The system includes: The dataset building module is used to generate chained reasoning data containing the step-by-step reasoning process from input text to classification labels for training samples of text classification tasks using large language models. The fine-tuning module is used to fine-tune the target large language model based on the chained inference data using a lightweight LoRA-based fine-tuning method to obtain a text classification model. The classification module is used to generate multiple inference paths for the text to be classified using the fine-tuned text classification model, and obtain the final classification label based on the classification results output by each inference path through a majority voting mechanism.
[0014] Compared with the prior art, the beneficial effects of the present invention are: by generating high-quality inference chains, lightweight fine-tuning based on inference chains, and multi-path consistency majority voting, the multi-step inference ability and classification stability of the model are effectively improved, overcoming the technical bottlenecks of inference blind spots and result inconsistency in the prior art. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.
[0016] Figure 1 An example diagram of the algorithm process provided in the embodiments of the present invention.
[0017] Figure 2 The diagram shows an experimental diagram of hyperparameter analysis provided in an embodiment of the present invention. Detailed Implementation
[0018] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.
[0019] Figure 1 This is an example diagram illustrating the algorithm process of a reasoning-enhanced text classification method that integrates a large language model. In this embodiment of the invention, a reasoning-enhanced text classification method integrating a large language model includes: Using a large language model, chained reasoning data containing a step-by-step reasoning process from input text to classification labels is generated for training samples of a text classification task. Based on the chained inference data, a lightweight fine-tuning method based on LoRA is used to fine-tune the target large language model to obtain a text classification model. For the text to be classified, multiple inference paths are generated using the fine-tuned text classification model, and the final classification label is obtained through a majority voting mechanism based on the classification results output by each inference path.
[0020] In this embodiment, the Large Language Model (LLM) demonstrates powerful capabilities in natural language understanding and generation, enabling it to complete various downstream tasks without requiring a large amount of task-specific labeled data. Especially in text classification tasks with complex semantics and multi-layered nested logical relationships, traditional keyword-based or shallow semantic matching methods struggle to accurately capture the inference chains within the text. LLM, however, introduces an explicit inference chain generation mechanism, allowing for step-by-step analysis of text content, extraction of key information, and deduction of the final classification conclusion. This invention significantly improves the model's performance and stability in complex semantic classification tasks by generating high-quality chained inference data, fine-tuning the model using inference chains, and employing multi-path consensus majority voting during the inference phase.
[0021] As a preferred embodiment of the present invention, the step of generating chained reasoning data containing a step-by-step reasoning process from input text to classification labels for training samples of a text classification task using a large language model specifically includes: In response to receiving a training sample and its correct label, the generation parameters of the large language model are adjusted to generate multiple diverse chain-like inference paths for a single training sample. The generated chain reasoning path is subjected to quality screening, which includes logical consistency testing and / or label correctness testing; The quality-filtered chained inference paths are converted into a structured training data format, generating chained inference data that contains a step-by-step inference process from input text to classification labels.
[0022] In this embodiment, for each training sample, an LLM with strong reasoning capabilities (such as DeepSeek-R1) is used to generate a step-by-step inference chain under the condition that the correct label is known, and a diverse set of inference paths is obtained by changing the sampling temperature and other methods.
[0023] like Figure 1 As shown, given each training sample and its known correct label, a step-by-step inference chain is generated using LLM. The inference chain is generated as follows: Step-by-step analysis of text content: LLM breaks down the input text into reasoning steps, with each step extracting key information from the text and establishing connections between different parts.
[0024] Example of inference chain: "Although the SD500 produces high-quality photos, the sensitivity of the LCD screen affects the overall quality." Step 1: Identify positive statements in the text: "great quality photos".
[0025] Step 2: Identify words that introduce contrast: "Although" indicates that the following content may be negative.
[0026] Step 3: Identify negative impacts: "compromises the quality overall".
[0027] Step 4: Draw the conclusion: "Criticism has a greater impact than praise, leading to negative emotions."
[0028] The generated inference chain data is cleaned to remove samples with incoherent logic, incomplete reasoning, or incorrect labels, and the retained data is converted into a unified Alpaca format for easy downstream fine-tuning.
[0029] Logical consistency check: Each generated reasoning chain is reviewed, and chains with incoherent logic or incomplete reasoning steps are removed.
[0030] Label accuracy check: Check whether each inference chain correctly derives a classification result consistent with the original label, and remove inference chains with incorrect labels.
[0031] Standardized format: All inference chain data that passes the screening will be converted into a standardized Alpaca format.
[0032] The Alpaca format is a concise and standardized data format that includes: input text; a step-by-step inference chain (each inference step); category labels; and the logical relationships and inference paths of the generated inference chain.
[0033] The LoRA lightweight parameter update method is used on the target LLM, with high-quality inference chain data as training samples, which enables the model to internalize multi-step inference patterns while maintaining low training cost. The LoRA method is used to fine-tune the target LLM using the generated high-quality inference chain data.
[0034] The advantages of LoRA are: it uses lightweight parameter updates, only adjusting the low-rank parameters of the model, without requiring large-scale retraining of the entire model; it reduces training overhead and computational burden while maintaining the multi-step learning capability of the inference chain; and the fine-tuning process helps the model understand multi-step inference patterns through inference chain data, thereby improving classification performance.
[0035] In a preferred embodiment of the present invention, the step of generating multiple inference paths using the fine-tuned text classification model for the text to be classified, and obtaining the final classification label based on the classification results output by each inference path through a majority voting mechanism, specifically includes: Using a preset chain-like reasoning prompt template, the text classification model is guided to generate K reasoning paths for the text to be classified, where K is an integer greater than 1; Analyze each reasoning path and extract its corresponding category label; Count the frequency of each category label and use the label with the highest frequency as the final category label.
[0036] The K inference paths are obtained by adjusting the generation parameters of the text classification model.
[0037] In this embodiment, during the prediction phase, multiple inference chains are generated for the test text using a specific prompt template (such as "Let's classify step by step"), and the classification conclusions of each chain are subject to majority voting. Finally, the label that appears most frequently is output as the prediction result.
[0038] During the prediction phase, a specific prompt template (such as "Let's classify step by step") is used to generate an inference chain for the test text.
[0039] Each inference chain analyzes the text step by step during the classification process and provides intermediate inference results and final classification labels.
[0040] Majority voting mechanism: The classification results of all generated inference chains are voted on; the classification result of each inference chain is used as a vote, and the label with the most occurrences is finally output as the prediction result. By combining the conclusions of multiple inference paths, this method effectively enhances the stability of the classification results, especially in text classification tasks with complex semantics and multiple nesting.
[0041] In the training data preparation phase, this invention utilizes a large language model with strong reasoning capabilities (such as DeepSeek-R1) to generate multiple chained inference paths for each training sample. Each path includes a step-by-step reasoning process from the input text to the predicted label, and the diversity of the inference paths is increased by adjusting methods such as sampling temperature. Subsequently, the generated inference path data undergoes quality screening and formatting, eliminating incomplete, logically inconsistent, or incorrectly labeled inference paths, and converting them into a unified training format (such as Alpaca format) to ensure data structure and usability. Next, in the model training phase, a lightweight fine-tuning method based on LoRA is employed to inject inference path knowledge into some weights of the target large language model, enabling the model to internalize multi-step reasoning patterns, thereby enhancing reasoning capabilities without significantly increasing computational costs. Finally, in the inference and prediction phase, for the input test text, a specific "Let's classify step by step" prompt is used to guide the model to generate multiple inference paths, and the classification conclusions of these paths are subject to majority voting to obtain the final predicted label.
[0042] This invention also provides a reasoning-enhanced text classification system that integrates large language models, used to implement the aforementioned reasoning-enhanced text classification method integrating large language models. The system includes: The dataset building module is used to generate chained reasoning data containing the step-by-step reasoning process from input text to classification labels for training samples of text classification tasks using large language models. The fine-tuning module is used to fine-tune the target large language model based on the chained inference data using a lightweight LoRA-based fine-tuning method to obtain a text classification model. The classification module is used to generate multiple inference paths for the text to be classified using the fine-tuned text classification model, and obtain the final classification label based on the classification results output by each inference path through a majority voting mechanism.
[0043] To verify the effectiveness of our method, we conducted comparative experiments on six publicly available text classification datasets with different domains and text length features: Biomedical, CR, DBLP, Ohsumed, R52, and TREC. We compared RICO with several representative methods, selecting two categories of algorithms for comparison: one is classic pre-trained language models (such as fine-tuned BERT and RoBERTa), and the other is large language models with zero or few samples (such as Gemma2-2B, LLaMA3.1-8B, and larger-scale models).
[0044] The Biomedical and Ohsumed datasets, derived from MEDLINE Medical Abstracts, are suitable for evaluating the classification capabilities of domain-specific corpora. The DBLP dataset contains titles and abstracts of computer science papers, used for academic topic classification. The CR dataset is a binary sentiment analysis task for product reviews. The TREC dataset consists of six categories of factual questions, used for question type identification. R52 is a subset of the Reuters-21578 dataset, containing long news texts across 52 topic categories. The datasets exhibit significant differences in the number of categories, text length distribution, and domain characteristics, posing multifaceted challenges to the generalization ability and robustness of the classification models. Experimental results show that RICO significantly outperforms the comparison methods on all datasets, achieving an ACC improvement of over 50 percentage points on the Biomedical dataset and maintaining a stable advantage on other datasets.
[0045] Table 1. Dataset Statistics
[0046] In the text classification task, we use accuracy (ACC) and macro-average F1 score (F1) as evaluation metrics. ACC measures the proportion of predicted labels that match the true labels, while F1 combines precision and recall and performs macro-averaging across classes to mitigate the impact of class imbalance on the results. For the experimental setup, we use the DeepSeek-R1 API to generate high-quality chained inference data and perform LoRA fine-tuning on two pedestal models: LLaMA-3.1-8B and Gemma2-2B. The training batch size is 32 GPUs, the gradient accumulation step is 4, and the learning rate is set to 5×10⁻⁻⁶. 5 The system employs a cosine learning rate scheduler, with 3 training epochs and a computational precision of FP16, and is completed entirely on 8 NVIDIA A800 80G GPUs. During the inference phase, the maximum token length for direct task generation is 256 tokens, while the maximum length for chained inference mode is 1024 tokens, with an inference batch size of 512.
[0047] As shown in Tables 2 and 3, in the main experiments, the RICO framework significantly outperformed various baseline methods on all six public datasets. For example, on the Biomedical dataset, Gemma2-2B-RICO achieved an ACC of 68.74% and an F1 score of 67.36%, while the original Gemma2-2B only achieved 11.83% and 10.77%, respectively. On the DBLP dataset, LLaMA3.1-8B-RICO achieved 84.58% and 83.37%, respectively, surpassing the larger-scale LLaMA3.1-70B zero-shot model. In the ablation experiments, by comparing different training and inference strategies, it was found that simple LoRA fine-tuning provided a certain improvement compared to the baseline without training, while CoT fine-tuning with the inference chain brought significant performance gains. Furthermore, after adding a majority voting mechanism in the inference stage, it achieved the best performance on almost all datasets. This indicates that the combination of inference chain training and consensus voting is a key factor in improving the accuracy and stability of complex text classification.
[0048] Table 2 Main Experiment Results
[0049] Table 3 Ablation Experiment Results
[0050] In summary, the experiments fully demonstrate that the RICO method proposed in this invention can effectively overcome the reasoning blind spots of large language models in text classification tasks. Through high-quality inference chain training and multi-path consistency majority voting mechanism, it significantly improves the accuracy, stability and interpretability of the model in multi-domain, long text and fine-grained classification tasks.
[0051] Example: This embodiment demonstrates the effectiveness of the RICO algorithm in classifying long texts in the professional medical field by performing a medical literature classification task on the Biomedical dataset.
[0052] Using the LLaMA3.1-8B model as the basic model, combined with the RICO framework proposed in this invention, it includes four core modules: high-quality inference chain generation, inference chain screening and formatting, LoRA-based inference-aware fine-tuning, and multi-path consensus majority voting.
[0053] First, the DeepSeek-R1 model is used to generate multiple diverse inference chains for each training sample in the Biomedical dataset under known labels, and temperature sampling is used to increase diversity. Then, the inference chains are quality-screened, removing samples with inconsistent logic, incomplete reasoning, or incorrect labels, and converted to a unified Alpaca format. Subsequently, the LLaMA3.1-8B model is fine-tuned using LoRA with high-quality inference chain data as the training set to internalize the multi-step inference pattern. Finally, in the inference stage, ten inference paths are generated for the test samples using specific chain-like classification hints, and the final prediction result is obtained by majority voting.
[0054] Significant improvements were achieved in both ACC and F1 scores, with ACC increasing from 17.06% to 73.09% and F1 from 17.82% to 73.03%. These improvements significantly outperformed control methods without inference chain fine-tuning and majority voting, fully validating the effectiveness and robustness of the RICO framework in medical text classification.
[0055] This embodiment verifies the impact of the number of inference chains in majority voting on model performance, in order to evaluate the stability and optimal value range of the algorithm in terms of consensus integration.
[0056] Composition: Based on the Gemma2-2B model, the same RICO framework as the above embodiment is used, but the number of inference chains K for majority voting is changed during the inference phase for experimentation.
[0057] Implementation process: On six public text classification datasets, different values of K were set, such as 1, 3, 5, 7, 10, and 15. Under the condition of keeping the training configuration, model parameters and data processing flow unchanged, the effects of different values of K on ACC and F1 were recorded.
[0058] Effects: As shown in the figure, experimental results indicate that as K increases from 1 to 10, the model performance steadily improves, reaching its optimal performance around K=10 on most datasets. When K further increases to 15, the performance improvement tends to saturate, indicating that after the diversity of inference chain generation reaches a certain level, the benefit of increasing the number of voting chains is limited. This result demonstrates that the present invention has good stability in consensus ensembles and that a balance between performance and computational cost can be achieved by reasonably selecting the value of K.
[0059] like Figure 2 As shown, the method of this invention has been validated on several publicly available text classification datasets, including Biomedical, CR, DBLP, Ohsumed, R52, and TREC. The results show that it significantly outperforms the comparison methods in both accuracy (ACC) and macro-average F1 (F1). Taking the Biomedical dataset as an example, the LLaMA3.1-8B model without this method had an ACC of only 17.06% and an F1 of 17.82%, while after RICO optimization, the ACC improved to 73.09% and the F1 improved to 73.03%. Furthermore, this method demonstrates good cross-domain generalization ability in long texts, fine-grained categories, and domain-specific tasks. Small-scale models under the RICO framework even outperform larger-scale original large language models. Simultaneously, due to the introduction of an explicit inference chain, the model's classification process has high interpretability, facilitating user review and understanding of the prediction results, thereby enhancing the system's credibility and practicality.
[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A reasoning-enhanced text classification method integrating a large language model, characterized in that, The method includes: Using a large language model, chained reasoning data containing a step-by-step reasoning process from input text to classification labels is generated for training samples of a text classification task. Based on the chained inference data, a lightweight fine-tuning method based on LoRA is used to fine-tune the target large language model to obtain a text classification model. For the text to be classified, multiple inference paths are generated using the fine-tuned text classification model, and the final classification label is obtained through a majority voting mechanism based on the classification results output by each inference path.
2. The reasoning-enhanced text classification method integrating a large language model according to claim 1, characterized in that, The step of generating chained reasoning data containing a step-by-step reasoning process from input text to classification labels using a large language model for training samples of a text classification task specifically includes: In response to receiving a training sample and its correct label, the generation parameters of the large language model are adjusted to generate multiple diverse chain-like inference paths for a single training sample. The generated chain reasoning path is subjected to quality screening, which includes logical consistency testing and / or label correctness testing; The quality-filtered chained inference paths are converted into a structured training data format, generating chained inference data that contains a step-by-step inference process from input text to classification labels.
3. The reasoning-enhanced text classification method integrating a large language model according to claim 2, characterized in that, The generated parameters include the sampling temperature.
4. The reasoning-enhanced text classification method integrating a large language model according to claim 2, characterized in that, The structured training data is in Alpaca format.
5. The reasoning-enhanced text classification method integrating a large language model according to claim 1, characterized in that, The steps of generating multiple inference paths using the fine-tuned text classification model for the text to be classified, and obtaining the final classification label based on the classification results output by each inference path through a majority voting mechanism, specifically include: Using a preset chain-like reasoning prompt template, the text classification model is guided to generate K reasoning paths for the text to be classified, where K is an integer greater than 1; Analyze each reasoning path and extract its corresponding category label; Count the frequency of each category label and use the label with the highest frequency as the final category label.
6. The reasoning-enhanced text classification method integrating a large language model according to claim 5, characterized in that, The K inference paths are obtained by adjusting the generation parameters of the text classification model.
7. A reasoning-enhanced text classification system integrating a large language model, used to implement the reasoning-enhanced text classification method integrating a large language model as described in any one of claims 1-6, characterized in that, The system includes: The dataset building module is used to generate chained reasoning data containing the step-by-step reasoning process from input text to classification labels for training samples of text classification tasks using large language models. The fine-tuning module is used to fine-tune the target large language model based on the chained inference data using a lightweight LoRA-based fine-tuning method to obtain a text classification model. The classification module is used to generate multiple inference paths for the text to be classified using the fine-tuned text classification model, and obtain the final classification label based on the classification results output by each inference path through a majority voting mechanism.