Text classification model training method, system and equipment based on reinforcement learning

Abstract

The invention provides a text classification model training method, system and equipment based on reinforcement learning, and the method comprises the steps: constructing a training data set passing initial verification, and generating a response text containing a reasoning process and a classification result based on a pre-training language model for a to-be-classified text in the training data set; acquiring a historical reasoning process corresponding to the classification result in the dynamic historical cache region, calculating the similarity between the reasoning process and the historical reasoning process, judging whether the similarity falls into a preset similarity interval or not, and determining whether the reasoning process meets diversity requirements or not; determining a target reward value based on the correctness of the target classification result and whether the reasoning process meets diversity requirements; and updating parameters of the pre-training language model through a reinforcement learning algorithm by using the target reward value. According to the invention, the training data is generated based on self-consistent verification, so that the marking cost is reduced; and in combination with diversified rewards, hacker reward behaviors are inhibited, so that the real reasoning ability and classification performance of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text classification model training method and system based on reinforcement learning and a device. BACKGROUND

[0002] In modern service industries such as online car-hailing, a large amount of user feedback and complaint texts are generated daily. In order to improve operational efficiency and protect user rights, these texts need to be quickly and accurately automatically classified. Existing text classification model training methods, such as supervised fine-tuning methods, usually rely heavily on large-scale and high-quality human-labeled data, which results in high data preparation costs, long model iteration cycles, and poor scalability when new classification labels are introduced due to changes in business needs, which often requires large-scale relabeling.

[0003] In order to reduce the dependence on labeled data and improve the model inference ability, the industry has begun to use reinforcement learning methods to fine-tune the model. However, when reinforcement learning is applied to text classification tasks, especially in scenarios where the model needs to generate inference processes (such as thought chains), a behavior known as "reward hacking" may occur. That is, the model may "gouge" in order to obtain rewards, for example, it may find that a certain fixed and simple inference pattern always gets good reward scores, and it tends to generate this repetitive and low-quality inference process in various situations, rather than conducting real and effective logical thinking. Although this behavior may get high scores on some simple samples, it undermines the model's real logical inference ability, resulting in poor generalization ability and poor performance when facing complex or new complaints, ultimately affecting the accuracy and reliability of the overall classification. SUMMARY

[0004] To this end, the present application provides a text classification model training method and system based on reinforcement learning and a device, aiming to solve the technical problems of high training data acquisition cost in the prior art and the occurrence of reward hacking behavior in the reinforcement learning training process, resulting in poor diversity of model inference processes and weak generalization ability.

[0005] To achieve the above purpose, the present application adopts the following technical solutions: According to the first aspect of the present application, the present application provides a text classification model training method based on reinforcement learning, comprising: constructing a training data set verified by an initial verification, and for the text to be classified in the training data set, generating at least one response text containing an inference process and a classification result based on a pre-trained language model; obtaining a historical reasoning process corresponding to a target classification result of the text to be classified in a dynamic history buffer, calculating a similarity between a reasoning process in the response text and the historical reasoning process, and determining whether the reasoning process meets a diversity requirement by judging whether the similarity falls within a preset similarity interval; determining a target reward value for updating the pre-trained language model based on the correctness of the target classification result and whether the reasoning process meets the diversity requirement; updating parameters of the pre-trained language model by a reinforcement learning algorithm using the target reward value to obtain a trained text classification model.

[0006] Further, the training data set constructed through initial verification includes: obtaining a plurality of original text samples, predicting labels of the original text samples by at least one preset automatic labeling method to obtain first predicted labels, and predicting labels of the original text samples by a benchmark language model to obtain second predicted labels; original text samples and corresponding predicted labels that the first predicted labels are consistent with the second predicted labels are taken as a first data set; For original text samples whose first predicted labels are inconsistent with second predicted labels, high-quality samples in the original text samples are determined by consistency verification based on a consensus mechanism, and the high-quality samples and corresponding consensus labels, as well as the original text samples and corresponding first predicted labels, are taken as a second data set; The first data set and / or the second data set are taken as the training data set.

[0007] Further, the automatic labeling method includes regular expression matching, clustering analysis based on vector model embedding, and zero sample prediction based on large language model; and / or, The benchmark language model includes a QwQ-32B model; and / or, The consensus label for the original text sample is determined by consistency verification based on a consensus mechanism, including: controlling the benchmark language model to perform multiple inferences on the original text sample under varying inference parameters to generate a plurality of candidate labels; determining a consensus label from the plurality of candidate labels based on a dynamic voting rule.

[0008] Further, the pre-trained language model adopts a group relative strategy optimization model based on a Qwen2.5-7b framework full training of a QwQ as the pre-trained language model, and the loss function is represented as follows:

[0009] wherein, denotes the pre-training language model loss function; denotes the model parameters; denotes the policy gradient loss; denotes the penalty loss; denotes the penalty weight; denotes the importance weight, which is the ratio of the current policy to the reference policy ; denotes the in-group advantage function obtained by adopting the group relative strategy optimization algorithm under the valueless network; clip denotes the clipping function; denotes the constraint parameter.

[0010] Further, the historical reasoning process corresponding to the target classification result of the response text is obtained in the dynamic historical cache area, and the similarity between the reasoning process in the response text and the historical reasoning process is calculated, comprising: In the dynamic historical cache area, a preset number of historical reasoning processes corresponding to the classification results same as the target classification result of the to-be-classified text are screened; The cosine similarity between the reasoning process in the response text and each of the historical reasoning processes is calculated respectively, and the average value of each of the cosine similarities is taken as a target cosine similarity.

[0011] Further, the determination of whether the reasoning process meets the diversity requirement by judging whether the similarity falls within a preset similarity interval, comprising: If the target cosine similarity falls within a preset similarity interval, it is considered that the reasoning process in the response text meets the diversity requirement, and the corresponding text is included in the dynamic historical cache area; If the target cosine similarity does not fall within a preset similarity interval, it is considered that the reasoning process in the response text does not meet the diversity requirement; Wherein, the preset similarity interval is [0.7, 0.9].

[0012] Further, the target reward value for updating the pre-training language model is determined based on the correctness of the target classification result and whether the reasoning process meets the diversity requirement, comprising: If the target classification result is correct and the reasoning process meets the diversity requirement, the target reward value is determined as a first preset positive value; If the target classification result is correct but the reasoning process does not meet the diversity requirement, the target reward value is determined as a second preset positive value; the first preset positive value is higher than the second preset positive value; If the target classification result is incorrect, the target reward value is determined as zero or a preset negative value.

[0013] Further, the method further comprises: For the same text to be classified, a plurality of candidate response texts are generated by controlling core parameters of the text classification model; the core parameters include using temperature and / or random seeds; Based on the classification results contained in the plurality of candidate response texts, a final text classification result is determined through a preset selection rule; the preset selection rule includes a weighted aggregation strategy and / or a voting strategy.

[0014] According to the second aspect of the present application, the present application provides a text classification model training system based on reinforcement learning, comprising: A structured generation module is configured to construct a training data set verified by an initial verification, and for the text to be classified in the training data set, at least one response text containing an inference process and a classification result is generated based on a preset pre-trained language model; A diversity evaluation module is configured to obtain a historical inference process corresponding to a target classification result of the text to be classified in a dynamic history cache area, calculate the similarity between the inference process in the response text and the historical inference process, and determine whether the inference process meets the diversity requirement by judging whether the similarity falls within a preset similarity interval; A reward value calculation module is configured to determine a target reward value for updating the pre-trained language model based on the correctness of the target classification result and whether the inference process meets the diversity requirement; A model training module is configured to update the parameters of the pre-trained language model through a reinforcement learning algorithm using the target reward value to obtain a trained text classification model.

[0015] According to the third aspect of the present application, the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the text classification model training method based on reinforcement learning according to any one of the first aspect of the present application.

[0016] The present application adopts the above technical solution and has at least the following beneficial effects: By the scheme, a training data set passing initial verification is constructed, at least one response text containing an inference process and a classification result is generated based on a preset pre-training language model for a text to be classified in the training data set, a historical inference process corresponding to a target classification result of the text to be classified is obtained from a dynamic history cache area, a similarity between the inference process in the response text and the historical inference process is calculated, and whether the inference process meets a diversity requirement is determined by judging whether the similarity falls into a preset similarity interval; a target reward value for updating the pre-training language model is determined based on the correctness of the target classification result and whether the inference process meets the diversity requirement; and parameters of the pre-training language model are updated by using the target reward value and through a reinforcement learning algorithm to obtain a trained text classification model. Thus, training data is generated through self-consistent verification, and the labeling cost is reduced; and the model's real inference ability and classification performance are improved by suppressing reward hacking behavior through a diversity reward.

[0017] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A flowchart of a text classification model training method based on reinforcement learning provided by an embodiment of the present application is shown; Figure 2 A flowchart of constructing a training data set passing initial verification provided by an embodiment of the present application is shown; Figure 3 A flowchart of judging whether an inference process meets a diversity requirement provided by an embodiment of the present application is shown; Figure 4 A structural diagram of a text classification model training system based on reinforcement learning provided by an embodiment of the present application is shown; Figure 5 An entity structure diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] This invention provides a method for training a text classification model based on reinforcement learning, such as... Figure 1 As shown, it may include at least the following steps S101~S104: Step S101: Construct a training dataset that has passed the initial validation. For the text to be classified in the training dataset, generate at least one response text containing the reasoning process and classification result based on the preset pre-trained language model.

[0023] The present invention aims to propose a method for training a pre-trained speech model using reinforcement learning, so as to obtain a text classification model that can suppress reward hacking behavior, improve real reasoning ability, and accurately predict text labels.

[0024] Prior to this, it is necessary to automatically construct a high-quality training dataset from massive amounts of raw, unlabeled, or weakly labeled text data with low manual cost, so that it can be used for subsequent reinforcement learning training. Figure 2 The diagram shown illustrates the process of constructing a training dataset that has passed initial validation.

[0025] First, multiple raw text samples are acquired, and various preset automated annotation methods are used to predict labels on the raw text samples to obtain the first predicted labels. For example, approximately 30,000 real ride-hailing user complaint texts are collected as raw text samples. These complaint texts vary in content, such as "The driver asked me to cancel the order and also asked me to pay 3 yuan in compensation," "Why did several reassigned cars not come?", "This driver keeps looking at his phone while driving, it's too dangerous," and "The system assigned a route too far, the driver directly called me and told me not to wait," etc.

[0026] In this embodiment of the invention, the automated annotation method may include regular expression matching, clustering analysis based on vector model embedding, and zero-shot prediction based on large language models. The working principles of these three automated annotation methods are briefly explained below: For complaint texts with obvious patterns, specific regular expressions can be used for quick classification. For example, a rule "driver (cancel / no show / make me cancel)" can be defined to match the category "driver no show"; a rule "driver (take a detour / take a longer route / inaccurate navigation)" can be defined to match the category "driver takes a detour". Regular expression matching can quickly assign labels to samples with obvious patterns.

[0027] For complaint texts that cannot be covered by regular expressions, a pre-trained text vectorization model (such as Sentence-BERT) can be used to convert each complaint text into a high-dimensional vector representation. Then, an unsupervised clustering algorithm (such as K-Means or DBSCAN) is applied to these vectors. After clustering, operations experts sample and review each cluster, assigning a unified initial label to clusters with clear business meaning. For example, if a cluster is found to contain text mostly related to "unpleasant odor in the car" or "unclean vehicle," the entire cluster can be labeled as "vehicle hygiene issues."

[0028] Furthermore, general-purpose large-scale language models (such as the base models of the Qwen series) can be used to design prompts for zero-shot or few-shot classification. For example, a prompt could be constructed as follows: "You are a ride-hailing complaint classification expert. Please classify the following complaint text into the most appropriate category. Optional categories include: [driver no-show / driver takes a detour / poor service attitude / vehicle hygiene issues / driving safety hazards]". This prompt is then combined with the complaint text and input into a large-scale language model to obtain its predicted category as the label.

[0029] In practical applications, each original text sample can be labeled using these three automated labeling methods. After the above labeling, each original text sample may have 2 to 3 corresponding labels. In this embodiment of the invention, while using the aforementioned automated annotation method to predict labels for the original text samples, a benchmark language model can also be used to predict labels for the original text samples, resulting in a second predicted label. The benchmark language model in this embodiment of the invention preferably adopts the Tongyi Qianwen QwQ-32B model. The QwQ-32B model is an inference model open-sourced by Alibaba Cloud in March 2025, possessing 32 billion parameters. With its unique reinforcement learning technology path, it significantly reduces the parameter size while achieving inference performance comparable to ultra-large-scale models, and can output inference data and labels containing the thinking process.

[0030] Furthermore, by comparing the first predicted label generated by the automated annotation method with the second predicted label generated by the QwQ-32B model, if the first predicted label and the second predicted label are completely consistent, the label of the original text sample is considered to have a high confidence level and can be directly adopted. The original text sample and its predicted label are then used as the first dataset. If the first predicted label and the second predicted label are not completely consistent, it indicates that the original text sample may be a difficult or boundary sample. At this point, consensus verification based on the consensus mechanism is initiated.

[0031] The core idea of ​​consensus verification is to discover "consensus" by leveraging the uncertainty of the model's output under random perturbations. Specifically, this involves controlling the baseline language model (QwQ-32B) to perform multiple independent inferences on the same original text sample under varying inference parameters. In this embodiment, the number of inferences can be set to 5. To introduce output diversity, two inference parameters are mainly changed: sampling temperature and random seed. The sampling temperature is a parameter used to control the randomness of the model's output, determining whether the model should be "conservative" or "boldly innovative" when generating text. In practical applications, the sampling temperature can be set to a higher value (e.g., 1.0). A higher temperature increases the randomness of the model's generated results, making the model more inclined to explore different candidate words, thus producing more diverse outputs. Five different random seeds can be set for each of the 5 inferences, such as [42, 84, 126, 230, 334]. Generally, deterministic problems will output the same label even with seed changes, but some meaningful samples will output different labels under different seeds. Different random seeds ensure that the random sampling process for each inference is unique, thus obtaining different inference results and 5 candidate labels.

[0032] In this embodiment of the invention, the consensus mechanism can be configured with a dynamic voting mechanism. Specifically, it counts the votes for five candidate labels, and when a candidate label receives a number of votes that meets a preset threshold (e.g., greater than or equal to four votes), it is considered that a high degree of consensus has been reached within the model, and that candidate label is determined as the final consensus label. This sample with the consensus label is considered a rigorously validated high-quality sample and is stored in a second dataset along with the original text sample labeled with the first predicted label.

[0033] In one optional embodiment, if after 5 rounds of voting, no candidate label receives 4 votes or more, the sample is considered to have too much disagreement, and the model cannot reach a stable consensus. Such original text samples can be pushed to a dedicated human review queue for final decision-making by human experts. The manually annotated original text samples can also be saved to a second dataset.

[0034] Finally, the first dataset directly adopted through consistency judgment and the second dataset generated through consistency verification can be used as training data for subsequent steps. For example, the first and second datasets can be merged to form a high-quality training dataset for the next stage of model training; and the first and second datasets can be used to train the SFT training method based on the MS-Swift framework and the RFT training method based on the VERL framework, which serve as the control group of the present invention.

[0035] Understandably, through the screening steps of the first and second datasets described above, worthless data has been removed to avoid errors in the training samples; multi-labeled samples are better able to utilize the RL hierarchical reward method to obtain the best prediction model; and reliable labeled data can be screened and generated to a large extent automatically, significantly reducing the reliance on purely manual labeling.

[0036] After constructing a training dataset that has passed initial validation, this embodiment of the invention utilizes this training dataset to train a text classification model that is not only accurate in classification but also has a reasonable and diverse reasoning process. Preferably, a Group Relative Policy Optimization (GRPO) model based on the Qwen2.5-7b framework and trained on the entire dataset is used as the pre-trained language model.

[0037] Qwen2.5-7B is a lightweight large-scale model with 7 billion parameters launched by the Alibaba Cloud Tongyi Qianwen team. It performs outstandingly in text processing, multimodal interaction, and other fields, while also being lightweight and adaptable to various application scenarios. Group Relative Policy Optimization (GRPO), as a category of reinforcement learning from verifiable rewards, is a reinforcement learning algorithm specifically optimized for large-scale language models. This invention applies its group relative advantage evaluation and valueless network (Critic) design, introducing a memory optimization mechanism to constrain reward values ​​during inference, preventing the model from overestimating reward information and effectively suppressing reward hacking.

[0038] The working principle of group relative advantage assessment is to generate N responses for each prompt word (typically with rollout=4, i.e., N = 4 responses), calculate the difference between the reward value of each response and the group mean (Advantage), and thus form the relative advantage. The valueless network (Critic) design refers to updating the strategy directly based on the relative advantage of the group responses, without requiring additional value function estimation. The loss function of GRPO is expressed as follows:

[0039] in, This represents the loss function of the pre-trained language model; Indicates model parameters; Represents the policy gradient loss; express Punishment loss; Indicates the penalty weight; This represents the importance weight, which is the current strategy. Reference Strategy The ratio; This represents the within-group advantage function obtained using a group relative strategy optimization algorithm in a valueless network; clip Represents the clipping function; Represents constraint parameters.

[0040] This invention, based on a pre-trained language model with the above-described structure, generates response text containing the reasoning process and classification results by inputting the text to be classified from the training dataset. It should be noted that the training data is designed to be in a structured format containing the input text, the reasoning process, and the classification results. Two examples are given below: Example 1: Input: Oh my god, the driver made me cancel the order and even asked me to pay him 3 yuan in compensation. (Input text) output: <think>According to the passenger's incoming text content, it is first necessary to determine whether it meets the definition of the cancellation fee problem. The cancellation fee problem refers to the cancellation of an order, with cancellation fees, waiting fees, etc. The cancellation fee is not recognized, such as the driver's reason causing the passenger to be forced to cancel the order, and the resulting cancellation fee should not be charged. From the text, it can be seen that the passenger mentioned that the driver asked him to cancel the order and needed to compensate 3 yuan, which meets the definition of the cancellation fee problem. Therefore, this text can be classified as a cancellation fee problem.< / think> (Reasoning process) <answer>Cancellation fee problem< / answer> (Classification results) Example 2: Input: Several reassigned vehicles haven't arrived (input text) output: <think>According to the passenger's incoming text content, it is necessary to determine whether it meets the definition of the rescheduling problem. The vehicle after rescheduling does not arrive, which meets the definition of the rescheduling problem.< / think> (Reasoning process) <answer>Rescheduling problem< / answer> (Reasoning process) Understandably, in the aforementioned application scenarios, reward hacking may occur, where the generated text excessively pursues a high reward score, leading to estimation fraud, duplicate labels, or generating multiple labels to inflate the score. Furthermore, the model may prematurely converge to a specific label, preventing it from generating text under subsequent prompts. <think>< / think> Key content such as... To address this issue, embodiments of the present invention introduce a memory optimization mechanism during the training process of the pre-trained language model to control... <think>< / think> The generation process allows it to consider various possible directions to determine the correct label, avoiding the use of identical labels. <think>< / think> The principle of this mechanism will be explained in detail below: Step S102: Obtain the historical reasoning process corresponding to the target classification result of the text to be classified from the dynamic historical cache area, calculate the similarity between the reasoning process in the response text and the historical reasoning process, and determine whether the reasoning process meets the diversity requirements by judging whether the similarity falls within the preset similarity interval.

[0041] In this embodiment of the invention, the dynamic history cache can be configured as a first-in, first-out buffer. Its function is to store historical reasoning processes that have been verified as correct, and each historical reasoning process corresponds to a category label. The dynamic history cache can maintain a set of queues divided by category labels, where the key is the category label (such as "driver no-show") and the value is a queue. This queue stores the most recent few (e.g., 10) historical reasoning processes that have been judged as correct and correspond to the category label "driver no-show".

[0042] Furthermore, in the dynamic historical cache, a preset number of historical reasoning processes corresponding to the same classification result as the target classification result of the text to be classified are selected; the cosine similarity between the reasoning process in the response text and each historical reasoning process is calculated, and the average value of each cosine similarity is taken as the target cosine similarity.

[0043] like Figure 3The diagram illustrates the process for determining whether the reasoning process of a pre-trained language model meets diversity requirements. In practice, the reasoning process in the newly generated response text can be converted into a vector representation using a text vectorization model. Then, based on the classification results (labels) in the current response text, 10 historical reasoning processes from the corresponding queue are extracted and converted into historical reasoning process vectors as well. The cosine similarity between the newly generated reasoning process vector and the 10 historical reasoning process vectors is calculated, resulting in 10 cosine similarity values. The average of these 10 cosine similarity values ​​is then used as the target cosine similarity.

[0044] Furthermore, if the target cosine similarity falls within the preset similarity range, the reasoning process in the response text is considered to meet the diversity requirements, and the corresponding text is included in the dynamic historical cache; if the target cosine similarity does not fall within the preset similarity range, the reasoning process in the response text is considered not to meet the diversity requirements.

[0045] To ensure accurate targeting, this invention, through extensive testing, preferably configures the preset similarity interval as [0.7, 0.9]. If the calculated target cosine similarity value is too high (e.g., greater than 0.9), the new reasoning process is considered too similar to historical records, lacking innovation, and a sign of "rewarding hacking" behavior; therefore, it is deemed not to meet the diversity requirement. If the calculated target cosine similarity value is too low (e.g., less than 0.7), the new reasoning process is considered to have weak relevance to the topic, logical drift, and is invalid or low-quality reasoning; therefore, it is also deemed not to meet the diversity requirement. Only when the target cosine similarity value falls within the preset interval [0.7, 0.9] is the reasoning process considered to maintain relevance to the topic while demonstrating sufficient innovation and diversity. In this case, it is determined to meet the diversity requirement, and the newly generated reasoning process is added to the corresponding queue in the dynamic history cache. Simultaneously, the oldest historical reasoning process in the queue is removed to maintain a fixed queue capacity and complete the update of the dynamic history cache.

[0046] Step S103: Based on the correctness of the target classification result and whether the reasoning process meets the diversity requirements, determine the target reward value used to update the pre-trained language model.

[0047] This invention employs a non-sparse reward design to determine the target reward value. Specifically: if the target classification result is correct and the reasoning process meets the diversity requirements, the target reward value is determined to be a first preset positive value; if the target classification result is correct but the reasoning process does not meet the diversity requirements, the target reward value is determined to be a second preset positive value; the first preset positive value is higher than the second preset positive value; if the target classification result is incorrect, the target reward value is determined to be zero or a preset negative value.

[0048] In other words, the target classification result of the text to be classified is compared with the corresponding label in the training dataset. If the target classification result matches the label, i.e., the prediction is correct, there are two scenarios: If the target cosine similarity of the inference process falls within [0.7, 0.9], a first-preset positive reward of the highest value is assigned, for example, reward=1.0, aiming to incentivize the model to actively explore new and effective thinking methods while ensuring the correctness of the result. If the target cosine similarity of the inference process does not fall within [0.7, 0.9], a second-preset positive reward of a lower value is assigned, for example, reward=0.5, ensuring that the model can still receive positive feedback as long as the answer is correct, even if the exploration is unsuccessful, but the reward is significantly lower than in the previous scenario, thus guiding the model out of its comfort zone. If the target classification result does not match the label, i.e., the prediction is incorrect, regardless of the inference process, a reward of zero or one preset positive value is assigned, for example, reward=0 or -0.2, giving the model the strongest penalty signal to ensure that the accuracy of classification is the primary goal.

[0049] By using the above methods, positive and negative rewards can be clearly distinguished, effectively avoiding the problem of reward hacking while maintaining training efficiency.

[0050] Step S104: Using the target reward value, update the parameters of the pre-trained language model through a reinforcement learning algorithm to obtain the trained text classification model.

[0051] The reward value generated in step S103 is immediately fed back to the GRPO algorithm to calculate the policy gradient and update the parameters of the pre-trained language model.

[0052] First, calculate the average reward value (group mean) under the same prompt, which serves as the value function in the Critic network: V(s)≈mean(R_group)=mean(R1, R2,..., Ri,..., RN) Here, Ri represents the reward value corresponding to each reasoning process.

[0053] Secondly, by calculating the mean and standard deviation of the group reward, the odds value of each reply is: =(Ri-mean(R_group)) / standard deviation Furthermore, based on the loss function shown in step S101, the population relative policy optimization model is updated by advantage value weighting, and a KL divergence penalty term is introduced to limit the policy update magnitude and avoid overfitting.

[0054] It is understandable that the loss function The calculation depends on the policy gradient loss and an optional KL penalty loss. In another alternative embodiment, a compound reward value can also be calculated as the advantage function. The core input, namely =Ri-mean(R_group), which in turn affects the magnitude and direction of the policy gradient loss. Accordingly, higher reward values ​​will encourage the model to increase the probability of generating similar high-quality responses; lower positive reward values ​​provide weaker positive gradients; while negative reward values ​​will encourage the model to reduce the probability of generating incorrect responses. Finally, through the backpropagation algorithm, this loss is used to calculate the gradient of the model parameters and adjust the weights of the pre-trained language model accordingly, thus completing one training iteration.

[0055] In this way, the pre-trained language model is not only motivated to find the correct answer, but also to explore and generate diverse and high-quality reasoning paths, thereby training a text classification model with truly powerful logical reasoning capabilities.

[0056] In an optional embodiment, the text classification model training method based on reinforcement learning proposed in this invention has enhanced reasoning diversity, so the final output can adopt a preset selection rule based on temperature sampling to output the text classification result.

[0057] Similar to the consensus rules mentioned above, this embodiment of the invention generates multiple candidate response texts for the same text to be classified by controlling the core parameters of the text classification model (using temperature and random seeds); based on the classification results contained in the multiple candidate response texts, the final text classification result is determined by a preset selection rule.

[0058] In this embodiment of the invention, the preset selection rule can be a weighted aggregation strategy or a voting strategy. Below is a specific example of text classification and inference processes under three different adoption temperature settings: Input: Oh my god, the driver made me cancel the order and even asked me to pay 3 yuan in compensation. First setting: temperature=1, seed=42: <think>According to the passenger's incoming text content, it is first necessary to determine whether it meets the definition of the cancellation fee problem. The cancellation fee problem refers to the cancellation of an order, with cancellation fees, waiting fees, etc. The cancellation fee is not recognized, such as the driver's reason causing the passenger to be forced to cancel the order, and the resulting cancellation fee should not be charged. From the text, it can be seen that the passenger mentioned that the driver asked him to cancel the order and needed to compensate 3 yuan, which meets the definition of the cancellation fee problem. Therefore, this text can be classified as a cancellation fee problem.< / think> <answer>Cancellation fee problem< / answer> The second setting: temperature=0.7, seed=128. <think>According to the passenger's incoming text content, it is first necessary to determine whether it meets the definition of the cancellation fee problem. The cancellation fee problem refers to the cancellation of an order, with cancellation fees, waiting fees, etc. The cancellation fee is not recognized, such as the driver's reason causing the passenger to be forced to cancel the order, and the resulting cancellation fee should not be charged. From the text, it can be seen that the passenger mentioned that the driver asked him to cancel the order and needed to compensate 3 yuan, which meets the definition of the cancellation fee problem. Therefore, this text can be classified as a cancellation fee problem.< / think> <answer>Cancellation fee problem< / answer> The third setting: temperature=0.3, seed=200. <think>According to the passenger's incoming text content "Oh my god, the driver asked me to cancel the order and let me compensate 3 yuan.", first, the text mentions that the driver asked the passenger to cancel the order, which meets the definition of "driver's no-show", i.e. the driver induces the passenger to cancel the order. At the same time, the text also mentions that the driver asked the passenger to compensate 3 yuan, which may involve the cancellation fee problem, i.e. "cancellation fee problem". However, since the text does not explicitly state that the passenger was forced to cancel the order due to the driver's reason, the definition of the cancellation fee problem does not fully apply. Considering that the definition of "driver's no-show" is more general and has higher priority, it should be given priority to "driver's no-show".< / think> <answer>Driver's no-show< / answer> As shown above, the results generated from the three settings are "Cash-out problem", "Cash-out problem", and "Driver no-show". If a voting strategy is used, "Cash-out problem" becomes the final text classification result because it appears twice. If a weighted aggregation strategy is used, multiple candidate response texts are generated for the same input, and their corresponding confidence scores are obtained. The total confidence score for the same classification result is calculated and compared with the total confidence scores of other classification results. The classification result with the highest total confidence score is taken as the final text classification result. In practical applications, specific preset selection rules can be set according to actual needs, and this invention does not limit this.

[0059] The text classification model trained using the embodiments of this invention achieves parameter convergence after sufficient training iterations. When faced with new ride-hailing complaint texts, it can generate logically clear and diverse reasoning processes and provide accurate classification results. Comparative experiments show that the SFT training method based on the MS-Swift framework and the RFT training method based on the VERL framework achieves a classification accuracy of 88%, while the RFT training method based on the VERL framework achieves 85%. The text classification model trained using the embodiments of this invention, however, achieves a classification accuracy of 95%.

[0060] This invention provides a method for training a text classification model based on reinforcement learning, comprising: constructing a training dataset that has passed initial validation; for the text to be classified in the training dataset, generating at least one response text containing a reasoning process and a classification result based on a preset pre-trained language model; obtaining the historical reasoning process corresponding to the target classification result of the text to be classified from a dynamic historical cache; calculating the similarity between the reasoning process in the response text and the historical reasoning process; and determining whether the reasoning process meets the diversity requirement by judging whether the similarity falls within a preset similarity interval; determining a target reward value for updating the pre-trained language model based on the correctness of the target classification result and whether the reasoning process meets the diversity requirement; and updating the parameters of the pre-trained language model using a reinforcement learning algorithm with the target reward value to obtain a trained text classification model. This invention has at least the following beneficial effects: 1) By introducing reward signals based on the diversity of reasoning processes, reward hacking behavior in the reinforcement learning training process can be effectively suppressed, and the model can be encouraged to generate diverse and high-quality reasoning paths, rather than repetitive and low-quality "shortcut" reasoning, thereby improving the model's real logical reasoning ability and generalization ability. 2) By introducing a self-consistent verification mechanism in the data generation stage, when the automated pre-labeling is inconsistent with the baseline model prediction, high-quality training samples can be automatically generated through multiple inferences and consensus decisions, which greatly reduces the dependence on expensive and time-consuming manual fine labeling and significantly reduces the cost of data preparation. 3) Due to the higher quality of the training data and the stronger reasoning ability of the model, the trained model has higher accuracy and automation rate in text classification tasks, which can significantly improve business processing efficiency and improve user experience.

[0061] Furthermore, as Figure 1 In specific implementation, embodiments of the present invention provide a text classification model training system based on reinforcement learning, such as... Figure 4 As shown, the device may include: a structured generation module 410, a diversity assessment module 420, a reward value calculation module 430, and a model training module 440.

[0062] The structured generation module 410 can be used to construct a training dataset that has passed initial validation, and generate at least one response text containing the reasoning process and classification result based on a preset pre-trained language model for the text to be classified in the training dataset. The diversity assessment module 420 can be used to obtain the historical reasoning process corresponding to the target classification result of the text to be classified from the dynamic historical cache, calculate the similarity between the reasoning process in the response text and the historical reasoning process, and determine whether the reasoning process meets the diversity requirements by judging whether the similarity falls within the preset similarity range. The reward value calculation module 430 can be used to determine the target reward value for updating the pre-trained language model based on the correctness of the target classification result and whether the reasoning process meets the diversity requirements. The model training module 440 can be used to update the parameters of the pre-trained language model using the target reward value through a reinforcement learning algorithm to obtain the trained text classification model.

[0063] It should be noted that other corresponding descriptions of the functional modules involved in the text classification model training system based on reinforcement learning provided in this embodiment of the invention can be found in [reference needed]. Figure 1 The corresponding description of the method shown will not be repeated here.

[0064] Based on the above, Figure 1 Accordingly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the reinforcement learning-based text classification model training method of any of the above embodiments.

[0065] Based on the above, Figure 1The method shown and as Figure 4 The embodiment of the system shown in the invention also provides a physical structure diagram of a computer device, such as... Figure 5 As shown, the computer device may include a communication bus, a processor, a memory, and a communication interface. It may also include input / output interfaces and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory, performing the steps of the reinforcement learning-based text classification model training method described in the above embodiments.

[0066] Those skilled in the art will clearly understand that the specific working process of the systems, devices, modules and units described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0067] Furthermore, the functional units in the various embodiments of the present invention can be physically independent of each other, or two or more functional units can be integrated together, or all functional units can be integrated into one processing unit. The integrated functional units described above can be implemented in hardware, or in software or firmware.

[0068] Those skilled in the art will understand that if the integrated functional unit is implemented in software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of the present invention when running the instructions. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as a computing device, personal computer, server, or network device) related to program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the computing device, the computing device executes all or part of the steps of the methods described in the various embodiments of the present invention.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to depart from the protection scope of the present invention.

Claims

1. A method for training a text classification model based on reinforcement learning, characterized in that, The method comprises: constructing a training data set verified by an initial verification, generating at least one response text containing an inference process and a classification result based on a preset pre-training language model for the text to be classified in the training data set; obtaining the historical inference process corresponding to the target classification result of the text to be classified in the dynamic history cache area, calculating the similarity between the inference process in the response text and the historical inference process, and determining whether the inference process meets the diversity requirement by judging whether the similarity falls within a preset similarity interval; based on the correctness of the target classification result and whether the inference process meets the diversity requirement, determining the target reward value for updating the pre-training language model; using the target reward value, updating the parameters of the pre-training language model through a reinforcement learning algorithm to obtain a trained text classification model.

2. The method of claim 1, wherein, The method comprises: obtaining a plurality of original text samples, predicting the labels of the original text samples using at least one preset automatic labeling method to obtain first predicted labels, and predicting the labels of the original text samples using a benchmark language model to obtain second predicted labels; the original text samples and corresponding predicted labels whose first predicted labels and second predicted labels are consistent are taken as a first data set; for original text samples whose first predicted labels and second predicted labels are inconsistent, high-quality samples in the original text samples are determined through consistency verification based on a consensus mechanism, and the high-quality samples and corresponding consensus labels, as well as the original text samples and corresponding first predicted labels, are taken as a second data set; the first data set and / or the second data set are taken as the training data set.

3. The method of claim 2, wherein, The automatic labeling method comprises regular expression matching, vector model embedding-based clustering analysis, and zero-sample prediction based on a large language model; and / or The benchmark language model comprises a QwQ-32B model; and / or The method comprises: controlling the benchmark language model to infer the original text samples multiple times under varying inference parameters to generate multiple candidate labels; determining a consensus label from the multiple candidate labels based on a dynamic voting rule.

4. The method of claim 2, wherein, The pre-training language model adopts a group relative strategy optimization model based on a Qwen2.5-7b framework full training as the pre-training language model, and the loss function is represented as follows: wherein, denotes the pre-trained language model loss function; denotes the model parameters; denotes the policy gradient loss; denotes the penalty loss; denotes the penalty weight; denotes the importance weight, being the ratio of the current policy to the reference policy ; denotes the in-group advantage function obtained by employing the group relative policy optimization algorithm under the valueless network; clip denotes the clipping function; denotes the constraint parameter.

5. The method of claim 1, wherein, The method comprises: in the dynamic history cache area, filtering a preset number of historical inference processes corresponding to the classification results identical to the target classification result of the text to be classified; respectively calculating the cosine similarity between the inference process in the response text and each of the historical inference processes, and taking the average value of each of the cosine similarities as a target cosine similarity.

6. The method of claim 5, wherein, The determining whether the reasoning process meets the diversity requirement by judging whether the similarity falls within a preset similarity interval comprises: If the target cosine similarity falls within a preset similarity interval, it is considered that the reasoning process in the response text meets the diversity requirement, and the corresponding text is included in the dynamic history cache area; If the target cosine similarity does not fall within a preset similarity interval, it is considered that the reasoning process in the response text does not meet the diversity requirement. The preset similarity interval is [0.7, 0.9].

7. The method of claim 1, wherein, The determining the target reward value for updating the pre-trained language model based on the correctness of the target classification result and whether the reasoning process meets the diversity requirement comprises: If the target classification result is correct and the reasoning process meets the diversity requirement, the target reward value is determined to be a first preset positive value; If the target classification result is correct but the reasoning process does not meet the diversity requirement, the target reward value is determined to be a second preset positive value; the first preset positive value is higher than the second preset positive value; If the target classification result is incorrect, the target reward value is determined to be zero or a preset negative value.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: For the same text to be classified, a plurality of candidate response texts are generated by controlling core parameters of the text classification model; the core parameters include using temperature and / or random seeds; Based on the classification results contained in the plurality of candidate response texts, a final text classification result is determined by a preset selection rule; the preset selection rule includes a weighted aggregation strategy and / or a voting strategy. 9.A system for training a text classification model based on reinforcement learning, characterized in that, The system comprises: A structured generation module is configured to construct an initial verified training data set, and for the text to be classified in the training data set, generate at least one response text containing a reasoning process and a classification result based on a preset pre-trained language model; A diversity evaluation module is configured to obtain a historical reasoning process corresponding to a target classification result of the text to be classified in a dynamic history cache area, calculate the similarity between the reasoning process in the response text and the historical reasoning process, and determine whether the reasoning process meets the diversity requirement by judging whether the similarity falls within a preset similarity interval; A reward value calculation module is configured to determine a target reward value for updating the pre-trained language model based on the correctness of the target classification result and whether the reasoning process meets the diversity requirement; A model training module is configured to update the parameters of the pre-trained language model by a reinforcement learning algorithm using the target reward value to obtain a trained text classification model.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the reinforcement learning-based text classification model training method of any one of claims 1-8.

Images