Reinforcement learning-based long chain optimization method and device for large inference model

Abstract

This disclosure relates to the field of artificial intelligence technology, and in particular provides a method and apparatus for optimizing long-chain inference models based on reinforcement learning. The method includes: acquiring a fine-tuned model obtained by fine-tuning a basic large-scale inference model using a thought chain dataset, and an inference optimization dataset; using the inference optimization dataset, sequentially performing long-chain reinforcement learning training and short-chain reinforcement learning training on the fine-tuned model to obtain a large-scale inference model; wherein, long-chain reinforcement learning training is used to guide the fine-tuned model to learn to generate long inference chains that are accurate and formatted correctly; short-chain reinforcement learning training is used to guide the fine-tuned model to learn to compress long inference chains to generate short inference chains, provided that the inference is accurate and formatted correctly. This disclosure can solve the problems of difficulty in balancing inference accuracy and formatted correctness, and the redundancy of long-chain inference, by using a hierarchical progressive optimization approach of "first converting short chains to long chains, then converting long chains to short chains," thereby improving the inference efficiency of large-scale inference models.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus for optimizing long chains of large inference models based on reinforcement learning. Background Technology

[0002] In the field of artificial intelligence, large language models are widely used to solve complex reasoning problems, such as mathematical calculations or code generation. To make the model's thought process clearer, a common method is to have it generate a complete thought chain, that is, a step-by-step reasoning process. However, this thought chain reasoning method usually has the following drawbacks: the generated thought chains are often too long, containing a lot of repetitive or unnecessary content, resulting in low answering efficiency and consuming a lot of computing resources; at the same time, the model is prone to getting stuck in loops or irregular answer formats when generating long chains, affecting the actual use effect.

[0003] Among related technologies, there are two common optimization methods: one is to forcibly shorten the output by designing prompt words or modifying training data, but this often impairs the model's reasoning ability; the other is to use reinforcement learning methods to guide the model to optimize itself by setting rewards, but the existing models have a long thought chain and low reasoning efficiency. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides a method and apparatus for long-chain optimization of large inference models based on reinforcement learning.

[0005] According to one aspect of this disclosure, a long-chain optimization method for large inference models based on reinforcement learning is provided, comprising: Obtain the fine-tuned model obtained by fine-tuning the basic large model using the MindChain dataset, as well as the inference optimization dataset; Using the aforementioned inference optimization dataset, the fine-tuned model is sequentially trained using long-chain reinforcement learning and short-chain reinforcement learning to obtain a large inference model. The long-chain reinforcement learning training is used to guide the fine-tuning model to learn to generate long inference chains that are accurate and formatted correctly; the short-chain reinforcement learning training is used to guide the fine-tuning model to learn to compress the long inference chains to generate short inference chains, provided that the inference is accurate and formatted correctly.

[0006] According to another aspect of this disclosure, a long-chain optimization device for large inference models based on reinforcement learning is provided, comprising: The acquisition module is used to acquire the fine-tuned model obtained by fine-tuning the basic large model through the MindChain dataset, as well as the inference optimization dataset; The training module is used to perform long-chain reinforcement learning training and short-chain reinforcement learning training on the fine-tuned model sequentially using the inference optimization dataset to obtain the large inference model. The long-chain reinforcement learning training is used to guide the fine-tuning model to learn to generate long inference chains that are accurate and formatted correctly; the short-chain reinforcement learning training is used to guide the fine-tuning model to learn to compress the long inference chains to generate short inference chains, provided that the inference is accurate and formatted correctly.

[0007] In another aspect of exemplary embodiments of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the methods described in exemplary embodiments of this disclosure.

[0008] In another aspect of exemplary embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described in exemplary embodiments of the present disclosure.

[0009] In another aspect of the exemplary embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the exemplary embodiments of this disclosure.

[0010] As will be described in detail below, the reinforcement learning-based long-chain optimization method for large inference models according to embodiments of this disclosure obtains a fine-tuned model obtained by fine-tuning a basic large model using a thought chain dataset, and an inference optimization dataset. Using the inference optimization dataset, the fine-tuned model is sequentially trained with long-chain reinforcement learning and short-chain reinforcement learning to obtain a large inference model. The long-chain reinforcement learning training guides the fine-tuned model to learn to generate long inference chains that are accurate and formatted correctly. The short-chain reinforcement learning training, under the premise of accurate and formatted inference, guides the fine-tuned model to learn to compress long inference chains to generate short inference chains. This hierarchical, progressive optimization, first converting short chains to long chains and then long chains to short chains, ensures both the accuracy and standardization of long inference chains and the efficiency and completeness of short inference chains. This solves the shortcomings of existing optimization methods that, due to their "one-step" approach, struggle to balance inference accuracy and formatted correctness, and suffer from redundant long-chain inference, thereby improving the inference efficiency of the large inference model.

[0011] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 A flowchart illustrating the long-chain optimization method for large inference models based on reinforcement learning provided in an exemplary embodiment of this disclosure is shown. Figure 2 A schematic diagram of the structure of a long-chain optimization device for large inference models based on reinforcement learning provided in an exemplary embodiment of this disclosure is shown. Figure 3 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown; Figure 4 A schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0015] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0016] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] Long to Short Reasoning: This refers to a technical approach that systematically optimizes complex long reasoning chains generated by large language models into efficient short reasoning chains, thereby achieving intensive use of computing resources while ensuring the accuracy of reasoning.

[0020] Grouped Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that updates parameters by calculating the relative advantages within a policy group.

[0021] Chain of Thought (CoT) refers to a method of problem-solving that allows machine learning models to mimic human thought processes. By constructing a series of logically coherent steps, rather than jumping all the way to the final answer, CoT enhances the model's understanding and interpretability of complex problems. This technique is commonly used to improve the performance of large language models in areas such as open-ended question answering and mathematical reasoning.

[0022] Large language models refer to language processing neural network models with a massive number of parameters (such as billions or even more). These models, trained on large amounts of text data, demonstrate exceptional capabilities in natural language understanding and generation. They are currently a hot topic in artificial intelligence research.

[0023] Deep reasoning large-scale models: These are large-scale neural network models focused on performing deep logical reasoning tasks. In addition to possessing powerful language understanding capabilities, these models can also perform complex causal relationship analysis, hypothesis testing, and other activities, and are widely used in fields such as legal consulting and scientific research.

[0024] Rule-based rewards: In certain reinforcement learning scenarios, these are a set of explicit guiding principles used to encourage or punish specific behavioral patterns. Appropriately designed rule-based rewards can help accelerate the learning process and guide the agent towards desired goals.

[0025] Reward model: A function specifically designed to predict the reward value obtained by taking a certain action in a given state. It is one of the core components of the reinforcement learning framework and directly affects the performance of the entire learning system.

[0026] Value model: Estimates the total accumulated discounted reward that can be obtained by following the current strategy starting from a certain state. Value model provides a long-term evaluation criterion for decision-making and plays a key role in planning future actions.

[0027] Reinforcement learning is a learning method based on trial and error, which continuously adjusts its behavioral strategies through interaction with the environment to maximize expected rewards. Reinforcement learning is widely used in various fields such as game AI and robot navigation.

[0028] Instruction fine-tuning: The process of further training a pre-trained model for a specific task, with the aim of making it better suited to the requirements of the new task. This typically involves fine-tuning the model using a small amount of labeled data to improve its performance in a specific application scenario.

[0029] With the breakthrough development of long reasoning technology, reasoning models represented by Deep Seek R1 / R1Distill and QwQ-32B have demonstrated systematic reasoning capabilities in complex logical tasks such as mathematical problem solving, code generation, and multidisciplinary question answering, marking a key leap for large models from "probabilistic generation" to "human-like logical deduction". Its technological evolution path can be divided into three stages: (1) Prompt Engineering: The model is guided to generate intermediate reasoning steps through CoT prompts, but the reasoning chain capability is limited by the logical generalization ability of the basic model; (2) Instruction Tuning: High-density thinking chain data is used to inject structured reasoning prior knowledge, which poses a risk of knowledge solidification; (3) Reinforcement Learning Reasoning or RL Reasoning: Autonomous evolution mechanism, based on dynamic reward function (such as format reward, answer accuracy reward) to drive the model to autonomously explore the optimal reasoning strategy, breaking through the data limitations of instruction tuning, such as Deep Seek-R1, which uses GRPO and achieves a leap in reasoning ability through rule rewards.

[0030] In related technologies, long-to-short reasoning optimization solutions are mainly divided into two categories: (1) Cue word constraint method: This method forcibly constrains the output length of a large model by using cue words. However, this approach has shown low effectiveness in general large models. Although it constrains the length of the thought chain, it also constrains the model's reasoning ability. For example, the chain of drafts forcibly constrains the length of each reasoning step.

[0031] (2) Data compression and fine-tuning method: By modifying the thought chain data, the length of the thought chain is compressed, and then the model is trained to generate a shorter thought chain through instruction fine-tuning. Examples include Light Thinker, Token Skip, and Token-Budget-Aware LLM Reasoning.

[0032] The long-to-short chain inference optimization scheme has the following technical problems: (1) Cue words restrict reasoning ability: Although the method of forcibly constraining the output length of a large model by using cue words can reduce redundant information, it also restricts the model's autonomous reasoning ability, which can easily lead to the loss of key reasoning steps, thus hindering the solution of complex problems.

[0033] (2) Accuracy Loss in Mind Chain Compression: The effectiveness of schemes based on instruction fine-tuning and mind chain data compression is highly dependent on the compression quality of the mind chain data. These methods typically employ strategies such as token compression and path pruning to reduce output length. However, mind chain pruning based on manual rules leads to accuracy loss, which may be amplified in certain task scenarios, thus affecting the quality of the final decision. Furthermore, the model cannot autonomously explore shorter inference paths, exhibiting limitations in capability and effectiveness.

[0034] Among reinforcement learning inference schemes, Deep Seek-R1 is currently the most mainstream and best-performing. It employs the GRPO reinforcement learning scheme, eliminating the need for reward and value models. Based on rule-based rewards of accuracy and format, it enables large models to possess leading deep inference capabilities and boasts high training efficiency. However, it suffers from the following technical issues: (1) Abnormal content generation: The model exhibits abnormal content generation. In practical applications, the model is prone to deadlocks in the thought process. For example, when testing Deep Seek-type reasoning models, it was found that the output of some math or coding problems would get stuck in an infinite loop of reasoning steps, failing to reach the final solution smoothly. This not only increases reasoning time but also wastes computing resources, and at the same time, makes it impossible to solve the problem effectively.

[0035] (2) Bottleneck of reasoning chain efficiency: The thought chains generated by long-chain reasoning models usually contain a lot of redundant information, such as repeatedly deriving the same formula or over-explaining basic concepts. Although such lengthy outputs may improve the accuracy of the final answer, they also significantly increase the consumption of reasoning time and computing resources, thus reducing reasoning efficiency.

[0036] (3) Inaccurate answer format: The current solution only standardizes the format of the thought process chain, without explicitly rewarding the format of the final answer. Therefore, the generated answers may have non-standard formats (such as numbers not arranged according to rules, codes not placed in code blocks, etc.), which will cause the automatic answer extraction to fail, thus affecting the accuracy of the evaluation results and hindering the guidance of algorithm optimization.

[0037] Therefore, in order to solve the above problems, this disclosure provides a method for optimizing long chains of large inference models based on reinforcement learning.

[0038] The reinforcement learning-based long-chain optimization method for large inference models provided in this disclosure can be executed by a terminal or by a chip applied to the terminal.

[0039] For example, the aforementioned terminal may include one or more of the following: mobile phone, tablet computer, wearable device, in-vehicle device, laptop computer, ultra-mobile personal computer (UMPC), netbook, handheld computer (PDA), and wearable device based on augmented reality (AR) and / or virtual reality (VR) technology. The exemplary embodiments disclosed herein do not impose specific limitations on these.

[0040] Figure 1 A flowchart illustrating a long-chain optimization method for large inference models based on reinforcement learning, provided in an exemplary embodiment of this disclosure, is shown. Figure 1 As shown, this reinforcement learning-based long-chain optimization method for large inference models includes: S101, Obtain the fine-tuned model obtained by fine-tuning the basic large model through the MindChain dataset, and the inference optimization dataset; S102, using the inference optimization dataset, the fine-tuning model is sequentially trained with long-chain reinforcement learning and short-chain reinforcement learning to obtain a large inference model; among them, long-chain reinforcement learning training is used to guide the fine-tuning model to learn to generate long inference chains that are accurate and have a standard format; short-chain reinforcement learning training is used to guide the fine-tuning model to learn to compress long inference chains to generate short inference chains, provided that the inference is accurate and has a standard format.

[0041] Specifically, the aforementioned basic large model can be a pre-trained large language model, capable of semantic parsing, information extraction, and preliminary logical deduction of input text, providing a core model foundation for subsequent optimization of reasoning capabilities. However, because the basic large model has not undergone fine-tuning training related to thought chains and reinforcement learning, its reasoning output is mostly direct conclusions, lacking explicit step-by-step reasoning processes. Furthermore, in complex multi-step reasoning tasks, it suffers from problems such as incoherent reasoning logic, missing steps, insufficient accuracy, and non-standard formatting.

[0042] The aforementioned thought chain dataset can be a supervised learning dataset for training the reasoning ability of large language models. It can be labeled with high quality, and each sample can include three parts: question, explicit thought chain, and standard answer. The explicit thought chain can be a sequence of intermediate reasoning steps, which is essentially a step-by-step reasoning process from question to answer. It includes ordered logical deduction steps, conditional judgments, and intermediate conclusions, and is used to convey the reasoning logic of "problem decomposition - step-by-step deduction - conclusion convergence" to the model.

[0043] The embodiments disclosed herein can pre-train a basic large model using a thought chain dataset to obtain a fine-tuned model with basic inference chain generation capabilities, providing a stable starting point for subsequent reinforcement learning training.

[0044] Each sample in the aforementioned reasoning optimization dataset can include a question, a standard long reasoning chain, a standard short reasoning chain, reasoning accuracy annotations, and a standard answer. The difference between the sample composition of the reasoning optimization dataset and the thinking chain dataset is that the samples in the thinking chain dataset only focus on guiding the model to learn the basic logic of step-by-step reasoning; while the samples in the reasoning optimization dataset, in addition to including basic reasoning-related content, additionally include standard long reasoning chains, standard short reasoning chains, and reasoning accuracy annotations, balancing the training needs of long chain standardization and short chain compactness.

[0045] This embodiment of the disclosure, based on the fine-tuned model, utilizes an inference optimization dataset to sequentially perform long-chain reinforcement learning training and short-chain reinforcement learning training on the fine-tuned model, gradually optimizing inference accuracy, format standardization, and inference chain length, ultimately forming a large-scale inference model that meets the requirements. This large-scale inference model is also known as a deep thinking large-scale model.

[0046] Here, long-chain reinforcement learning training (the first stage of reinforcement learning training) is used to guide the fine-tuning model to learn to generate long inference chains that are accurate and formatted correctly. During long-chain reinforcement learning training, the foundation of the fine-tuning model's multi-step reasoning ability is strengthened, and the length of the generated thought chains is gradually increased. This helps the model acquire the reasoning ability required to solve complex problems, addressing the issue of insufficient accuracy in long-chain reasoning in the fine-tuning model and ensuring that the long inference chains can realistically and reliably support the generation of the final answer. Simultaneously, standardized training guidance standardizes the output format of the long inference chains, ensuring that the model-generated long inference chains have a unified step structure and clear logical hierarchy, avoiding problems such as chaotic steps, non-standard expressions, and lack of a unified paradigm. This improves the readability and traceability of the inference chains, making them suitable for professional scenarios such as mathematics and coding, which have high requirements for the standardization of the reasoning process. The generation process of long inference chains is essentially the model's decomposition and step-by-step derivation of complex problems. Through long chain reinforcement learning training, the model can be guided to master the decomposition logic of complex problems, improve the robustness of multi-step inference, and provide core support for the accuracy of long chain compression in subsequent short chain reinforcement learning training, avoiding the problem of missing key inference steps in the subsequent compression process.

[0047] Short-chain reinforcement learning training (the second stage of reinforcement learning training) guides the fine-tuning model to learn how to compress long inference chains and generate shorter ones, provided that the inference is accurate and the format is standardized. In the short-chain reinforcement learning training process, with "accurate inference and standardized format" as the premise, the model is guided to compress long inference chains to generate shorter ones. Its technical effects are synergistic and complementary with long-chain reinforcement learning training, specifically: achieving efficient compression of long inference chains while ensuring inference quality. Unlike simple text truncation or random abbreviation, this short-chain reinforcement learning training can guide the model to accurately identify the core logic, key derivation steps, and necessary intermediate conclusions in long inference chains, eliminating redundant expressions, repetitive derivations, and other irrelevant content, ensuring that the generated short inference chains are both compact and concise, while completely retaining the core reasoning logic supporting the final answer; reducing inference costs and improving inference efficiency. Compared to long inference chains, short inference chains require fewer tokens, consume less computational and memory resources during inference, and are faster. This effectively solves the problems of high inference latency and high resource consumption associated with long-chain inference, balancing inference quality and efficiency, and broadening the application scenarios of large-scale inference models.

[0048] In this embodiment, reinforcement learning training is divided into two stages. The first stage is long-chain reinforcement learning training, also known as the long-chain reasoning foundation stage, which involves increasing the length of generated chains to improve the standardization of thought chain format and the accuracy of reasoning. The second stage is short-chain reinforcement learning training, also known as the long-chain reasoning optimization stage, which involves decreasing the length of generated chains to seek shorter and better reasoning paths. Long-chain reinforcement learning training and short-chain reinforcement learning training proceed sequentially, working synergistically to form an optimization logic of "establishing standards first, then simplifying." The synergistic technical effect is reflected in the fact that both together solve the technical pain points of inaccurate reasoning, non-standard format, redundant and inefficient long chains, or missing key logic in short chains in existing large-scale reasoning models. First, long-chain reinforcement learning training provides a high-quality foundation of long inference chains for short-chain reinforcement learning training, avoiding errors and omissions in short-chain compression. Short-chain reinforcement learning training upgrades efficiency based on long-chain reinforcement learning training, avoiding redundancy and inefficiency. The combination of the two enables the final large-scale inference model to possess both accurate and standardized long-chain inference capabilities and efficient and compact short-chain inference capabilities. This satisfies both the requirements for high interpretability and high accuracy in inference, as well as the deployment requirements for low latency and low resource consumption, ultimately achieving the core goal of long-chain optimization of the large-scale inference model.

[0049] This reasoning model, when generating similar answers, helps to generate shorter reasoning paths, reduce reasoning time, save computing resources, and improve user experience.

[0050] According to the technical solution of the exemplary embodiments of this disclosure, a fine-tuned model obtained by fine-tuning a basic large model through a thought chain dataset and an inference optimization dataset are obtained. Using the inference optimization dataset, the fine-tuned model is sequentially trained with long-chain reinforcement learning and short-chain reinforcement learning to obtain a large inference model. The long-chain reinforcement learning training guides the fine-tuned model to learn to generate long inference chains that are accurate and formatted correctly. The short-chain reinforcement learning training, under the premise of accurate and formatted inference, guides the fine-tuned model to learn to compress long inference chains to generate short inference chains. This hierarchical, progressive optimization, first converting short chains to long chains and then long chains to short chains, ensures both the accuracy and standardization of long inference chains and the efficiency and completeness of short inference chains. This solves the shortcomings of existing optimization methods that, due to their "one-step" approach, struggle to balance inference accuracy and formatted correctness, and suffer from redundant long-chain inference, thereby improving inference efficiency.

[0051] In some embodiments, both long-chain reinforcement learning training and short-chain reinforcement learning training are based on the group relative policy optimization algorithm to perform policy gradient updates. The objective function of the group relative policy optimization algorithm includes a policy update magnitude constraint and a policy distribution constraint. The policy update magnitude constraint is used to constrain the range of policy update changes, and the policy distribution constraint is used to constrain the degree of difference between the policy distribution and the reference policy.

[0052] Specifically, this embodiment employs Group Relative Policy Optimization (GRPO) as the core algorithm for chain reinforcement learning training and short-chain reinforcement learning training to train the model and generate better inference paths. GRPO is a group-based reinforcement learning method that dynamically adjusts policy weights through relative comparisons among a set of candidate policies, thereby achieving efficient optimization. Multiple candidate responses are generated for the same input cue (group sampling), and a normalized advantage value is dynamically calculated based on the statistical characteristics (mean and standard deviation) of the rewards within each group. High-reward samples are assigned a positive advantage, while low-reward samples are suppressed, forming a survival-of-the-fittest optimization mechanism.

[0053] The core idea of ​​GRPO is to dynamically adjust policy parameters by evaluating the relative performance of multiple generated samples. Unlike traditional Proximal Policy Optimization (PPO), GRPO does not require training a separate value network (Critic Model) of similar size to the policy model; instead, it directly uses within-group statistical indicators to replace the baseline values ​​of the value network output. By using the mean and standard deviation of within-group rewards as a dynamic baseline, it avoids reliance on fixed or external evaluation metrics, enhancing adaptability to changes in task reward distribution. The advantage value is calculated using the standardized score (Z-score) of within-group rewards, as shown in the formula: ,in, This represents the advantage value of the i-th generated sample. This represents the in-group reward for the i-th generated sample. This represents the average reward within the group. This represents the standard deviation of the within-group reward, making the dominance estimate robust to changes in reward scaling.

[0054] Through GRPO, the model can progressively optimize the quality and efficiency of its generated content while avoiding getting trapped in local optima. During policy gradient updates, the variation in the probability ratio between the old and new policies is limited to prevent policy collapse caused by excessively large single update steps.

[0055] The objective function of the group relative policy optimization algorithm can be expressed as: (1) Where S represents the objective function of the group relative policy optimization algorithm; Represents the expectation operator; The current policy is represented by the learnable parameter θ, which represents the probability density of choosing action a in state s. This represents the old strategy before the current strategy is updated; A represents the shearing function; A represents the dominant function. β represents the shear coefficient; β represents the penalty coefficient for KL divergence. Indicates KL divergence; Indicates a reference strategy.

[0056] As can be seen from the above formula, the objective function of the group relative policy optimization algorithm can include a policy update magnitude constraint term (the first term on the right side of the equal sign, also known as the shearing coefficient term) and a policy distribution constraint term (the second term on the right side of the equal sign, also known as the KL divergence term). The policy update magnitude constraint term is used to constrain the range of policy update changes, and the policy distribution constraint term is used to constrain the degree of difference between the policy distribution and the reference policy.

[0057] Here, a shear coefficient is introduced into the objective function for optimization. (e.g., 0.2) can constrain policy updates to [1] ,1+ Within the range of [missing information]. A KL divergence term is added to the objective function to force the distribution of the new policy to remain close to that of the reference policy (old policy or SFT model), preventing policy updates from deviating too far.

[0058] This disclosed embodiment employs a group-relative strategy optimization algorithm, enabling the model to autonomously explore shorter and better thought chains through reinforcement learning. Unlike forcibly reducing the length of thought chains through prompts and modifications to the training data, this method has greater potential for optimizing long-chain inference.

[0059] In some embodiments, the inference optimization dataset includes a first-stage dataset and a second-stage dataset; By utilizing the inference optimization dataset, the fine-tuned model is sequentially trained using long-chain reinforcement learning and short-chain reinforcement learning to obtain a large-scale inference model for performing the target inference task, including: The fine-tuned model is trained using long-chain reinforcement learning on the first-stage dataset to obtain an intermediate model for generating long inference chains. If the intermediate model achieves a validation accuracy greater than or equal to the preset validation accuracy on the validation set, the intermediate model is trained using short-chain reinforcement learning on the second-stage dataset to obtain the large inference model.

[0060] Specifically, embodiments of this disclosure can divide the inference optimization dataset into a first-stage dataset and a second-stage dataset according to the different objectives of two-stage reinforcement learning. The first-stage dataset may include questions, standard long inference chains, accuracy annotations for long-chain inference, and standard answers, which are suitable for long-chain reinforcement learning training and guide the model to generate accurate and standardized long inference chains. The second-stage dataset may include questions, standard short inference chains, accuracy annotations for short-chain inference, and standard answers, which are suitable for short-chain reinforcement learning training. Under the premise of accurate and standardized inference, long inference chains are compressed to generate short inference chains.

[0061] The fine-tuned model is trained using long-chain reinforcement learning on the first-stage dataset to obtain an intermediate model for generating long inference chains. If the intermediate model's validation accuracy on the validation set is greater than or equal to a preset validation accuracy, the intermediate model is trained using short-chain reinforcement learning on the second-stage dataset to obtain the large inference model. Here, the preset validation accuracy can be set according to the selection of the validation set, and this embodiment does not specifically limit it. In the method of this embodiment, the preset validation accuracy can be set to 90%.

[0062] After long-chain reinforcement learning training and before short-chain reinforcement learning training, an intermediate model validation step is added. Only when the intermediate model's validation accuracy on the validation set is greater than or equal to the preset validation accuracy will it switch to subsequent short-chain reinforcement learning training. This ensures that compression optimization is performed only after the long-chain inference capability meets the standards. On one hand, this optimization method avoids using intermediate models with insufficient long-chain inference accuracy or non-standard formats for short-chain training, preventing short-chain compression based on substandard long chains, which could lead to inaccurate short-chain inference and logical gaps. This ensures the quality foundation of short-chain reinforcement learning training from the source. On the other hand, if the intermediate model does not reach the preset validation accuracy, it can return to the long-chain reinforcement learning training stage for iterative optimization, avoiding the waste of computational resources and prolonged training cycle caused by ineffective entry into the short-chain reinforcement learning training stage. This achieves closed-loop control of the training process, improving the economy and efficiency of training.

[0063] In some embodiments, the long-chain reward function used in long-chain reinforcement learning training includes an accuracy reward and a format reward. Among them, the accuracy reward item is used to evaluate the accuracy of the reasoning response content, and the format reward item is used to guide the reasoning process to conform to the preset structured format.

[0064] Specifically, the reinforcement learning strategy used in long-chain reinforcement learning training can be a long-chain reward function, which can include accuracy reward items and format reward items.

[0065] The accuracy reward is used to evaluate the accuracy of the inference response content. The inference response content, also known as the generated content, can be a long chain of inference output by the model trained through long-chain reinforcement learning (i.e., the intermediate model mentioned earlier). The accuracy reward can use an accuracy reward model to evaluate whether the inference response content is correct. For example, for mathematical problems with deterministic results, the model is required to provide the final answer in a specified format (e.g., within a box), thereby achieving reliable rule-based correctness verification. Similarly, for code problems, a compiler can be used to generate feedback based on predefined test cases.

[0066] Format-based rewards are used to guide the reasoning process to conform to a preset structured format. The format-based reward model forces the model to place the thought process within a predetermined structured format. <think>"and"< / think> Between tags.

[0067] The long-chain reward function used in long-chain reinforcement learning training can be: (2) in, This represents the reward value for the first stage; This represents the accuracy bonus value, with a recommended value of 1. The formatted reward value is 1; Indicates accuracy weight; Indicates format weight, and The recommended value for all of them is 1.

[0068] In some embodiments, the short-chain reward function used in short-chain reinforcement learning training includes an accuracy reward and a format reward, as well as a conciseness reward, an abnormal loop reward, and an answer format reward; Among them, the conciseness reward is used to encourage the generation of shorter reasoning chains, provided that the reasoning process is correct; The abnormal loop reward is used to suppress abnormal repetitive patterns in the inference response content; The answer format reward is used to guide reasoning responses to conform to the format specifications of the reasoning task.

[0069] Specifically, in the short-chain reinforcement learning training phase, the inference response content can be a short inference chain output by the model trained through short-chain reinforcement learning (i.e., the intermediate model mentioned earlier after short-chain reinforcement learning training). To shorten the inference response content, three innovative reward mechanisms are introduced in addition to the accuracy and format rewards. This allows the second phase to further optimize existing repetitive loops, optimize the length of inference paths, optimize the format of the final answer, and continuously seek shorter and better inference paths, thereby optimizing the content of the thought chain and reducing its length.

[0070] The short-chain reward function used in short-chain reinforcement learning training can include accuracy rewards, format rewards, conciseness rewards, abnormal loop rewards, and answer format rewards.

[0071] The short-chain reward function used in short-chain reinforcement learning training can be: (3) in, This represents the reward value for the second stage; Indicates a conciseness-based reward; This indicates a reward for the correct answer format; the recommended value is 1. This indicates an abnormal cyclical reward item; Indicates the weight of simplicity; Indicates the weight of the answer format; Indicates the weight of abnormal cycles; , and The recommended value for all values ​​is 0.3.

[0072] Here, the simplicity reward, also known as the shortest path reward, is used to encourage the generation of shorter inference chains, provided the reasoning process is correct. This reward can be weighted to enhance shorter, correct inference paths. For different generated content of the same sample, if it meets the correctness condition, it is assumed to be the shortest correct path, and a reward is given for that sample.

[0073] Preferably, the weighting coefficient is set as follows: , The range is set to [1, 5]. A course-based learning approach is adopted, gradually increasing the weight of the coefficients. Let the length of the shortest correct path within the group be... The longest correct path is , < If the length of the current correct sample is L, then: (4) in, Gradually increase from 1 to 5. The learning process is controlled by a course, and improvements are made slowly with each iteration.

[0074] By using a conciseness reward, it is easier to generate shorter reasoning paths, reduce reasoning time, save computing resources, and improve user experience when the generated answers are similar.

[0075] Abnormal cyclic reward items can be used to suppress anomalous repetition patterns in inference response content. This disclosure proposes a text duplication detection method based on sliding window hash fingerprinting to optimize generative text quality, specifically addressing abnormal cyclic reward items. Preferably, its technical features include: using fixed-length characters... The window preprocesses the text stream, calculating the fingerprint value of the character sequence within the window in real time using a rolling hash algorithm; a dynamic hash mapping table is established to record the frequency of fingerprint occurrences, and when the repetition frequency of a specific fingerprint reaches a preset threshold... When a character segment is identified as constituting a cyclic repetition pattern, an exponential penalty factor is introduced into the text generation quality assessment model based on the ratio of the total length of the repeated segments to the total text length. This factor, through a dynamic weight adjustment mechanism, suppresses the generation of low-information-content repetitive content, thereby improving the semantic density and information effectiveness of the text. A penalty is imposed on a large number of repetitive, cyclical content. The character length is repeated m times, which is optional. When m=3, =100, =3. This is the number of times the test is repeated. The calculation formula is as follows: =-min(2, (5) By using abnormal loop reward items, the probability of large model generation content getting stuck in repeated loops can be reduced, the generation quality of models can be improved, the waste of inference model generation can be reduced, computing resources can be saved, and the user experience can be improved.

[0076] Answer format rewards are used to guide reasoning responses to conform to the format specifications of the reasoning task. Reasoning tasks can be categorized by type, including but not limited to mathematical tasks, coding tasks, and multidisciplinary question-answering tasks. Rewards are based on whether the format of the answer can be extracted from the current training code answer. Format detection uses a dual verification mechanism of regular expression matching (e.g., r"\boxed{}") and keyword localization; a successful match triggers the format reward.

[0077] Preferably, the answer format reinforcement reward mechanism can be formally defined as the following reward function: (6) in, The weighting coefficients represent the mathematical task. Represents the weight coefficients of the code task; This represents the weighting coefficients for multidisciplinary question-answering tasks.

[0078] Different answer format reward mechanisms are set for different reasoning tasks, which helps to standardize the format of large-scale model reasoning and reduces the inaccuracy in evaluating the model's reasoning ability due to format issues. Specific details are described below: For mathematical tasks, a reward will be given for any description of the final answer, such as boxed {}. Mathematical answers must include LaTeX formatted tags, with the standard answer containing a boxed expression or explicitly marked "final answer is" followed by a number / expression.

[0079] For code tasks, a reward is given if a code block description exists and can be extracted. The complete code is contained within a language-marked block (e.g., Python / Java) or verified via an assert statement to validate function output. For example, the format is "codeblock".

[0080] For multidisciplinary question-and-answer tasks, the system also checks whether the generated content meets the required format, such as physical quantity values ​​needing to include unit labels. If the corresponding format is found, a reward is given.

[0081] Based on this, this implementation takes training efficiency into account and prioritizes various reward objectives, allocating different reward weights and reward stages. In the first stage, only accuracy and format rewards are used. In the second stage, other innovative reward modules (simplicity reward, abnormal loop reward, and answer format reward) are added, and the weight of some reward modules is gradually increased. This learning approach allows the model to first acquire the ability to form highly accurate reasoning chains, and then, while maintaining accuracy, seek shorter reasoning paths and better generated content.

[0082] In the course learning, the first step can be to divide the learning into stages to establish macro goals. For details, please refer to the previous article on switching from long chain reinforcement learning training to short chain reinforcement learning training by setting the verification accuracy. This will not be repeated here.

[0083] Secondly, by using a dynamic signal-guided micro-optimization mechanism, the weighting coefficient of the simplicity reward item (shorter path reward) is increased from small to large. Under the premise that the training model can deduce the correct answer, the exploration of the shorter path is gradually increased.

[0084] Based on this, the method provided in this embodiment may further include: during the short-chain reinforcement learning training process, using a course learning strategy to dynamically adjust the increase of the weighting coefficient of the simplicity reward item, so that the simplicity reward item increases adaptively with the training process.

[0085] Specifically, the progressive reinforcement of the simplicity reward item employs a three-stage approach—exploration, reinforcement, and convergence—to adjust its weights. The weighting coefficients for the simplicity reward item... Gradually increase from 1 to 5. During the exploration phase (0- step) Each step increase To avoid reward mutations; during the enhancement period ( -3 The increase (step) doubled to 2 Accelerate path compression; during the convergence period (3 (After the step) the increase decreased to 0.5 Fine-tuning. Combining formula (3), it can be seen that the simplicity reward also adaptively increases with the increase of the weighting coefficient. Here, the increase... The specific value can be set according to actual needs, and this embodiment does not impose specific limitations on it.

[0086] Finally, the course also incorporates an adaptive constraint mechanism to ensure process stability. The first phase strengthens the policy distribution constraint term, and subsequently, β is continuously reduced while monitoring for policy divergence risk. When KL divergence risk is detected, β is immediately increased.

[0087] Preferably, in the first stage, β=0.5; when When this occurs, β is temporarily restored to 1.2 times the value of the previous stage.

[0088] This course employs a three-tiered learning approach: establishing macro-level goals through phased segmentation, guiding micro-level optimization with dynamic signals, and ensuring process stability through adaptive constraints. This collaborative optimization enables the transfer of generation capabilities from "accurate but verbose" to "refined and correct."

[0089] Furthermore, by combining short and long training processes with course learning, the weights of the reward module are adjusted to gradually improve its capabilities and reduce the optimization difficulty of the model. At the same time, other reward modules are maintained to preserve the model's original capabilities. By reducing the optimization difficulty, the generation quality of the model is further improved.

[0090] Based on this, the method proposed in this disclosure solves the relevant problems in the prior art, specifically in the following ways: 1. Reinforcement Learning Architecture. Unlike other Long to Short Reasoning solutions, this embodiment uses a two-stage reinforcement learning training method to enable the model to autonomously explore and learn shorter and better thought chains. This design breaks through the upper limit of traditional methods and significantly improves the model's learning potential and reasoning efficiency.

[0091] 2. Short-to-Long Training Process. The first stage, long-chain reinforcement learning, gradually increases the length of the generated thought chains through mechanisms such as group relative policy optimization, accuracy rewards, and format rewards, helping the model acquire the reasoning ability needed to solve complex problems. The second stage, short-chain reinforcement learning, introduces innovation rule rewards, prompting the model to seek the shortest path (simplicity reward). Simultaneously, it combines anomaly penalty (abnormal loop reward) and answer format reward mechanisms to further optimize generated content, reduce abnormal outputs, and standardize answer format. This process enables the model to autonomously explore shorter reasoning chains, thereby improving reasoning efficiency.

[0092] 3. Innovative Rule Reward Module. Relative Reward for Shorter Inference Paths (Simplicity Reward): Introducing a relative reward module encourages the model to use shorter inference paths to solve problems when performance is similar. Through reinforcement learning, the model can autonomously explore shorter inference paths, thus achieving inference optimization from long to short. Abnormal Content Penalty (Abnormal Loop Reward): Introducing a penalty module for abnormal content generated by the large model reduces the probability of the model entering abnormal states such as repeated loops. This mechanism significantly improves model performance during the inference deployment phase, reduces inference time, and improves computational efficiency. Answer Format Reinforcement Reward (Answer Format Reward): Introducing an answer format reinforcement reward module for different tasks further standardizes the format of the final output answer, reducing the possibility of poor evaluation results due to format deviations.

[0093] 4. Learning Strategy Based on Reward Weights. By gradually adjusting the weights of some reward modules (simplicity reward items), the model's capabilities are progressively improved from weak to strong, reducing optimization difficulty. During this process, other reward modules remain active, ensuring the model's original capabilities are unaffected.

[0094] The foregoing mainly describes the solutions provided by the embodiments of this disclosure. It is understood that, in order to achieve the above functions, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0095] This disclosure embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0096] By dividing each functional module according to its corresponding function, an exemplary embodiment of this disclosure provides a long chain optimization device for a large inference model based on reinforcement learning. This long chain optimization device for a large inference model based on reinforcement learning can be a terminal or a chip applied to a terminal. Figure 2 A schematic diagram of the structure of a long-chain optimization device for large inference models based on reinforcement learning, provided in an exemplary embodiment of this disclosure, is shown. Figure 2 As shown, the device 200 includes: The acquisition module 201 is used to acquire the fine-tuned model obtained by fine-tuning the basic large model through the MindChain dataset, as well as the inference optimization dataset; Training module 202 is used to perform long-chain reinforcement learning training and short-chain reinforcement learning training on the fine-tuning model sequentially using the inference optimization dataset to obtain a large inference model; wherein, the long-chain reinforcement learning training is used to guide the fine-tuning model to learn to generate long inference chains that are accurate and have a standard format; the short-chain reinforcement learning training is used to guide the fine-tuning model to learn to compress the long inference chains to generate short inference chains, provided that the inference is accurate and has a standard format.

[0097] In some embodiments, the inference optimization dataset includes a first-stage dataset and a second-stage dataset; The training module 202 is also used to train the fine-tuned model using the first stage dataset to perform long-chain reinforcement learning training, so as to obtain an intermediate model for generating the long inference chain. If the intermediate model achieves a validation accuracy greater than or equal to a preset validation accuracy on the validation set, the intermediate model is trained using the second-stage dataset through short-chain reinforcement learning to obtain a large inference model.

[0098] In some embodiments, the long-chain reward function used in the long-chain reinforcement learning training includes an accuracy reward and a format reward. The accuracy reward item is used to evaluate the accuracy of the reasoning response content, and the format reward item is used to guide the reasoning process to conform to a preset structured format.

[0099] In some embodiments, the short-chain reward function used in the short-chain reinforcement learning training includes the accuracy reward item, the format reward item, the conciseness reward item, the abnormal loop reward item, and the answer format reward item; The simplicity reward is used to encourage the generation of shorter reasoning chains, provided that the reasoning process is correct. The abnormal cycle reward item is used to suppress abnormal repetitive patterns in the reasoning response content; The answer format reward is used to guide reasoning responses to conform to the format specifications of the reasoning task.

[0100] In some embodiments, the training module 202 is further configured to dynamically adjust the increase rate of the simplicity weight of the simplicity reward item by adopting a course learning strategy during the short chain reinforcement learning training process, so that the simplicity weight increases adaptively with the training process.

[0101] In some embodiments, both the long-chain reinforcement learning training and the short-chain reinforcement learning training are based on the group relative policy optimization algorithm to perform policy gradient updates. The objective function of the group relative policy optimization algorithm includes a policy update magnitude constraint term and a policy distribution constraint term. The policy update magnitude constraint term is used to constrain the range of policy update changes, and the policy distribution constraint term is used to constrain the degree of difference between the policy distribution and the reference policy.

[0102] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the methods disclosed in this disclosure.

[0103] Figure 3 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown. For example... Figure 3 As shown, the electronic device 300 includes at least one processor 301 and a memory 302 coupled to the processor 301, which can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0104] The processor 301 described above can also be called a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 301 or by software instructions. The processor 301 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 302, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 301 reads information from the memory 302 and, in conjunction with its hardware, completes the steps of the method described above.

[0105] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 4 The computer system 400 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those described above. Figure 4 A schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present disclosure is shown.

[0106] Computer system 400 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0107] like Figure 4As shown, the computer system 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the computer system 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0108] Multiple components in the computer system 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device capable of inputting information into the computer system 400. The input unit 406 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 408 may include, but is not limited to, a hard disk and an optical disk. The communication unit 409 allows the computer system 400 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ device, WiFi device, WiMax device, cellular communication device, and / or the like.

[0109] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0110] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0111] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0113] This disclosure also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the methods disclosed in the embodiments of this disclosure.

[0114] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0117] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0118] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0119] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for optimizing long chains of large inference models based on reinforcement learning, characterized in that, include: Obtain the fine-tuned model obtained by fine-tuning the basic large model using the MindChain dataset, as well as the inference optimization dataset; Using the aforementioned inference optimization dataset, the fine-tuned model is sequentially trained using long-chain reinforcement learning and short-chain reinforcement learning to obtain a large inference model. The long-chain reinforcement learning training is used to guide the fine-tuning model to learn to generate long inference chains that are accurate and formatted correctly; the short-chain reinforcement learning training is used to guide the fine-tuning model to learn to compress the long inference chains to generate short inference chains, provided that the inference is accurate and formatted correctly.

2. The method as described in claim 1, characterized in that, The inference optimization dataset includes a first-stage dataset and a second-stage dataset; The process involves using the inference optimization dataset to sequentially train the fine-tuned model with long-chain reinforcement learning and short-chain reinforcement learning to obtain a large-scale inference model for performing the target inference task, including: The fine-tuned model is trained using the first-stage dataset to obtain an intermediate model for generating the long inference chain. If the intermediate model achieves a validation accuracy greater than or equal to a preset validation accuracy on the validation set, the intermediate model is trained using the second-stage dataset through short-chain reinforcement learning to obtain a large inference model.

3. The method as described in claim 1, characterized in that, The long-chain reward function used in the long-chain reinforcement learning training includes an accuracy reward item and a format reward item; The accuracy reward item is used to evaluate the accuracy of the reasoning response content, and the format reward item is used to guide the reasoning process to conform to a preset structured format.

4. The method as described in claim 3, characterized in that, The short-chain reinforcement learning training uses a short-chain reward function that includes the accuracy reward, the format reward, the conciseness reward, the abnormal loop reward, and the answer format reward. The simplicity reward is used to encourage the generation of shorter reasoning chains, provided that the reasoning process is correct. The abnormal cycle reward item is used to suppress abnormal repetitive patterns in the reasoning response content; The answer format reward is used to guide reasoning responses to conform to the format specifications of the reasoning task.

5. The method as described in claim 4, characterized in that, The method further includes: During the short-chain reinforcement learning training process, a course learning strategy is used to dynamically adjust the increase rate of the simplicity weight of the simplicity reward item, so that the simplicity weight increases adaptively with the training process.

6. The method according to any one of claims 1 to 5, characterized in that, Both the long-chain reinforcement learning training and the short-chain reinforcement learning training are based on the group relative policy optimization algorithm to perform policy gradient updates. The objective function of the group relative policy optimization algorithm includes a policy update magnitude constraint term and a policy distribution constraint term. The policy update magnitude constraint term is used to constrain the range of policy update changes, and the policy distribution constraint term is used to constrain the degree of difference between the policy distribution and the reference policy.

7. A long-chain optimization device for large inference models based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire the fine-tuned model obtained by fine-tuning the basic large model through the MindChain dataset, as well as the inference optimization dataset; The training module is used to perform long-chain reinforcement learning training and short-chain reinforcement learning training on the fine-tuned model sequentially using the inference optimization dataset to obtain the large inference model. The long-chain reinforcement learning training is used to guide the fine-tuning model to learn to generate long inference chains that are accurate and formatted correctly; the short-chain reinforcement learning training is used to guide the fine-tuning model to learn to compress the long inference chains to generate short inference chains, provided that the inference is accurate and formatted correctly.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 6.

Images