Model training method and platform, text reasoning method and platform

Abstract

The embodiment of the specification provides a model training method and platform, and a text reasoning method and platform, wherein the model training method comprises the following steps: obtaining an initial sample set, a student reasoning model and a reasoning ability label of the student reasoning model, wherein the initial sample set comprises a plurality of sample thinking chains, the plurality of sample thinking chains are generated by using different reasoning methods by a teacher reasoning model, each sample thinking chain is labeled with a reasoning method label of a corresponding reasoning method, a sampling weight of the plurality of sample thinking chains is determined based on a deviation between the reasoning ability label of the student reasoning model and the reasoning method labels labeled by the plurality of sample thinking chains; a target sample set is obtained by sampling from the initial sample set based on the sampling weight; and the student reasoning model is trained based on the target sample set to obtain a target reasoning model. The target reasoning model obtained by the training method can dynamically adjust the reasoning process according to the complexity of the problem, and can ensure the reasoning accuracy while improving the reasoning efficiency.

Description

Model training methods and platforms, text reasoning methods and platforms Technical Field

[0001] The embodiments in this specification relate to the fields of artificial intelligence technology and machine learning technology, and in particular to a model training method, a text reasoning method, a model training platform, and a text reasoning platform. Background Technology

[0002] With the rapid development of artificial intelligence and machine learning technologies, large language models have made significant progress in reasoning ability, which has become a core element in improving the practical value and application effect of reasoning models.

[0003] Currently, inference models with large-scale model parameters have good problem reasoning capabilities, but their required storage and computation costs are also very high. Therefore, when inference resources are limited, problem reasoning can only be performed using inference models with smaller-scale model parameters. In existing technologies, the training process for smaller-scale inference models can be improved by collecting the thought chains generated by larger-scale inference models during the reasoning process as training data.

[0004] However, since large-scale and small-scale reasoning models have different model parameters, their cognitive or reasoning abilities also differ. Directly using the thought processes collected from large-scale reasoning models for training small-scale models, without considering the reasoning abilities and cognitive biases between models of different sizes, can negatively impact the training of small-scale models. This can lead to overthinking or underthinking in actual reasoning processes, affecting the efficiency and accuracy of problem-solving. Therefore, a more reasonable model training method is urgently needed. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a model training method. One or more embodiments of this specification also relate to a text reasoning method, a model training platform, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a model training method is provided, comprising:

[0007] Obtain an initial sample set, a student reasoning model, and reasoning ability labels for the student reasoning model. The initial sample set includes multiple sample thought chains, which are generated by the teacher reasoning model using different reasoning methods. Each sample thought chain is labeled with a reasoning method label corresponding to the reasoning method.

[0008] Based on the deviation between the reasoning ability label of the student reasoning model and the reasoning method label of multiple sample thinking chains, the sampling weight of multiple sample thinking chains is determined.

[0009] Based on the sampling weights, the target sample set is obtained by sampling from the initial sample set;

[0010] Based on the target sample set, train the student reasoning model to obtain the target reasoning model.

[0011] According to a second aspect of the embodiments of this specification, a text reasoning method is provided, comprising:

[0012] Obtain the target question text;

[0013] The target question text is input into the target reasoning model to obtain the target answer text. The target reasoning model is trained based on the model training method described above.

[0014] According to a third aspect of the embodiments of this specification, a model training platform is provided, including a training interface and a first response unit:

[0015] The training interface is used to receive model training requests;

[0016] The first response unit, in response to a model training request, executes the model training method as described above.

[0017] According to a fourth aspect of the embodiments of this specification, a text reasoning platform is provided, including a reasoning interface and a second response unit:

[0018] The inference interface is used to receive text inference requests;

[0019] The second response unit, in response to the text reasoning request, obtains the target question text; inputs the target question text into the target reasoning model to obtain the target answer text, wherein the target reasoning model is trained based on the model training method described above.

[0020] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0021] Memory and processor;

[0022] The memory is used to store computer-executable instructions, and the processor is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0023] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0024] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0025] One embodiment of this specification implements a model training method, comprising: obtaining an initial sample set, a student reasoning model, and reasoning ability labels of the student reasoning model, wherein the initial sample set includes multiple sample thought chains, which are generated by the teacher reasoning model using different reasoning methods, and each sample thought chain is labeled with a reasoning method label corresponding to the reasoning method; determining the sampling weights of the multiple sample thought chains based on the deviation between the reasoning ability labels of the student reasoning model and the reasoning method labels labeled on the multiple sample thought chains; obtaining a target sample set by sampling from the initial sample set based on the sampling weights; and training the student reasoning model based on the target sample set to obtain the target reasoning model.

[0026] By acquiring an initial sample set, a student reasoning model, and its reasoning ability labels, and based on the deviation between the student reasoning model's reasoning ability labels and the reasoning method labels of each sample's thought chain in the initial sample set, the sampling weights for each sample's thought chain are dynamically determined. This allows for the selection of a training sample set that matches the student reasoning model's cognitive or reasoning ability, and the student reasoning model is then trained. This enables the student reasoning model to learn reasoning methods adapted to its current cognitive level from the teacher's reasoning model during training, avoiding training bias caused by a mismatch between cognitive or reasoning abilities. The target reasoning model trained using this method can dynamically adjust its reasoning process according to the complexity of the actual problem, employing concise reasoning paths for simple problems and detailed reasoning paths for complex problems. This ensures reasoning accuracy while improving reasoning efficiency and achieving rational utilization of computing resources. Attached Figure Description

[0027] Figure 1 is a flowchart of a model training method provided in one embodiment of this specification;

[0028] Figure 2 is a schematic diagram of the framework of a model training method provided in one embodiment of this specification;

[0029] Figure 3 is a flowchart of a text reasoning method provided in one embodiment of this specification;

[0030] Figure 4 is a schematic diagram of the structure of a model training platform provided in one embodiment of this specification;

[0031] Figure 5 is a schematic diagram of the structure of a text reasoning platform provided in one embodiment of this specification;

[0032] Figure 6 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0033] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0034] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0036] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0037] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0038] Large Language Model (LLM) is a natural language processing model based on deep neural networks. It can understand and generate human language by training on large-scale text data and has powerful context awareness, reasoning and multi-task generalization capabilities.

[0039] Chain of Thought (CoT) is a technique that guides large language models through step-by-step logical reasoning. It improves the accuracy and interpretability of solving complex problems by explicitly outputting intermediate reasoning steps, and it shows significant advantages, especially in mathematical derivation and logical judgment.

[0040] Supervised fine-tuning (SFT) is a supervised process of adjusting the parameters of a pre-trained language model using labeled data to adapt it to a specific task or domain. It can serve as one of the fundamental training stages for building high-quality dialogue systems.

[0041] Reinforcement learning from human feedback (RLHF or RL) is the process of constructing a reward function using human preference signals and then using reinforcement learning algorithms to optimize the output strategy of a language model, enabling the model to generate more coherent and useful answers that align with human values.

[0042] Cognitive Difficulty (CD) is used to measure the comprehensive complexity of the breadth of knowledge, logical depth, and level of abstraction required by a model for a problem or task. In this specification, it is used to reflect the cognitive difficulty that a thought chain can be generated, that is, the difficulty of the reasoning method used in the reasoning process of generating the thought chain.

[0043] Reasoning Verbosity (RV) indicates the level of detail in a reasoning model's reasoning process for a problem. It can be used to reflect the complexity of the thought chain generated during the reasoning process. A high RV means that the model exhibits more intermediate thinking steps, hypothesis analysis, and logical connections, which helps to improve transparency and credibility.

[0044] The GRPO algorithm (Group Relative Policy Optimization) is a reinforcement learning policy optimization method oriented towards group preference modeling. By modeling the relative ranking relationship between multiple candidate responses, it improves the consistency and robustness of the model's decisions under diverse preferences.

[0045] Proximal Policy Optimization (PPO) is a stable and efficient policy gradient reinforcement learning algorithm. It introduces a pruning mechanism to limit the update step size, thereby avoiding policy collapse during training. It is widely used in fine-tuning processes based on human feedback, such as LLM.

[0046] Diverse Action Policy Optimization (DAPO) is a reinforcement learning framework that encourages policies to explore diverse output behaviors. It increases the diversity of generated results while ensuring performance and is suitable for tasks that require rich expression, such as creative writing and open-ended question answering.

[0047] Generalized State Policy Optimization (GSPO) is a novel reinforcement learning architecture that extends the traditional policy optimization paradigm. It extends the state space of language models from latent vectors to semantically structured representations (such as mind trees and knowledge graphs), supporting more complex environment modeling and long-term planning capabilities.

[0048] Prompt engineering is a technical practice that designs and optimizes input prompts to unleash the potential of large language models. It covers methods such as instruction construction, few-shot example injection, and role setting, and directly affects the quality of model output and task adaptability.

[0049] The Large Model-as-a-Judge (LLM-as-a-Judge) paradigm is a methodology that uses a large language model to evaluate the output quality of other models. By constructing standardized scoring criteria and comparative judgment templates, it achieves an efficient and low-cost automated evaluation system, which is widely used in model iteration and leaderboard construction.

[0050] With the rapid development of artificial intelligence and machine learning technologies, large language models have made significant progress in reasoning ability, which has become a core element in improving the practical value and application effect of reasoning models.

[0051] Currently, inference models with large-scale model parameters have good problem reasoning capabilities, but their required storage and computation costs are also very high. Therefore, when inference resources are limited, problem reasoning can only be performed using inference models with smaller-scale model parameters. In existing technologies, the training process for smaller-scale inference models can be improved by collecting the thought chains generated by larger-scale inference models during the reasoning process as training data.

[0052] However, since the model parameters of larger-scale reasoning models and smaller-scale reasoning models are different, there are also differences in their cognitive or reasoning abilities. Directly using the thought chains collected from larger-scale reasoning models for training smaller-scale reasoning models does not take into account the reasoning abilities and cognitive biases between reasoning models of different sizes. This affects the training of smaller-scale reasoning models, making it easy for the trained reasoning models to overthink or underthink in actual reasoning processes, thus affecting the efficiency and accuracy of problem reasoning.

[0053] In view of this, this specification provides a model training method, and also relates to a text reasoning method, a model training platform, a text reasoning platform, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0054] Referring to Figure 1, Figure 1 shows a flowchart of a model training method provided according to an embodiment of this specification, which specifically includes the following steps.

[0055] Step 102: Obtain the initial sample set, the student reasoning model, and the reasoning ability label of the student reasoning model. The initial sample set includes multiple sample thought chains, which are generated by the teacher reasoning model using different reasoning methods. Each sample thought chain is labeled with the reasoning method label of the corresponding reasoning method.

[0056] The model training method provided in the embodiments of this specification can be mainly applied to application scenarios that require high-performance inference models, especially those with limited inference resources, such as mobile devices, embedded systems, or edge computing devices, where efficient inference needs to be achieved with low computing resources. The applicable application platforms can include various artificial intelligence inference services, such as online inference services, mobile application backend services, or inference engines for embedded devices.

[0057] The initial sample set is a dataset containing multiple sample thought chains. It serves as the initial data for training the student's reasoning model and requires further sampling to form the target sample set for training the student's reasoning model. The multiple sample thought chains included in the initial sample set are generated by the teacher's reasoning model based on different reasoning methods. They can cover reasoning processes using various methods, ranging from relatively simple to complex tasks. For example, a sample thought chain for the simple addition problem "1+2=?" might include the brief reasoning process of "calculate 1 plus 2 equals 3," where the reasoning method used is simple addition, i.e., a single-step reasoning. A sample thought chain for the more complex algebraic problem "solve x²-5x+6=0" might include a detailed reasoning process such as "first, we need to determine the corresponding algebraic solution, such as using factorization, which decomposes the equation into (x-2)(x-3)=0, then we get x=2 or x=3...", which could include a sequence of multiple reasoning steps.

[0058] A student inference model is a reasoning model to be trained. Student inference models typically have a small parameter size and correspondingly lower cognitive or reasoning abilities, and are usually used in resource-constrained scenarios, such as mobile devices or embedded systems. Specifically, student inference models can gradually improve their reasoning abilities through subsequent training processes, eventually obtaining a fully trained target inference model. Student inference models can include reasoning ability labels to reflect the level of cognitive or reasoning ability corresponding to the reasoning methods used in the current reasoning process. For example, a reasoning model with a parameter size of 7B can be selected as a student inference model, and its corresponding reasoning ability label can be 5, indicating that the inference model can handle moderately difficult reasoning tasks using the corresponding reasoning methods.

[0059] Reasoning ability tags are standardized identifiers used to quantify the level of a model's reasoning ability. They reflect the cognitive or reasoning capabilities of the reasoning methods a model can use during the reasoning process. These tags can be categorized into levels; higher levels indicate that the model can employ more complex reasoning tools or logical strategies to solve more cognitively challenging reasoning problems. Specifically, reasoning ability tags can be divided into 10 sub-levels (0-9), and adjacent sub-levels can be combined into 5 major levels (0-1, 2-3, etc.). Different major levels correspond to different reasoning or cognitive abilities.

[0060] Specifically, the higher the reasoning or cognitive ability of a reasoning model, the higher the cognitive difficulty and complexity of the thought chain it generates during the reasoning process. That is, there is a corresponding correlation between the reasoning ability label of a reasoning model and the reasoning method label marked on the thought chain generated by the model during the reasoning process.

[0061] Optionally, the reasoning ability label of the reasoning model can be determined comprehensively based on the reasoning process and output results of the reasoning model. Specifically, various methods can be used to evaluate and determine the reasoning ability of the reasoning model, such as indicators like reasoning accuracy, time to solution, and logical consistency.

[0062] Reasoning methods are the specific reasoning tools or logical strategies actually applied by the reasoning model in the process of generating the thought chain. They can be labeled in the thought chain using reasoning method tags. Specifically, reasoning methods can include mathematical formulas, algorithmic tools, or logical derivation rules. For example, in reasoning the problem "to solve x²-5x+6=0", the reasoning method used in the reasoning process of using factorization is "factorization".

[0063] Reasoning method tags are standardized identifiers used to identify the specific reasoning tools or strategies employed in the generation of a thought chain. They reflect the cognitive difficulty of the thought chain. Reasoning method tags can be categorized into levels; the higher the level, the more difficult the reasoning methods within the thought chain, and the more difficult the thought chain is to understand and recognize. Specifically, reasoning ability tags can be graded based on the cognitive difficulty of the thought chain, dividing it into 10 sub-levels (0-9). Adjacent sub-levels are combined into 5 major levels (0-1, 2-3, etc.), with different major levels corresponding to different cognitive difficulties. A detailed classification of reasoning method tags can be found in Table 1, which serves as a basis for such classification.

[0064] Table 1: Criteria for Classifying Reasoning Ability Tags

[0065]

[0066] Teacher reasoning models are pre-trained reasoning models. They typically have a large parameter scale and correspondingly high cognitive or reasoning abilities, enabling them to handle complex reasoning tasks and generate high-quality thought processes. Specifically, teacher reasoning models can be used to generate source models for initial sample sets, providing sample thought processes as training data to train student reasoning models, transferring the advanced reasoning knowledge from the teacher model to the student model. For example, a reasoning model with a parameter scale of 32 bytes can be selected as the teacher reasoning model, with a corresponding reasoning ability label of 8, indicating that the model can handle various reasoning processes ranging from simple basic problems to complex, highly abstract problems, thus providing rich training data for the student reasoning model.

[0067] Sample thought chains are reasoning process data generated by the teacher's reasoning model when solving a specific problem. A sample thought chain can include a complete thought path from problem understanding, step derivation, to determining the answer. Specifically, sample thought chains can be included in an initial sample set and used as training data for the student's reasoning model.

[0068] Specifically, each sample thought chain can be labeled with a corresponding reasoning method tag, which represents the reasoning method used by the teacher in the reasoning model process. The reasoning method tag measures the difficulty of recognizing the sample thought chain, and can be determined by the difficulty of the reasoning tools, concepts, and strategies used in the sample thought chain. For example, for the problem "Calculate 2 × 3 + 4 + ?", the corresponding sample thought chain could include a sequence of reasoning steps such as "This is a relatively simple multi-step mathematical calculation problem. First, we should calculate multiplication 2 multiplied by 3 equals 6, then calculate addition 6 plus 4 equals 10." Therefore, the reasoning method tag for this sample thought chain could be 2-3.

[0069] Optionally, the initial sample set may include not only multiple sample thought chains, but also sample question texts corresponding to the multiple sample thought chains, thereby providing model input during the training process of the student reasoning model.

[0070] Optionally, in addition to labeling the reasoning method, multiple sample thought chains can also be labeled with a reasoning complexity label to measure the reasoning completeness of each sample thought chain. Specifically, there is a correlation between the reasoning completeness of each sample thought chain and the cognitive difficulty; the higher the cognitive difficulty of the sample thought chain, the higher the reasoning completeness generally is.

[0071] Optionally, the reasoning method labels corresponding to the reasoning methods annotated in the sample thought chain annotation can be manually labeled or labeled by a pre-trained labeling model. The labeling model is a pre-trained evaluation model, which can be a natural language model or other types of text processing models. The labeling model can be trained on a set of thought chains with labeled reasoning method tags and can understand the mapping relationship between the sequence length of the reasoning steps in the thought chain and the reasoning method labels. Specifically, the labeling model can annotate the sample thought chains using different annotation methods, such as the "Prompt Engineering" and "LLM-as-Judge" paradigms.

[0072] Optionally, the labeling model can also label the reasoning complexity of the sample's thought chain. Specifically, this can be determined by judging whether the correlation between the reasoning exhaustion of the sample's thought chain and the cognitive difficulty matches.

[0073] In actual training, an initial sample set can be constructed by acquiring multiple sample thought chains generated during the reasoning process of the teacher's reasoning model using different reasoning methods. This initial sample set can be obtained from storage media such as local storage or cloud storage, or it can be generated in real-time by the teacher's reasoning model; this specification does not specifically limit the specific methods used in this embodiment.

[0074] In this step, by obtaining an initial sample set containing multiple sample thought chains, and with each sample thought chain labeled with a corresponding reasoning method label, the subsequent training process for the student reasoning model can sample the sample thought chains based on the reasoning method labels to determine the target sample set. This ensures that the student reasoning model can learn reasoning methods that match its own cognitive or reasoning abilities, thereby avoiding cognitive biases caused by directly using the thought chains of a large-scale teacher reasoning model for training, and improving training efficiency and model reasoning ability.

[0075] Step 104: Determine the sampling weights of multiple sample thought chains based on the deviation between the reasoning ability labels of the student reasoning model and the reasoning method labels of multiple sample thought chains.

[0076] The deviation between the reasoning ability label and the reasoning method label represents the degree of difference (i.e., the degree of conformity) between the reasoning ability label of the student's reasoning model and the reasoning method label of the sample thought chain. It can be used to measure the degree of match between the cognitive difficulty of the sample thought chain and the student's current cognitive or reasoning ability. Specifically, since both reasoning ability and reasoning method labels can be represented using hierarchical levels, this deviation can be quantified by calculating the distance between the student's reasoning model's reasoning ability label and the sample thought chain's reasoning method label. The smaller the deviation, the higher the degree of matching, and the more suitable the corresponding sample thought chain is as training data for the student's reasoning model, resulting in a higher sampling weight. For example, if the student's reasoning ability label is 5, and the sample thought chain's reasoning method label is also 5, the deviation is 0, indicating a perfect match, and the sample thought chain is suitable as training data for the student's reasoning model. If the sample thought chain's reasoning method label is 7, the deviation is 2, indicating a certain degree of mismatch, and the sample thought chain may not be suitable as training data for the student's reasoning model, resulting in a lower sampling weight.

[0077] Sampling weight is a weight value assigned to each sample thought chain during the selection process from the initial sample set for training. The sampling weight reflects the probability that each sample thought chain will be selected into the target sample set. Specifically, the sampling weight can be calculated based on the deviation between the reasoning ability label and the reasoning method label. The smaller the deviation, the larger the sampling weight, indicating that the cognitive difficulty of the sample thought chain is more consistent with the student's cognitive or reasoning ability, and the more likely it is to be selected into the target sample set.

[0078] Specifically, in the actual model training process, although it is hoped that the training data of the student reasoning model will match their own cognitive or reasoning abilities, the overall training expectation of the model is to improve the cognitive or reasoning abilities of the reasoning model. Therefore, instead of directly filtering multiple sample thought chains based on the reasoning ability labels of the student reasoning model, the sampling probability is determined based on the bias, so as to ensure that the target sample set matches the cognitive or reasoning abilities of the student reasoning model as much as possible, while improving the cognitive or reasoning abilities of the student reasoning model during the training process.

[0079] Specifically, to determine the deviation between the reasoning ability label and the reasoning method label, each sample thought chain in the initial sample set can be selected, and the deviation can be calculated with the reasoning ability label of the student's reasoning model. The sampling weight of that sample thought chain can then be determined accordingly. That is, each sample thought chain has its own sampling weight.

[0080] Optionally, the sampling weights can also be determined by the relationship between the reasoning method label of any sample thought chain and the reasoning ability label of the student reasoning model. Specifically, if the reasoning method label of any sample thought chain does not exceed the reasoning ability label of the student reasoning model, the sampling weight of that sample thought chain can be determined based on the largest reasoning method label among multiple sample thought chains in the initial sample set; if the reasoning method label of any sample thought chain exceeds the reasoning ability label of the student reasoning model, the sampling weight of that sample thought chain needs to be determined based on the largest reasoning method label among multiple sample thought chains in the initial sample set, and the reasoning method label of that sample thought chain.

[0081] Specifically, in determining the sampling weights of multiple sample thought chains, normalization or other mathematical methods can be used to process the data so that the final sampling weights are represented as weight values ​​between 0 and 1. This facilitates the subsequent sampling of the initial sample set, ensuring that the sampling weights accurately reflect the degree of matching between the cognitive difficulty of each sample thought chain and the student's cognitive or reasoning ability.

[0082] In this step, by determining the sampling weights of multiple sample thought chains based on the deviation between the reasoning ability labels of the student reasoning model and the reasoning method labels of multiple sample thought chains, it is possible to effectively filter out sample thought chains that match the cognitive or reasoning abilities of the student reasoning model, thus forming a customized target sample set suitable for training the student reasoning model. This ensures that during the training process for the student reasoning model, the student reasoning model can learn reasoning methods that match its own cognitive or reasoning abilities, avoiding cognitive biases caused by directly using the thought chains of a large-scale teacher reasoning model for training, and improving training efficiency and model reasoning ability.

[0083] Step 106: Based on the sampling weights, sample from the initial sample set to obtain the target sample set.

[0084] The process of obtaining the target sample set by sampling from the initial sample set is a process of randomly selecting multiple sample thought chains included in the initial sample set according to a predetermined sampling weight when selecting sample thought chains for training the student reasoning model from the initial sample set. This sampling process ensures that the obtained target sample set can match the current cognitive or reasoning ability level of the student reasoning model, thereby providing training data for the student reasoning model that matches its cognitive or reasoning ability.

[0085] Specifically, the sampling process can normalize the sampling weights of each sample thought chain in the initial sample set and then select samples probabilistically according to their weight ratios. This increases the probability of selecting thought chains with higher sampling weights, ensuring that the thought chains in the target sample set match the cognitive or reasoning abilities of the student's reasoning model. For example, if the initial sample set contains three thought chains with sampling weights of 0.8, 0.5, and 0.2, the system will randomly select a thought chain with a weight of 0.8 with approximately a 50% probability of selection, a thought chain with a weight of 0.5 with approximately a 31% probability, and a thought chain with a weight of 0.2 with approximately a 19% probability. This makes thought chains with higher weights more likely to be selected into the target sample set, thus forming a customized training dataset that matches the cognitive or reasoning abilities of the student's reasoning model.

[0086] In practical applications, one possible implementation of the sampling process is through mathematical methods, including but not limited to the roulette wheel algorithm, linear weighted sampling algorithm, and exponential decay sampling algorithm. Specifically, the roulette wheel algorithm is a commonly used sampling method. After converting the sampling weights into a probability distribution, a random number generator is used to sample from the probability distribution.

[0087] Another alternative implementation is to use hierarchical sampling based on the range of inference method labels. For example, the multiple sample thought chains in the initial sample set are divided into multiple subsets according to the distribution range of the inference method labels, and then sampling is performed in each subset according to the sampling weight to ensure that the target sample set includes sample thought chains with different cognitive difficulty levels.

[0088] Alternatively, a strategy of dynamically adjusting sampling weights can be adopted. That is, the sampling weights are dynamically adjusted according to the training progress of the student reasoning model during the sampling process, so that the model pays more attention to samples that match its own cognitive or reasoning abilities in the early stage of training. As training progresses, some more difficult sample thought chains are gradually introduced to promote the improvement of the model's cognitive or reasoning abilities, thereby achieving adaptive adjustment in the training process.

[0089] In this step, by sampling from the initial sample set based on sampling weights to obtain the target sample set, it is ensured that the selected sample thought chains match the current cognitive or reasoning abilities of the student's reasoning model, forming a customized training dataset. This allows the student's reasoning model to learn reasoning methods adapted to its own cognitive or reasoning abilities during training, avoiding cognitive biases caused by directly using the sample thought chains of a large-scale teacher reasoning model as the training set. This improves training efficiency and model reasoning ability, provides a high-quality data foundation for the subsequent model training process, and ensures that the trained model has better reasoning ability and higher reasoning efficiency.

[0090] Step 108: Based on the target sample set, train the student reasoning model to obtain the target reasoning model.

[0091] The process of training a student reasoning model involves using a target sample set to train the student reasoning model. By adjusting the model parameters, a fully trained target reasoning model is ultimately obtained. Specifically, the training process for the student model can employ different training methods, such as supervised fine-tuning training and / or reinforcement learning training.

[0092] Specifically, the training process can utilize the reasoning method labels of the sample thought chains in the target sample set, as well as the sample question text corresponding to the sample thought chains, to use the student reasoning model to reason about the sample question text, obtain the predicted thought chains, and further train based on the reasoning method labels of the predicted thought chains.

[0093] Supervised fine-tuning training (SFT) is a training method that uses supervised learning to fine-tune a student reasoning model based on a target sample set, thereby obtaining the target reasoning model. Specifically, supervised fine-tuning training can input sample question texts from the target sample set into the student reasoning model to generate predicted thought chains. By comparing the predicted thought chains with the sample thought chains, the loss value is calculated, thereby optimizing the model parameters. This allows the student reasoning model to better learn reasoning methods that match its own cognitive or reasoning abilities.

[0094] Reinforcement learning (RL) training is a method that uses reinforcement learning algorithms to train a model based on a target sample set to obtain a target reasoning model. Specifically, reinforcement learning training can define reward signals (e.g., rewards for correctness, format, reasoning method, and reasoning result), define a reasonable range for the target reasoning method label, and provide corresponding reward signals based on the predictive thought chain of the student reasoning model to guide the student reasoning model to learn better reasoning strategies.

[0095] The target reasoning model is the final trained model obtained after training the student reasoning model on a target sample set. The target reasoning model demonstrates improved reasoning ability compared to the student reasoning model before training, possessing the ability to dynamically adjust the reasoning process based on its own cognitive or reasoning capabilities. Specifically, the target reasoning model can dynamically adjust the reasoning methods used to adapt to problems of different difficulty levels; for example, it can generate shorter reasoning paths for simple problems and longer reasoning paths for complex problems, thereby achieving efficient utilization of computational resources.

[0096] In this embodiment, an initial sample set, a student reasoning model, and the reasoning ability labels of the student reasoning model are obtained. Based on the deviation between the reasoning ability labels of the student reasoning model and the reasoning method labels of the thought chains of each sample in the initial sample set, the sampling weights for each thought chain are dynamically determined. This allows for the selection of a training sample set that matches the cognitive or reasoning ability of the student reasoning model, and the student reasoning model is trained. This enables the student reasoning model to learn reasoning methods adapted to its current cognitive level from the teacher reasoning model during training, avoiding training bias caused by mismatch between cognitive or reasoning abilities. The target reasoning model trained using this method can dynamically adjust the reasoning process according to the complexity of the actual problem, using concise reasoning paths for simple problems and detailed reasoning paths for complex problems. This ensures reasoning accuracy while improving reasoning efficiency and achieving rational utilization of computing resources.

[0097] In one optional embodiment of this specification, any sample thought chain is labeled with a reasoning complexity tag corresponding to the reasoning result;

[0098] Based on the deviation between the reasoning ability labels of the student's reasoning model and the reasoning method labels of multiple sample thought chains, the sampling weights of multiple sample thought chains are determined, including:

[0099] Calculate multiple first label deviations between the reasoning ability labels of the student's reasoning model and the reasoning method labels of multiple sample thought chain annotations;

[0100] Based on multiple first label biases, determine the first distribution weights of the inference method labels;

[0101] Calculate multiple second label deviations between the reasoning method label and the reasoning complexity label of the multiple sample thought chain annotations;

[0102] Based on multiple second label biases, a second distribution weight is determined to determine the correlation between the inference method label and the inference complexity label;

[0103] Based on the first distribution weight and the second distribution weight, the sampling weights of multiple sample thought chains are determined.

[0104] Reasoning complexity labels are standardized identifiers used to quantify the structural complexity of a thought chain, measuring the reasoning exhaustion of a sample thought chain. Specifically, reasoning complexity labels can be graded based on the reasoning exhaustion of the output sample thought chain, using corresponding level labels. Higher levels indicate more steps in the thought chain, a more complex logical structure, and higher topological complexity. Corresponding to reasoning method labels, reasoning complexity labels can be divided into 10 sub-levels (0-9) based on the reasoning exhaustion of the sample thought chain. Adjacent sub-levels are combined into 5 major levels (0-1, 2-3, etc.), with different major levels corresponding to different levels of reasoning exhaustion. Specific reasoning complexity label classifications can be found in Table 2, which serves as a basis for such classifications.

[0105] Table 2: Criteria for Defining Reasoning Complexity Labels

[0106]

[0107] Optionally, the reasoning complexity label can also be determined based on the topological complexity of the reasoning step sequence of the sample thought chain. Topological complexity can include different measurement types such as linear chains, tree-like branching structures, and recursive nested calls, each corresponding to a different reasoning complexity label.

[0108] Specifically, the reasoning complexity label reflects the output form of the sample thought chain generated by the reasoning model. The output form of the sample thought chain reflects the specific thinking path adopted by the reasoning model when solving a specific reasoning problem. The higher the reasoning complexity label, the longer the thinking path and the longer the thinking process of the reasoning model in the process of generating the thought chain. Conversely, the lower the reasoning complexity label, the longer the thinking process. Therefore, the reasoning process using a thought chain with higher reasoning complexity can be called a slow-thinking process, and the reasoning process using a thought chain with lower reasoning complexity can be called fast-thinking.

[0109] Slow thinking processes are generally used for more difficult reasoning problems, and the corresponding reasoning methods are also more difficult. Fast thinking processes are generally used for simpler reasoning problems, and the corresponding reasoning methods are also simpler. However, these correspondences may not always match. For example, for a simple sample problem text "1+1=?", fast thinking is generally used, and the generated normal thought chain might be: "Identify as a simple arithmetic problem, find the sum of two 1s, perform addition: 1+1=2". However, if the reasoning model uses an incorrect reasoning process, i.e., slow thinking, the generated abnormal thought chain might be: "Problem identification: This is an arithmetic summation problem involving the addition of two positive integers. Define the addition principle: According to Peano's axioms, addition is a repeated successor operation. Analyze the operands: Take the first operand: the integer 1, whose Arabic numeral symbol is "1". Take the second operand: integer 1, the same as the first operand; Execute the calculation process: Step 1: Take the successor of the first operand → S(1) = 2, Step 2: Verify whether the second operand is 0 → 1 ≠ 0, Step 3: Subtract 1 from the second operand: 1 - 1 = 0, Step 4: Recursively calculate: S(S(1) + 0) = S(2) = 3? Correct the recursive logic: Correct recursive formula: S(a) + b = S(a + b), Actual calculation: S(1) + 1 = S(1 + 1) = S(2); Confirm the result: S(2) = 3? → Check the number line: 1 → 2 → 3 (secondary verification), Eliminate errors: Number line verification shows 1 + 1 = 2, Output conclusion: The final result is 2.

[0110] As can be seen from the examples above, for relatively simple reasoning problems, if slow thinking is adopted, the reasoning methods used in the generated thought chain are relatively simple, that is, the reasoning method label is low. However, the reasoning detail of the thought chain is high, that is, the reasoning complexity label is high. This mismatch between the reasoning method label and the reasoning complexity label will lead to excessive redundancy in the thinking process, resulting in a longer reasoning process and lower efficiency.

[0111] Therefore, in practical applications, it is expected that the reasoning model can dynamically select the corresponding thinking process based on the difficulty of the actual reasoning problem. In the generated thought chain, there should be a corresponding relationship between the reasoning method label and the reasoning complexity label, and the reasoning complexity label should not deviate too far from the reasoning method label. This type of reasoning process can be called dynamic thinking or adaptive thinking.

[0112] Specifically, the adaptive thinking process can constrain the reasoning complexity label based on the reasoning method label, thereby constraining the generated sampling probability from two dimensions: the reasoning method label and the reasoning complexity label. This results in a target sample set consisting of sample thinking chains that conform to the actual reasoning process, ensuring that the student reasoning model can learn a reasonable reasoning process that conforms to reality during the process of developing the student reasoning model.

[0113] The first label bias is the difference between the reasoning ability label of the student's reasoning model and the reasoning method label of multiple sample thought chains. This bias is used to measure the degree of matching between the cognitive difficulty of the sample thought chain and the current cognitive or reasoning ability of the student's reasoning model, and is further used to determine the first weight distribution.

[0114] The first weight distribution is a weight distribution calculated based on multiple first label biases. It reflects the degree of matching between the cognitive or reasoning abilities of different sample thought chains and the student's reasoning model, and can further determine the sampling probability to guide the sampling process. Specifically, the first weight distribution can be constructed by determining the sampling probability based on the first label bias.

[0115] The second label bias is the difference between the reasoning method label and the reasoning complexity label of each sample thought chain across multiple sample thought chains. It reflects the correlation between the reasoning method label and the reasoning complexity label of that sample thought chain. Specifically, the second label bias can be quantified by calculating the distance between the reasoning method label and the reasoning complexity label; a smaller second label bias indicates a higher correlation, and a larger second label bias indicates a lower correlation.

[0116] Relevance is the degree of matching between the reasoning method label and the reasoning complexity label, used to measure the rationality of the reasoning process corresponding to the sample's thought chain. Specifically, relevance reflects the logical relationship between cognitive difficulty and reasoning exhaustion; that is, thought chains with higher cognitive difficulty generally also have higher reasoning exhaustion, and vice versa. The higher the relevance, the more rational the reasoning process corresponding to the sample's thought chain.

[0117] The second weight distribution is a weight distribution calculated based on multiple second label biases. It reflects the correlation between the reasoning method label and the reasoning complexity label of each sample's thought chain. It can further supplement the constraints on the reasoning method label and jointly determine the sampling probability to guide the sampling process. Specifically, the second weight distribution can be constructed by determining the sampling probability based on the second label bias.

[0118] Specifically, similar to how inference model labels are determined, inference complexity labels can also be manually labeled or labeled using a pre-trained labeling model. The labeling model can be trained on a set of thought chains with labeled inference complexity labels, and can understand the mapping relationship between the topological complexity of the inference step sequence of the thought chain and the inference complexity labels.

[0119] Given multiple sample thought chains labeled with reasoning method and reasoning complexity, the second label bias can be determined by calculating the deviation between the reasoning method label and the reasoning complexity label of the sample thought chain.

[0120] Once the second label bias is determined, the second weight distribution can be further determined based on multiple second label biases.

[0121] Once the first and second weight distributions are determined, the weights of the first and second distributions can be summed using preset weighting coefficients to determine the total sampling probability. This comprehensively considers the matching degree between the sample thought chain and the cognitive or reasoning ability of the student's reasoning model, as well as the rationality of the reasoning process corresponding to the sample thought chain generation process.

[0122] In other words, determining the sampling weights for sampling the sample thought chains can be constrained from two dimensions. One dimension is to ensure that the reasoning method label (i.e., cognitive difficulty) of the sample thought chain does not exceed the model reasoning ability label of the student's reasoning model as much as possible. The other dimension is to ensure that the reasoning complexity label (i.e., reasoning exhaustion) of the sample thought chain does not deviate too far from its own reasoning method label, so as to conform to a reasonable reasoning process. That is, sample thought chains with lower cognitive difficulty generally have lower reasoning exhaustion, and vice versa.

[0123] Optionally, the preset weighting coefficients for the first and second distribution weights can be initially set to 0.5, meaning the first and second distribution weights are added together with equal weights to obtain the final sampling weights. Furthermore, as the training process progresses, the weighting coefficients can be flexibly adjusted to adapt to the specific needs of the training scenario.

[0124] In the embodiments of this specification, the first weight distribution and the second weight distribution are further determined by the first label deviation between the reasoning ability label of the student reasoning model and the reasoning method label of multiple sample thought chains, and the second label deviation between the reasoning method label and the reasoning complexity label of the sample thought chains. Finally, the sampling weights of multiple sample thought chains are determined. This can simultaneously consider the matching degree between the sample thought chains and the cognitive or reasoning ability of the student reasoning model, as well as the rationality of the reasoning process generated by the sample thought chains. This allows for the selection of sample thought chains that both match the current cognitive or reasoning ability of the student reasoning model and have a reasonable reasoning process, forming a high-quality customized training dataset. This enables the student reasoning model to learn reasoning methods that match its own cognitive or reasoning ability and have a reasonable reasoning process during training. This avoids the cognitive bias and unreasonable reasoning problems caused by directly using sample thought chains of large-scale teacher reasoning models for training, thus improving training efficiency and model reasoning ability.

[0125] In one optional embodiment of this specification, determining the first distribution weight of the inference method label based on multiple first label biases includes:

[0126] Determine the maximum first label deviation from multiple first label deviations, and calculate the total first label deviation from multiple first label deviations;

[0127] The first distribution weights of the inference method labels are determined based on the maximum first label bias and the total first label bias.

[0128] Based on multiple second label biases, a second distribution weight is used to determine the correlation between the inference method label and the inference complexity label, including:

[0129] Determine the maximum second label deviation from multiple second label deviations, and calculate the total second label deviation of multiple deviation labels;

[0130] Based on the maximum second label bias and the total second label bias, the second distribution weights of the correlation between the inference method label and the inference complexity label are determined.

[0131] The maximum first label bias is the maximum difference value determined in the process of calculating the first label bias between the reasoning ability label of the student reasoning model and the reasoning method label of multiple sample thinking chains. In other words, it is the difference between the reasoning method label of one (or more) sample thinking chains with the same reasoning method label and the reasoning ability label of the student reasoning model. It represents the maximum degree of mismatch between the cognitive ability or reasoning ability of the student reasoning model and the cognitive difficulty of the sample thinking chain.

[0132] Optionally, the determination of the maximum first label bias can also be based on the relationship between the inference method label of the sample thought chain and the inference ability label of the student inference model, specifically:

[0133] If the reasoning method label of the sample thinking chain does not exceed the reasoning ability label of the student reasoning model, the maximum first label deviation of the sample thinking chain is determined based on the largest reasoning method label among multiple sample thinking chains.

[0134] When the reasoning method label of a sample thinking chain exceeds the reasoning ability label of the student reasoning model, the maximum first label bias of the sample thinking chain is determined based on the largest reasoning method label among multiple sample thinking chains and the reasoning method label of the sample thinking chain.

[0135] Specifically, the maximum first label bias can be used to measure whether each sample's thought chain conforms to the student's reasoning model; that is, it can be used to evaluate whether a sample's thought chain is suitable to be included in the target sample set. The larger the maximum first label bias, the higher the score of the sample's thought chain, and the more suitable it is as a sample thought chain in the target sample set. For example, when the reasoning ability label of the student's reasoning model is 5, and the reasoning method labels of multiple sample thought chains are 2, 3, 4, and 5 respectively, the maximum reasoning method label is 5. Then, for these sample thought chains, the positive maximum first label deviation is the same and is |2-5|=3, indicating that the reasoning difficulty of these sample thought chains completely matches the cognitive or reasoning ability of the student's reasoning model, and does not exceed the difficulty of the cognitive or reasoning ability. When the reasoning method labels of multiple sample thought chains are 6, 7, 8, and 9 respectively, the maximum reasoning method label is 9. Then, the corresponding negative maximum first label deviations of each sample thought chain are 3, 2, 1, and 0 respectively. This indicates that among these sample thought chains, the sample thought chain with the reasoning method label 6 scores the highest and matches the cognitive or reasoning ability of the student's reasoning model, while the thought chain with the reasoning method label 9 scores the lowest and does not match the cognitive or reasoning ability of the student's reasoning model.

[0136] The total first label bias is the sum of multiple first label biases between the reasoning ability label of the student's reasoning model and the reasoning method label of multiple sample thought chains. It can reflect the overall matching degree between all sample thought chains and the cognitive or reasoning ability of the student's reasoning model. Specifically, the total first label bias, together with the maximum first label bias, is used to calculate the first weight distribution, so that sample thought chains with smaller biases receive higher sampling weights.

[0137] The maximum second label bias is the largest difference obtained in the process of calculating multiple second label biases between the reasoning method labels and reasoning complexity labels of multiple sample thought chains. In other words, it represents the difference between the reasoning method label and reasoning complexity label of a particular (or multiple) sample thought chain with the same reasoning method and reasoning complexity labels, indicating the maximum degree of mismatch between the reasoning method label and reasoning complexity label of the sample thought chain. Specifically, the maximum second label bias can be used to determine the benchmark for the second weight distribution; a larger bias indicates a lower rationality of the reasoning process corresponding to the sample thought chain.

[0138] The total second label bias is the sum of multiple second label biases between the reasoning method labels and the reasoning complexity labels of multiple sample thought chains. It reflects the overall level of rationality of the reasoning process of all sample thought chains. Specifically, the total second label bias, together with the maximum second label bias, is used to calculate the second weight distribution, giving higher sampling weights to sample thought chains with higher relevance.

[0139] In actual training, given multiple first label biases and multiple second label biases, the maximum first label bias and the total first label bias can be determined based on the multiple first label biases, and the maximum second label bias and the total second label bias can be determined based on the multiple second label biases.

[0140] Specifically, the first label deviation between the reasoning ability label of the student's reasoning model and the reasoning method label of each sample's thinking chain can be calculated separately. Then, the maximum value among all the first label deviations is determined as the maximum first label deviation, and all the first label deviations are added together to obtain the total first label deviation. The maximum second label deviation and the total second label deviation are calculated in the same way.

[0141] For example, further explanation can be provided in conjunction with actual calculation methods.

[0142] The reasoning ability label for the student's reasoning model is: Then, for any sample's thought chain Its reasoning method label can be denoted as The reasoning complexity label can be denoted as .

[0143] Correspondingly, the maximum first label deviation It can be represented as:

[0144]

[0145] in, For the initial sample set, This represents the total number of thought chains in the initial sample set. For the first The reasoning method label for each sample's thought chain.

[0146] Correspondingly, the maximum second label deviation It can be represented as:

[0147]

[0148] in, For the initial sample set, This represents the total number of thought chains in the initial sample set. For the first The reasoning method tags for individual sample thought chains. For the first The reasoning complexity label of a sample thought chain.

[0149] Correspondingly, the first distribution weight It can be represented as:

[0150]

[0151] in, Indicates the first The first label bias corresponding to the thought chain of each sample This represents the total first label deviation.

[0152] Correspondingly, the second distribution weights It can be represented as:

[0153]

[0154] in, Indicates the first The second label bias corresponding to the thought chain of each sample This represents the total second label deviation.

[0155] Correspondingly, the sampling weights are determined based on the first distribution weights and the second distribution weights. It can be represented as:

[0156]

[0157] in, These are the preset weighting coefficients.

[0158] In the embodiments of this specification, by determining the maximum first label deviation and the total first label deviation based on multiple first label deviations, and by determining the maximum second label deviation and the total second label deviation based on multiple second label deviations, the first distribution weight of the reasoning method label and the second distribution weight of the correlation between the reasoning method label and the reasoning complexity label can be accurately calculated. This comprehensively considers the matching degree between the sample thought chain and the student's reasoning model's cognitive ability or reasoning ability, as well as the rationality of the reasoning process. As a result, the sampling weights determined based on the first and second distribution weights can more accurately reflect the quality of the sample thought chain, and select high-quality sample thought chains that not only match the student's current cognitive ability or reasoning ability but also have a reasonable reasoning process. This forms a more accurate customized training dataset, enabling the student's reasoning model to learn reasoning methods that match its own cognitive ability or reasoning ability and have a reasonable reasoning process during the training process, effectively improving the model's reasoning ability and training efficiency.

[0159] In one optional embodiment of this specification, training a student reasoning model based on a target sample set to obtain a target reasoning model includes:

[0160] Based on the target sample set, the student reasoning model is trained under supervised fine-tuning to obtain the target reasoning model;

[0161] or,

[0162] Based on the target sample set, the target reasoning model is trained by reinforcement learning to obtain the target reasoning model.

[0163] Supervised fine-tuning training (SFT) is a training method that uses supervised learning to fine-tune a student reasoning model based on a target sample set, thereby obtaining the target reasoning model. Specifically, supervised fine-tuning training can input sample question texts from the target sample set into the student reasoning model to generate a predicted thought chain. By comparing this predicted thought chain with the sample thought chain, the loss value is calculated, thereby optimizing the model parameters. This allows the student reasoning model to better learn reasoning methods that match its own cognitive or reasoning abilities.

[0164] Specifically, supervised fine-tuning training can use the sample thought chains in the target sample set as labels, and calculate the cross-entropy loss value based on the output prediction thought chain of the student inference model to adjust the model parameters of the student inference model.

[0165] Reinforcement learning (RL) training is a method that uses reinforcement learning algorithms to train a model based on a target sample set to obtain a target reasoning model. Specifically, reinforcement learning training can define reward signals (e.g., rewards for correctness, format, reasoning method, and reasoning result), define a reasonable range for the target reasoning method label, and provide corresponding reward signals based on the predictive thought chain of the student reasoning model to guide the student reasoning model to learn better reasoning strategies.

[0166] Specifically, reinforcement learning training can construct a reward signal with multiple dimensions by using the reasoning method labels and reasoning complexity labels of the sample thinking chain annotations in the target sample set. Based on the reward signal, the student's reasoning model can be guided to gradually adjust the reasoning process, thereby achieving a balance between reasoning efficiency and accuracy.

[0167] Optionally, the training of the student reasoning model can employ supervised fine-tuning training or reinforcement learning training; alternatively, it can be combined with both methods. That is, supervised fine-tuning training is first performed on the student reasoning model to obtain a preliminarily trained fine-tuned reasoning model, and then reinforcement learning training is performed on the fine-tuned model to obtain the target reasoning model. The specific training method used, or a combination of methods, can be flexibly adjusted according to the actual reasoning model training and application scenario. This specification does not impose specific limitations on this aspect in the embodiments.

[0168] In the embodiments of this specification, by supervising and fine-tuning the student reasoning model based on the target sample set, or by performing reinforcement learning training, the student reasoning model can learn reasoning methods that match its own cognitive or reasoning abilities during the training process, and can also dynamically adjust its reasoning strategy according to the training objectives. This ensures that the target reasoning model obtained through training adopts concise reasoning paths for simple problems to avoid overthinking, and adopts detailed reasoning paths for complex problems to avoid underthinking, thereby achieving a balance between reasoning efficiency and accuracy. The trained target reasoning model can adaptively adjust the reasoning process according to the complexity of the problem, maximize the use of computing resources, and improve reasoning performance.

[0169] In one optional embodiment of this specification, in the target sample set, the thought chain of any sample also includes the corresponding sample question text;

[0170] Based on the target sample set, the student reasoning model is trained under supervised supervision to obtain the target reasoning model, including:

[0171] Using a student reasoning model, reasoning is performed based on sample question text to obtain the predictive thought chain generated by the student reasoning model;

[0172] Based on the predicted thinking chain and the sample thinking chain, the thinking chain loss value is determined.

[0173] Based on the thought chain loss value, the student reasoning model is trained to obtain the target reasoning model.

[0174] The sample question text corresponds to the sample thought chain; that is, the sample thought chain is generated by the teacher's reasoning model based on the sample question text. The sample question text can be used as input to the student's reasoning model, serving as input data provided during model training. Specifically, the sample question text can contain the specific problem content that the model needs to reason about, such as a mathematical problem description like "calculate the value of 1+2+3," or a scientific problem description like "explain the difference between the Earth's rotation and revolution." The sample question text and the sample thought chain together constitute the input-output pair of the training data, providing the student's reasoning model with the context of the reasoning task, enabling the model to generate the corresponding reasoning process based on the problem content.

[0175] A predictive thought chain is a sequence of reasoning steps generated by a student inference model after it has reasoned based on an input sample problem text. It can include the thought processes and logical derivations of the reasoning methods employed by the student inference model in solving a given sample problem text. For example, for the problem "Calculate the value of 1+2+3", the predictive thought chain could be "First calculate 1 plus 2 equals 3, then add 3 to get the result 6". The predictive thought chain is the output generated by the student inference model during training. It is used to compare with the sample thought chain to calculate the loss value and guide the adjustment of model parameters. Its level of detail and the number of reasoning steps depend on the model's reasoning ability and the current training stage.

[0176] The thought chain loss is a quantitative metric used to quantify the sequence differences between the predicted thought chain generated by the student reasoning model and the sample thought chain. The thought chain loss is calculated by comparing the similarity between the sequence of reasoning steps included in the predicted thought chain and the sequence of reasoning steps included in the sample thought chain. Optionally, the thought chain loss can be calculated using various forms such as cross-entropy loss, KL divergence loss, or cosine similarity loss to guide the student reasoning model in optimizing the generation quality of the reasoning process during training. By minimizing the thought chain loss, the model parameters of the student reasoning model are adjusted so that the predicted thought chain generated by the student reasoning model during the reasoning process better matches the sample thought chain, thereby ensuring that the model can learn reasoning methods adapted to its own cognitive or reasoning abilities.

[0177] Specifically, once the target sample set is obtained, the sample question texts corresponding to multiple sample thought chains in the target sample set can be input into the student reasoning model. The student reasoning model then performs reasoning based on the input sample question texts. Through its internal neural network structure and parameters, it gradually generates a sequence of reasoning steps in the reasoning process. For example, it can use an autoregressive approach to generate reasoning steps one by one, or it can generate the entire sequence of reasoning steps at once. Furthermore, based on the sequence of reasoning steps, the prediction thought chain is determined.

[0178] The specific predictive thought chain generated can include reasoning processes such as understanding the sample question text, deriving steps, and determining the answer. Its level of detail and the number of reasoning steps depend on the reasoning ability of the student's reasoning model and the difficulty level of the sample question text.

[0179] Once the predictive thought chain is generated, the difference between the predictive thought chain and the sample thought chain can be quantified by calculating the logarithmic loss value of the conditional probability, thereby determining the thought chain loss value. Furthermore, based on the thought chain loss value, the adjustment direction to minimize the thought chain loss value can be used to adjust the model parameters of the student reasoning model, thereby obtaining the target reasoning model.

[0180] In the embodiments of this specification, by inputting sample question text into the labeling model, a predicted thought chain is obtained. Furthermore, based on the difference between the predicted thought chain and the sample thought chain labeling, the thought chain loss value is determined to adjust the model parameters of the student reasoning model, thereby obtaining the target reasoning model. This effectively guides the student reasoning model to learn reasoning methods that match its own cognitive or reasoning abilities. During training, the model can gradually adjust the complexity of its reasoning process, ensuring that the generated predicted thought chain not only matches the model's cognitive or reasoning abilities but also maintains consistency with the difficulty level of the sample question text, thus improving the efficiency of model training and the quality of reasoning.

[0181] In one optional embodiment of this specification, in the target sample set, the thought chain of any sample also includes the corresponding sample question text;

[0182] Based on the target sample set, the target reasoning model is trained using reinforcement learning to obtain the target reasoning model, including:

[0183] Using a target reasoning model, reasoning is performed based on sample question text to obtain the predictive thought chain generated by the target reasoning model;

[0184] Using a reasoning method reward model, the reasoning method reward score is determined based on the predictive thought chain generated by the target reasoning model;

[0185] Based on the reasoning method reward score and the reasoning method label range, the deviation of the reasoning method is determined. The reasoning method label range is set according to the target reasoning ability of the target reasoning model.

[0186] Based on the deviation of the reasoning method, a reward signal is determined, and the student reasoning model is trained based on the reward signal to obtain the target reasoning model.

[0187] Reinforcement learning (RL) training is a method that uses reinforcement learning algorithms to train a model based on a target sample set to obtain a target reasoning model. Specifically, reinforcement learning training can define reward signals (e.g., rewards for correctness, format, reasoning method, and reasoning result), define a reasonable range for the target reasoning method label, and provide corresponding reward signals based on the predictive thought chain of the student reasoning model to guide the student reasoning model to learn better reasoning strategies.

[0188] Specifically, reinforcement learning training can guide students' reasoning models to learn reasoning processes that match the target reasoning method by identifying reward signals (such as correctness rewards, format rewards, reasoning method rewards, reasoning result rewards, etc.) and evaluating the predictive thought chains generated by students' reasoning models based on the reward signals. This enables the models to use concise reasoning paths for simple problems and detailed reasoning paths for complex problems.

[0189] A predictive thought chain is a sequence of reasoning steps generated by a student reasoning model after it has reasoned based on an input sample problem text. It can include the thought processes and logical derivations of the reasoning methods employed by the student reasoning model in solving a given sample problem text. The predictive thought chain is the output generated by the student reasoning model during training, used to determine the reward score for the reasoning method, and further combined with the reasoning method label range to determine the reward signal.

[0190] The reasoning method reward model is a pre-trained reward model that maps scores to the predicted thought chains generated by a student's reasoning model, determining the score of the predicted thought chain on the reasoning method label. Specifically, by learning the mapping relationship between thought chains and reasoning method labels, the reasoning method reward model can score the reasoning method label of the predicted thought chain, thus providing a reasoning method reward score. More specifically, the reasoning method reward model can be trained on a set of thought chains labeled with reasoning method labels, understanding the mapping relationship between the sequence length of the reasoning steps in the thought chain and the reasoning method label. The reasoning method reward model typically employs a neural network architecture such as MLP or Transformer.

[0191] The reasoning method reward score is a quantitative indicator obtained by the reasoning method reward model after scoring the reasoning method label of the predictive thought chain. The reasoning method reward score can measure the performance of the predictive thought chain in terms of cognitive difficulty. The higher the score, the more the cognitive difficulty of the predictive thought chain matches the actual difficulty level of the sample question text.

[0192] The reasoning method label range is a pre-defined reasonable range of reasoning method labels based on the target reasoning ability of the desired training target reasoning model. The reasoning method label range can represent the cognitive or reasoning ability level that the trained target reasoning model should achieve in actual reasoning processes, and can be used as a training objective and constraint for student reasoning models. For example, if the target reasoning ability label of the desired training target reasoning model is 7, then the reasoning method label range can be set to [6, 8]. This means that predictive thought chains with reasoning method labels between 6 and 8 are consistent with the target reasoning ability, while thought chains outside this range will receive lower rewards or higher penalties.

[0193] Target reasoning methods are the reasoning methods that the trained target reasoning model should achieve during model training, representing the ideal reasoning ability and corresponding reasoning methods. In other words, they represent the cognitive or reasoning ability level that the target reasoning model should reach during the reasoning process. Target reasoning methods can be set according to actual model training objectives. The range of reasoning method labels is determined by the target reasoning ability, and reward signals are determined by combining the model's score with the reasoning method reward system. This ensures that the student reasoning model develops towards the desired cognitive or reasoning ability during training.

[0194] Inference method deviation measures the degree of deviation between the inference method label and the range of inference method labels in the predictive thought chain. It aims to avoid extreme rewards by limiting the deviation to a specific range. Specifically, inference method deviation can be calculated using a clipping function to restrict the deviation of the inference method label to a range within which the reward signal aligns with the direction of the target inference model.

[0195] In reinforcement learning training, the reward signal is a comprehensive score used to guide the adjustment of model parameters in student inference models. It can include multiple dimensions such as correctness reward, format reward, inference method reward, and inference result reward. The reward signal is obtained by weighted summation of the rewards from each dimension and is used to evaluate the overall quality of the model's inference process. A higher reward signal indicates that the model's inference process is more in line with the training objective, i.e., the target inference model, which is more conducive to model training. The inference ability of the model can be optimized by maximizing the reward signal.

[0196] Specifically, once the target sample set is obtained, the student reasoning model can be used to reason based on the sample question text in the target sample set to obtain the predictive thought chain, and then the reasoning method reward model can be used to determine the reasoning method reward score.

[0197] Among them, the reasoning method reward model can be trained based on the sample thought chain data in the initial sample set, thereby learning the score mapping relationship of the predicted thought chain with respect to cognitive difficulty.

[0198] Once the reward score for the reasoning method is determined, the deviation of the reasoning method can be determined by combining the label range of the reasoning method, and the reward signal can be further determined to train the student's reasoning model.

[0199] Among these methods, training the student inference model based on reward signals can employ the GRPO (Generalized Reward-based Policy Optimization) algorithm. This algorithm, in addition to standard reinforcement learning, adds extra rewards determined by the degree of deviation from the inference method and guides model training by defining multi-dimensional reward signals. Alternatively, reinforcement learning algorithms such as PPO (Proximal Policy Optimization), DAPO (Diverse Action Policy Optimization), or GSPO (Generalized State Policy Optimization) can also be used.

[0200] For example, further explanation can be provided in conjunction with actual calculation methods.

[0201] Based on the deviation of the reasoning method, the reward signal is determined and can be expressed as:

[0202]

[0203] in, The reasoning method reward model is based on the reasoning method reward score determined by the predictive thought chain. and Labels for the scope of reasoning methods. The deviation of the reasoning method can be represented by a pruning function. Sure.

[0204] Then the deviation of the reasoning method The smaller the value, the stronger the reward signal. The smaller the penalty signal generated, the better. If the deviation of the reasoning method is 0, then there is no penalty signal.

[0205] In the embodiments of this specification, by determining the predictive thought chain of the student reasoning model and the reward score of the reasoning method, and further determining the deviation of the reasoning method based on the reward score and the label range of the reasoning method, and then determining the reward signal based on the deviation of the reasoning method, the student reasoning model is trained through the reward signal. This can guide the student reasoning model to learn the reasoning process that matches the target reasoning method, so that the trained target reasoning model adopts a concise reasoning path for simple problems and a detailed reasoning path for complex problems. This improves reasoning efficiency while ensuring reasoning accuracy and realizes the rational use of computing resources.

[0206] In one optional embodiment of this specification, any sample thought chain is labeled with a reasoning complexity tag corresponding to the reasoning result;

[0207] Before determining the reward signal based on the deviation of the reasoning method, the following is also included:

[0208] Using the reasoning result reward model, the reward score for the reasoning result is determined based on the predictive thought chain generated by the target reasoning model;

[0209] Based on the reasoning result reward score and the reasoning complexity label range, the deviation of the reasoning result is determined, wherein the reasoning complexity label range is set according to the target output result of the target reasoning model;

[0210] Based on the deviation of the reasoning method, the reward signal is determined, including:

[0211] Based on the deviation of the reasoning method, the reward signal of the reasoning method is determined, and based on the deviation of the reasoning result, the reward signal of the reasoning result is determined.

[0212] The reward signal is determined based on the reward signal of the reasoning method and the reward signal of the reasoning result.

[0213] The reasoning outcome reward model is a pre-trained reward model that maps scores to the predicted thought chains generated by a student's reasoning model, determining the score of each predicted thought chain on a reasoning complexity label. Specifically, by learning the mapping relationship between thought chains and reasoning complexity labels, the reasoning outcome reward model can score the reasoning complexity label of the predicted thought chain, thus providing a reasoning outcome reward score to assess whether the reasoning exhaustion of the predicted thought chain's reasoning process meets expectations. More specifically, the reasoning outcome reward model can be trained on a set of thought chains labeled with reasoning complexity labels, and can understand the mapping relationship between the topological complexity of the reasoning step sequence of the thought chain and the reasoning complexity label.

[0214] For example, for the relatively simple sample question text "1+2=?", the generated predictive thought chain "calculate 1 plus 2 equals 3" can give a low reward score to the inference result reward model. However, for the more complex sample question text "solve x²-5x+6=0", the generated predictive thought chain "first, we need to determine the corresponding algebraic solution, such as using factorization, we can decompose the equation into (x-2)(x-3)=0, and then get x=2 or x=3", the inference result reward model can give a higher reward score.

[0215] The reasoning outcome reward score is a quantitative indicator obtained by the reasoning outcome reward model after scoring the reasoning complexity label of the predictive thought chain. The reasoning outcome reward score can measure the score of the predictive thought chain on reasoning exhaustion. The higher the score, the more the reasoning exhaustion of the predictive thought chain matches the actual reasoning needs of the sample question text.

[0216] The reasoning complexity label range is a pre-defined reasonable range of reasoning complexity labels based on the target output of the desired training reasoning model. The reasoning complexity label range can represent the level of reasoning exhaustion that the trained target reasoning model should achieve in actual reasoning processes, and can serve as both a training objective and a training constraint for student reasoning models.

[0217] The target output is the ideal reasoning result of the target reasoning model expected to be trained during the model training process. In other words, it represents the level of reasoning exhaustion that the thought chains generated by the target reasoning model should meet during the reasoning process. The target output can be set according to the model's training objectives and used to determine the range of reasoning complexity labels, ensuring that the student reasoning model develops towards the desired level of reasoning exhaustion during training.

[0218] Inference outcome deviation measures the degree of deviation between the inference complexity label of the predictive thought chain and the range of inference complexity labels. By limiting the deviation to a specific range, extreme rewards can be avoided. Specifically, the calculation of inference outcome deviation can also use a pruning function to limit the deviation of the inference complexity label to the range of the inference complexity label, so that the reward signal can accurately reflect the degree of matching between the inference exhaustion and the target output result.

[0219] The inference method reward signal is a reward signal determined based on the inference method deviation. It measures the degree of matching between the inference method label of the predictive thought chain and the range of inference method labels. The higher the inference method reward signal, the more the predictive thought chain matches the requirements of the target inference method in terms of cognitive difficulty, and the more beneficial it is for model training.

[0220] The inference result reward signal is determined based on the deviation of the inference result. It measures the degree of matching between the inference complexity label and the range of inference complexity labels in the predictive thought chain. The higher the inference result reward signal, the more the predictive thought chain matches the requirements of the target output in terms of inference exhaustion, which is more conducive to model training.

[0221] Specifically, given the predictive thought chain, the reasoning result reward model can be further used to determine the reasoning result reward score of the predictive thought chain.

[0222] Among them, the reasoning result reward model can be trained based on the sample thinking chain data in the initial sample set, thereby learning the score mapping relationship of the prediction thinking chain with respect to the reasoning exhaustion.

[0223] Once the reward score for the reasoning result is determined, the deviation of the reasoning result can be determined by combining the range of reasoning complexity labels, and then the reward signal for the reasoning result can be further determined.

[0224] Having determined the reward signal for the reasoning method and the reward signal for the reasoning result, the total reward signal can be further determined to train the student reasoning model.

[0225] For example, using the previous example and combining it with actual calculation methods, we will further explain.

[0226] Based on the deviation of the inference result, the reward signal for the inference result can be determined as follows:

[0227]

[0228] in, The reasoning result reward model is based on the reasoning result reward score determined by the predictive thought chain. and Labels for the range of inference results. The deviation of the inference result can be represented by a pruning function. Sure.

[0229] The deviation of the reasoning result The smaller the value, the higher the reward signal for the reasoning result. The smaller the penalty signal generated, the better. If the deviation of the reasoning method is 0, then there is no penalty signal.

[0230] Correspondingly, the determination of the reward signal can be expressed as:

[0231]

[0232] in, As a reward signal, and These are the corresponding weighting coefficients.

[0233] In the embodiments of this specification, the reasoning result reward score of the predictive thought chain of the student reasoning model is determined, and further, based on the reasoning result reward score and the range of reasoning complexity labels, the reasoning result deviation is determined. Then, the reasoning result reward signal is determined based on the reasoning result deviation. Finally, the total reward information is determined by combining the reasoning method reward signal determined based on the reasoning method deviation. This is used to train the student reasoning model. During the training process, the matching degree of the reasoning method label and the reasoning complexity label is considered simultaneously. This guides the student reasoning model to learn the reasoning process that matches the target reasoning method and the target output result. The trained target reasoning model can use a concise reasoning path for simple problems and a detailed reasoning path for complex problems. This improves reasoning efficiency while ensuring reasoning accuracy and achieves rational utilization of computing resources.

[0234] In an optional embodiment of this specification, before determining the reward signal based on the inference method reward signal and the inference result reward signal, the method further includes:

[0235] Based on the logic of the predictive thinking chain and the sample thinking chain, the correctness reward signal is determined, and based on the thinking chain format of the predictive thinking chain and the sample thinking chain, the format reward signal is determined.

[0236] Based on the reward signal of the reasoning method and the reward signal of the reasoning result, the reward signal is determined, including:

[0237] The reward signal is determined based on the reasoning method reward signal, the reasoning result reward signal, the correctness reward signal, and the format reward signal.

[0238] Thought chain logic refers to the logical relationships and reasoning paths between the sequence of reasoning steps used by a reasoning model to solve a specific problem. It reflects the rationality, coherence, and logical rigor of the reasoning process and can be used to measure the quality of reasoning. Thought chain logic can include whether the order of reasoning steps is reasonable, whether the causal relationship between reasoning steps is clear, and whether the reasoning process conforms to mathematical or scientific principles. For example, for the sample problem text "Calculate 2 × 3 + 4", a logically sound thought chain could be "Step 1: Calculate 2 × 3 = 6; Step 2: Calculate 6 + 4 = 10; Final answer: 10". This thought chain logic clearly demonstrates the order of multiplication followed by addition, and there is a clear causal relationship between the reasoning steps. An illogical thought chain could be "Step 1: Calculate 3 + 4 = 7; Step 2: Calculate 2 × 7 = 14; Final answer: 14". This thought chain logic incorrectly changes the order of operations, the causal relationship between the reasoning steps is invalid, leading to an incorrect reasoning result.

[0239] The correctness reward signal is a quantitative indicator used to measure the logical correctness and answer accuracy of the predictive thought chain generated by the reasoning model. The correctness of the reasoning process can be evaluated by comparing the logic of the predictive thought chain with that of the sample thought chain, as well as the consistency between the final answer obtained by the predictive thought chain and the correct answer.

[0240] The format of a thought chain refers to the standardization of the structure, expression, and presentation of the thought chain generated by the reasoning model. It reflects the standardization and readability of the reasoning process and can be used to measure the quality of the reasoning result. The format of a thought chain can include whether the numbering of the reasoning steps is standardized, whether the separation between reasoning steps is clear, whether the symbols and terminology used in the reasoning process conform to standards, and whether the reasoning steps include necessary explanations. For example, a thought chain that conforms to the format standard could be "Step 1: Calculate 2 × 3 = 6; Step 2: Calculate 6 + 4 = 10; Final answer: 10". This thought chain clearly lists each step and the final answer, with standardized step numbering and clear separation between steps. A thought chain that does not conform to the format standard could be "2 × 3 = 6, then 6 plus 4 equals 10, so the answer is 10". This thought chain lacks step numbering and clear separation, and there is a mixture of calculation formulas and verbal descriptions, resulting in poor readability and non-standard terminology.

[0241] The format reward signal is a quantitative indicator used to measure the standardization and readability of the predictive thought chain generated by the inference model in terms of thought chain format. It can be used to evaluate the degree of structure and standardization of the inference results by comparing the thought chain format of the predictive thought chain with that of the sample thought chain.

[0242] Specifically, once the predictive thought chain is obtained, the correctness reward signal and the format reward signal can be determined based on the predictive thought chain and the thought chain logic and thought chain format between the thought chains.

[0243] Once the correctness reward signal and the format reward signal are determined, the total reward signal can be determined by combining the reasoning method reward signal and the reasoning result reward signal.

[0244] For example, using the previous example and combining it with actual calculation methods, we will further explain.

[0245] Based on the reasoning method reward signal, reasoning result reward signal, correctness reward signal, and format reward signal, the reward signal is determined and can be expressed as:

[0246]

[0247] in, As a reward signal, As a reward signal for correctness, For formatted reward signals,

[0248] Reward signals for reasoning methods, As a reward signal for the reasoning result, and These are the corresponding weighting coefficients.

[0249] In the embodiments of this specification, a correctness reward signal is determined by the logic of the thought chain based on the predictive thought chain and the sample thought chain, and a format reward signal is determined by the thought chain format based on the predictive thought chain and the sample thought chain. Then, the reasoning method reward signal and the reasoning result reward signal are combined to form a comprehensive reward signal. This can guide the student reasoning model to optimize the rationality of the reasoning logic, the accuracy of the answer, the completeness of the reasoning process, and the standardization of the output during the training process. This enables the target reasoning model obtained through training to produce logically rigorous and accurate reasoning results, and to present the reasoning process in a standardized and clear format. Thus, while ensuring the quality of reasoning, the readability and interpretability of the reasoning process are improved, allowing the target reasoning model to better meet user needs in practical applications.

[0250] In one optional embodiment of this specification, a schematic diagram of the framework of a model training method is provided. Specifically, see Figure 2, which shows a schematic diagram of the framework of a model training method according to an embodiment of this specification.

[0251] During model training, an initial sample set is obtained, which includes multiple sample thought chains labeled with reasoning method and reasoning complexity labels. Based on the desired training objective of the model, namely, to obtain the reasoning ability of the reasoning model with adaptive thinking, and the reasoning ability label of the student reasoning model to be trained, the first distribution weight and the second distribution weight of the corresponding reasoning method label are determined respectively. The first distribution weight is used to sample based on the reasoning method label, and the second distribution weight is used to sample based on the reasoning complexity label to obtain the target sample set. With the target sample set obtained, the student reasoning model can be trained using supervised fine-tuning and / or reinforcement learning to obtain the target reasoning model.

[0252] In one optional embodiment of this specification, a text reasoning method is provided. Specifically, referring to Figure 3, Figure 3 shows a flowchart of a text reasoning method provided according to an embodiment of this specification, which specifically includes the following steps.

[0253] Step 302: Obtain the target question text.

[0254] Step 304: Input the target question text into the target reasoning model to obtain the target answer text, wherein the target reasoning model is trained based on the model training method described above.

[0255] The target question text is the input question description that needs to be processed by the target reasoning model in a real-world question-answering reasoning scenario. The content and complexity of the target question text determine the target reasoning method that the target reasoning model needs to adopt and the target reasoning result obtained. Specifically, the target question text can contain various types of questions that need to be solved through reasoning, such as mathematical calculation problems, logical reasoning problems, or scientific explanation problems.

[0256] The target reasoning model is the final reasoning model obtained through training using the model training methods described above. Its reasoning ability is significantly improved compared to the untrained student reasoning model, avoiding problems of overthinking and underthinking. The target reasoning model has the ability to dynamically adjust the reasoning process based on its own cognitive or reasoning abilities, and can select appropriate reasoning methods for target problem texts of varying difficulty, achieving a balance between reasoning efficiency and accuracy. The target reasoning model is the final model obtained by training the student reasoning model based on a target sample set.

[0257] The target answer text is the final answer generated by the target reasoning model after reasoning about the target question text. It is the output of the reasoning process, and its content and level of detail are directly related to the reasoning exhaustion of the reasoning model. Specifically, the target answer text may include the solution to the question, a summary statement of the reasoning process, or a detailed explanation of the question. Its expression should conform to the level of reasoning exhaustion set by the reasoning ability label of the reasoning model.

[0258] In the embodiments of this specification, in scenarios such as mobile devices, embedded systems, or edge computing devices with limited inference resources, the target question text is input into a target inference model trained based on a customized training framework. This allows the target inference model to adaptively adjust the inference process according to the complexity of the target question text, using a concise inference path for simple questions and a detailed inference path for complex questions. This maximizes the use of limited computing resources while ensuring inference accuracy, achieving a good balance between inference efficiency and accuracy. The target inference model can efficiently execute inference tasks in scenarios with limited inference resources, meeting the dual requirements of inference speed and accuracy in practical application scenarios.

[0259] Corresponding to the above method embodiments, this specification also provides a model training platform embodiment. Figure 4 shows a schematic diagram of the structure of a model training platform provided in one embodiment of this specification. As shown in Figure 4, the platform includes a training interface 402 and a first response unit 404:

[0260] Training interface 402 is used to receive model training requests;

[0261] The first response unit 404, in response to the model training request, executes the model training method as described above.

[0262] The model training platform is a systematic technical architecture based on a customized training framework. It is used to execute the training process for student reasoning models. By acquiring input elements such as the initial sample set, the student reasoning model and its reasoning ability labels, it dynamically constructs a target sample set that is adapted to the cognitive or reasoning ability of the student model. Based on this sample set, it completes supervised fine-tuning and / or reinforcement learning training of the student reasoning model, and finally obtains a target reasoning model with dynamic reasoning ability.

[0263] The training interface is a system interface used to receive model training requests. It can serve as the entry point for the model training platform to interact with external users or systems. It can support multiple communication protocols and data formats, such as HTTP / HTTPS or gRPC, enabling external systems to submit model training tasks through standard network requests.

[0264] A model training request is a structured request that includes the parameters and data required for model training, used to submit training tasks to the model training platform. Specifically, a model training request may include key information such as the storage path of the initial sample set, the initial parameter configuration of the student inference model, the specified range of inference method labels, and the selection of the training algorithm. For example, it may specify the storage location of the initial sample set, the model parameter scale of the student model to be trained, and the specific training method used.

[0265] The first response unit is the core processing module in the model training platform responsible for handling model training requests. After receiving the request forwarded by the training interface, it executes each step of the model training method. Optionally, the first response unit may include sub-units such as an initial sample set acquisition module, a sampling weight calculation module, a target sample set construction module, and a model training module. These sub-units work together to complete the entire training process from the initial sample set to the target inference model.

[0266] This statement describes the model training platform provided in the embodiments. It receives model training requests through a training interface and executes customized model training methods by the first response unit. This effectively improves the training efficiency and inference quality of the inference model, enabling the trained target inference model to dynamically adjust the inference process according to the complexity of the problem, avoiding overthinking and underthinking. Thus, it achieves a good balance between inference efficiency and accuracy in scenarios with limited inference resources, maximizing the use of limited computing resources and meeting the dual requirements of inference speed and accuracy in practical application scenarios.

[0267] The above is an illustrative scheme of a model training platform according to this embodiment. It should be noted that the technical solution of this model training platform and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training platform, please refer to the description of the technical solution of the model training method described above.

[0268] Corresponding to the above method embodiments, this specification also provides a text reasoning platform embodiment. Figure 5 shows a schematic diagram of the structure of a text reasoning platform provided in one embodiment of this specification. As shown in Figure 5, the platform includes a reasoning interface 502 and a second response unit 504.

[0269] Inference interface 502 is used to receive text inference requests;

[0270] The second response unit 504, in response to the text reasoning request, obtains the target question text; inputs the target question text into the target reasoning model to obtain the target answer text, wherein the target reasoning model is trained based on the model training method described above.

[0271] The text reasoning platform is a systematic technical architecture based on a customized training framework. It performs reasoning tasks targeting a reasoning model by receiving the target question text and using a customized, trained target reasoning model to generate a reasoning result appropriate to the question's complexity. The platform can adaptively adjust the reasoning process based on the complexity of the target question text in scenarios with limited reasoning resources, such as mobile devices, embedded systems, or edge computing devices. It employs concise reasoning paths for simple questions and detailed reasoning paths for complex questions, thus maximizing the use of limited computing resources while ensuring reasoning accuracy, achieving a good balance between reasoning efficiency and accuracy.

[0272] The inference interface is a system interface used to receive text inference requests. As the entry point for the text inference platform to interact with external users or systems, it can support a variety of communication protocols and data formats, such as HTTP / HTTPS or gRPC, enabling external systems to submit inference tasks through standard network requests.

[0273] A text reasoning request is a structured request containing the parameters and data required for a reasoning task, used to submit a reasoning task to a text reasoning platform. Specifically, a text reasoning request may include key information such as the content of the target problem text, the format requirements for the reasoning results, and the expected level of detail in the reasoning process.

[0274] The second response unit is the core processing module in the text reasoning platform responsible for handling text reasoning requests. After receiving the request forwarded by the reasoning interface, it executes each step of the reasoning task. Optionally, the second response unit may include sub-units such as a target question text acquisition module, a target reasoning model loading module, a reasoning execution module, and a target answer text generation module. These sub-units work together to complete the entire reasoning process from the target question text to the target answer text.

[0275] The text reasoning platform provided in this specification receives text reasoning requests and the second response unit executes the reasoning task, enabling the target reasoning model to adaptively adjust the reasoning process according to the complexity of the target question text. It generates concise reasoning results for simple questions and solutions containing detailed reasoning steps for complex questions. This maximizes the use of limited computing resources while ensuring reasoning accuracy, achieving a good balance between reasoning efficiency and accuracy. It enables the reasoning model to run efficiently on low-power, low-memory devices, meeting the dual requirements of reasoning speed and accuracy in practical application scenarios.

[0276] The above is an illustrative scheme of a text reasoning platform according to this embodiment. It should be noted that the technical solution of this text reasoning platform and the technical solution of the text reasoning method described above belong to the same concept. For details not described in detail in the technical solution of the text reasoning platform, please refer to the description of the technical solution of the text reasoning method described above.

[0277] Figure 6 shows a structural block diagram of a computing device 600 according to one embodiment of this specification. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0278] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0279] In one embodiment of this specification, the aforementioned components of the computing device 600, as well as other components not shown in FIG. 6, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 6 is merely illustrative and not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0280] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0281] The processor 620 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above method.

[0282] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above method.

[0283] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0284] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the method described above belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the method described above.

[0285] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0286] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the method described above.

[0287] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0288] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0289] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0290] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0291] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, comprising: The process involves obtaining an initial sample set, a student reasoning model, and reasoning ability labels for the student reasoning model. The initial sample set includes multiple sample thought chains, generated by the teacher reasoning model using different reasoning methods. Each sample thought chain is labeled with a reasoning method label and a reasoning complexity label corresponding to the given reasoning method. The reasoning model is used to perform text reasoning tasks based on input question text. Based on the deviation between the reasoning ability labels of the student reasoning model and the reasoning method labels labeled on the multiple sample thought chains, the sampling weights of the multiple sample thought chains are determined. Based on the sampling weights, a target sample set is obtained by sampling from the initial sample set. Based on the target sample set, the student reasoning model is trained to obtain a target reasoning model. In the target sample set, each sample thought chain also includes a corresponding sample question text. Training the student reasoning model based on the target sample set to obtain the target reasoning model includes: using the student reasoning model to perform text reasoning tasks based on the sample question text. The process involves: performing reasoning to obtain the predicted thought chain generated by the student reasoning model; using a reasoning method reward model, determining the reasoning method reward score based on the predicted thought chain generated by the student reasoning model, and using a reasoning result reward model, determining the reasoning result reward score based on the predicted thought chain generated by the student reasoning model; determining the reasoning method deviation based on the reasoning method reward score and the reasoning method label range, and determining the reasoning result deviation based on the reasoning result reward score and the reasoning complexity label range, where the reasoning method label range is set according to the target reasoning ability of the target reasoning model, and the reasoning complexity label range is set according to the target output result of the target reasoning model; determining the reasoning method reward signal based on the reasoning method deviation, and determining the reasoning result reward signal based on the reasoning result deviation; determining the reward signal based on the reasoning method reward signal and the reasoning result reward signal, and training the student reasoning model based on the reward signal to obtain the target reasoning model.

2. The method according to claim 1, wherein each sample thought chain is labeled with a reasoning complexity label corresponding to the reasoning result; the determination of the sampling weight of the multiple sample thought chains based on the deviation between the reasoning ability label of the student reasoning model and the reasoning method label of the multiple sample thought chains includes: Calculate multiple first label deviations between the reasoning ability label of the student reasoning model and the reasoning method label of the multiple sample thought chain annotations; Based on the multiple first label biases, the first distribution weights of the labels in the inference method are determined; Calculate multiple second label deviations between the reasoning method labels and the reasoning complexity labels of the multiple sample thought chains; based on the multiple second label deviations, determine the second distribution weight of the correlation between the reasoning method labels and the reasoning complexity labels; Based on the first distribution weight and the second distribution weight, the sampling weights of the multiple sample thought chains are determined.

3. The method according to claim 2, wherein determining the first distribution weight of the inference method label based on the plurality of first label biases includes: The largest first label deviation is determined from the plurality of first label deviations, and the total first label deviation of the plurality of first label deviations is calculated; Based on the maximum first label deviation and the total first label deviation, the first distribution weight of the label in the inference method is determined; The second distribution weight for determining the correlation between the inference method label and the inference complexity label based on the plurality of second label deviations includes: determining the maximum second label deviation from the plurality of second label deviations, and calculating the total second label deviation of the plurality of second label deviations; Based on the maximum second label deviation and the total second label deviation, a second distribution weight is determined for the correlation between the inference method label and the inference complexity label.

4. The method according to any one of claims 1-3, wherein training the student reasoning model based on the target sample set to obtain the target reasoning model includes: Based on the target sample set, the student reasoning model is subjected to supervised fine-tuning training to obtain the target reasoning model.

5. The method according to claim 4, wherein in the target sample set, any sample thought chain further includes the corresponding sample question text; the step of supervising and fine-tuning the student reasoning model based on the target sample set to obtain the target reasoning model includes: Using the student reasoning model, reasoning is performed based on the sample question text to obtain the predictive thought chain generated by the student reasoning model; Based on the predicted thought chain and the sample thought chain, the thought chain loss value is determined; based on the thought chain loss value, the student reasoning model is trained to obtain the target reasoning model.

6. The method according to claim 1, further comprising, before determining the reward signal based on the inference method reward signal and the inference result reward signal: Based on the logic of the predictive thinking chain and the sample thinking chain, a correctness reward signal is determined, and based on the thinking chain format of the predictive thinking chain and the sample thinking chain, a format reward signal is determined. The step of determining the reward signal based on the reasoning method reward signal and the reasoning result reward signal includes: determining the reward signal based on the reasoning method reward signal, the reasoning result reward signal, the correctness reward signal, and the format reward signal.

7. A textual reasoning method, comprising: Obtain the target question text; The target question text is input into the target reasoning model to obtain the target answer text, wherein the target reasoning model is trained based on the model training method described in any one of claims 1-6.

8. A model training platform apparatus, comprising a training interface and a first response unit: the training interface is configured to receive a model training request; the first response unit, in response to the model training request, executes the model training method as described in any one of claims 1-6 to obtain a target inference model, wherein, The inference model is used to perform text reasoning tasks based on the input question text.

9. A text reasoning platform apparatus, comprising a reasoning interface and a second response unit: the reasoning interface is used to receive a text reasoning request; the second response unit, in response to the text reasoning request, acquires a target question text; inputs the target question text into a target reasoning model to obtain a target answer text, wherein, The target reasoning model is obtained by training based on the model training method described in any one of claims 1-6.

10. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 6.

11. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

12. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images