Training method and device of reasoning model, equipment, medium and product

Abstract

The application discloses a training method and device of a reasoning model, equipment, a medium and a product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring sample data, wherein the sample data is used for describing task requirements of a reasoning task; generating a first answer sequence for replying to the sample data by using a first model, and generating a second answer sequence for replying to the sample data by using a second model; acquiring at least two selected text word elements from a plurality of first text word elements based on prediction probabilities corresponding to the plurality of first text word elements; and training the second model based on the prediction probabilities corresponding to the at least two selected text word elements and prediction probabilities corresponding to a plurality of second text word elements, so as to obtain a trained second model, wherein the trained second model is used for executing the reasoning task. The prediction probabilities corresponding to the at least two selected text word elements are used to train the second model, the generation preferences of the first model are learned, and the training efficiency of the reasoning model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, medium and product for training a reasoning model. Background Technology

[0002] Inference models are large language models used to perform complex tasks. For inference tasks such as mathematical problem solving, code generation, and logical reasoning, inference models generate multi-step, multi-solution paths, and interpretable inference results.

[0003] In related technologies, the training process of the inference model includes: generating output results corresponding to sample data through a pre-trained teacher model, and training the student model based on the output results as labels to obtain the inference model.

[0004] However, in the above-mentioned training methods for inference models, using the output of the teacher model as a label to supervise the training of the student model has a high degree of limitation in the way of supervising training, and the training efficiency of the inference model is low. Summary of the Invention

[0005] This application provides a training method, apparatus, device, medium, and product for an inference model. The technical solution provided by this application includes the following aspects.

[0006] According to one aspect of the embodiments of this application, a method for training an inference model is provided, the method comprising: Acquire sample data, which is used to describe the task requirements of the reasoning task; A first answer sequence for responding to the sample data is generated by a first model, and a second answer sequence for responding to the sample data is generated by a second model; wherein, the first sequence position of the first answer sequence corresponds to a plurality of first text words, and the first text words are associated with the predicted probabilities corresponding to the first text words; the first sequence position of the second answer sequence corresponds to a plurality of second text words, and the second text words are associated with the predicted probabilities corresponding to the second text words. Based on the predicted probabilities corresponding to the plurality of first text words, at least two selected text words are obtained from the plurality of first text words; Based on the predicted probabilities corresponding to the at least two selected text words and the predicted probabilities corresponding to the plurality of second text words, the second model is trained to obtain the trained second model, which is used to perform inference tasks.

[0007] According to one aspect of the embodiments of this application, a training apparatus for an inference model is provided, the apparatus comprising: The acquisition module is configured to acquire sample data, which is used to describe the task requirements of the inference task. The generation module is configured to generate a first answer sequence for responding to the sample data using a first model, and to generate a second answer sequence for responding to the sample data using a second model; wherein, the first sequence position of the first answer sequence corresponds to a plurality of first text words, and the first text words are associated with the predicted probabilities corresponding to the first text words; the first sequence position of the second answer sequence corresponds to a plurality of second text words, and the second text words are associated with the predicted probabilities corresponding to the second text words. The acquisition module is further configured to acquire at least two selected text words from the plurality of first text words based on the predicted probabilities corresponding to the plurality of first text words respectively; The training module is configured to train the second model based on the prediction probabilities corresponding to the at least two selected text words and the prediction probabilities corresponding to the plurality of second text words, thereby obtaining the trained second model, which is used to perform inference tasks.

[0008] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described training method for the inference model.

[0009] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described training method for the inference model.

[0010] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, wherein a processor reads from the computer-readable storage medium and executes the computer program to implement the above-described training method for the inference model.

[0011] The technical solutions provided in this application can bring at least the following beneficial effects: By determining at least two selected text words based on the predicted probabilities of multiple first text words generated by the first model during the generation of the first answer sequence, and training the second model based on the predicted probabilities of the at least two selected text words and the predicted probabilities of multiple second text words, the bias of the first model in generating the answer sequence is preserved, and the basic accuracy of the trained second model in subsequent training processes is improved, thereby improving the execution accuracy of the reasoning task. Furthermore, by extracting selected text words, the amount of data to be processed during the training of the second model is reduced, the consumption of computing resources is reduced, and the training efficiency of the reasoning model is improved. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a training system for an inference model provided in an exemplary embodiment of this application; Figure 2 This is a flowchart of a training method for an inference model provided in an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating the acquisition of the Top-K selected words according to an exemplary embodiment of this application; Figure 4 This is a flowchart of a training method for an inference model provided in another exemplary embodiment of this application; Figure 5 This is a schematic diagram of a training process based on the probability distributions corresponding to the first model and the second model, respectively, provided in an exemplary embodiment of this application. Figure 6 This is a flowchart of a training method for an inference model provided in yet another exemplary embodiment of this application; Figure 7 This is a schematic diagram of the inference model training process provided in an exemplary embodiment of this application; Figure 8 This is a structural block diagram of a training apparatus for an inference model provided in an exemplary embodiment of this application; Figure 9 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0014] First, the terms used in the embodiments of this application will be introduced.

[0015] Reasoning model: refers to a large language model used to perform complex Natural Language Processing (NLP) tasks. For tasks such as solving mathematical problems, generating code, and logical reasoning, it generates multi-step, multi-solution paths, interpretable reasoning processes and corresponding answer sequences for the task.

[0016] Hard labels: These are the encoded labels corresponding to the lexical units generated by the large language model. For example, if the lexical unit generated by the large language model is the digit "3", the hard label for that lexical unit would be [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]. The hard label only includes the binary information used to indicate the lexical unit. Optionally, for a pre-trained large language model, the hard labels corresponding to the lexical units are stored as text in a readable dataset. Accordingly, the hard label resources corresponding to the large language model can be directly obtained by accessing this dataset.

[0017] Model distillation is a training process for inference models that transfers knowledge from a complex teacher model to a lightweight student model. Model distillation guides the training of the student model using the results generated by the teacher model, enabling the student model to approximate the generalization ability of the teacher model while maintaining lower computational costs. The teacher model has a large number of parameters, a complex structure, and has undergone extensive training; in contrast, the student model has a simpler structure and fewer parameters. Based on model distillation, the student model offers faster inference speed, lower memory usage, and higher accuracy in inference tasks compared to the teacher model.

[0018] In related technologies, the training process of the inference model involves generating high-probability words corresponding to sample data using a pre-trained teacher model. These high-probability words are then used as hard labels to fine-tune the model parameters of the student model. The student model, with its finely tuned parameters, is then trained using reinforcement learning to sample the answer paths corresponding to the sample data, thus obtaining the inference model. Specifically, the answer sequence generated by the teacher model for responding to the sample data includes at least one sequence position, with the same sequence position corresponding to multiple text words. A high-probability word is the word with the highest predicted probability among these multiple text words. The answer path includes the answer sequence corresponding to the inference task and the inference process.

[0019] Optionally, when the inference model is implemented as a bidirectional encoder representation from transformers (BERT) model, the parameters of the student model are fine-tuned based on the full generation results of the teacher model, wherein the full generation results include multiple text units corresponding to each sequence position in the answer sequence; when the inference model is implemented as a transformer model, the parameters of the student model are fine-tuned based on the highest probability generation results corresponding to the teacher model, wherein the highest probability generation results are the aforementioned high probability units.

[0020] However, in the training methods of the aforementioned inference models, the teacher model knowledge corresponding to hard labels only corresponds to high-probability words, which limits the sampling of answer paths during training. This results in low accuracy of the sampled answer paths and low training efficiency for the inference model. In related technologies, when obtaining the student model after parameter fine-tuning during the model distillation stage, the process of sampling answer paths corresponding to subsequent reinforcement learning data and training the model is not considered, making it impossible to balance the quality of sampling results and training efficiency.

[0021] For example, the limitations of sampling the answer path during the training process of the above model include, but are not limited to, at least one of the following situations.

[0022] 1. Hard labels only provide the correctness of the word corresponding to the sequence position, and cannot reflect the rationality, validity, or consistency of the local logic in the reasoning process. For example, when the word generated by the reasoning model is the number "3", the hard label corresponding to the word is [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]. That is, the information that can be obtained based on the hard label is only the number "3" corresponding to the word, and lacks information such as the possibility of the number "3" corresponding to the word, the process of reasoning to obtain the number "3" corresponding to the word, or whether the number "3" corresponding to the word contradicts the generation results of other sequence positions.

[0023] 2. Methods based on hard labels for training inference models cannot distinguish between different incorrect answer paths, such as those containing logical fallacies, calculation errors, or irrelevant steps. They also cannot identify some correct answer paths, such as those with correct reasoning directions but errors in the details of the reasoning process. This causes the training of the inference model to ignore the interpretability of the reasoning process, or to address different answer preferences in different application scenarios. For example, although the answer sequence generated by the inference model is correct, the reasoning process may be logically unclear or difficult to understand.

[0024] 3. For reasoning tasks with multiple answer sequences or reasoning processes, hard labels can only match a single answer, which leads to the training process of the reasoning model suppressing the diversity of answer paths and ignoring the possibility of low answer generation efficiency in the actual execution of the reasoning task; where each answer sequence is reasonable and each reasoning process is effective or equivalent.

[0025] Optionally, when the full generation results of the teacher model are used to fine-tune the parameters of the student model, the full generation results serve as the basis for training the student model. The amount of data is large, the computational complexity of the training process is high, and the computational resources required for training increase significantly as the vocabulary of the reasoning task increases. For the lightweight student model, the training cost is too high, the advantages of model distillation cannot be reflected, and the model training efficiency is low.

[0026] Please refer to Figure 1 This illustration shows a schematic diagram of a training system for an inference model provided in one embodiment of this application. The computer system 100 includes: a terminal 120, or a terminal 120 and a server 140.

[0027] The training method for the inference model provided in this application embodiment can be executed independently by the terminal 120, independently by the server 140, or jointly by the terminal 120 and the server 140; no limitation is made here. In some embodiments, the computer system 100 can implement the system architecture of the training method for the inference model.

[0028] The device type of terminal 120 includes at least one of the following: smartphone, laptop, desktop computer, tablet computer, smart robot, augmented reality (AR) device, virtual reality (VR) device, vehicle terminal, wearable device, etc.

[0029] Terminal 120 is connected to server 140 via a wireless network or a wired network.

[0030] Those skilled in the art will understand that the number of the aforementioned devices can be more or less. For example, there may be only one device, or there may be dozens or hundreds of devices, or even more. This application does not limit the number or type of devices.

[0031] Server 140 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 140 provides functional services for implementing model training. Optionally, server 140 undertakes the main computing work, and terminal 120 undertakes the secondary computing work; or, server 140 undertakes the secondary computing work, and terminal 120 undertakes the main computing work; or, server 140 and terminal 120 collaborate on computing using a distributed computing architecture.

[0032] In one example, the training method of the inference model is implemented collaboratively by terminal 120 and server 140. Illustratively, sample data is acquired through terminal 120, where the sample data describes the task requirements of the inference task. Server 140 deploys a first model and a second model. Server 140 receives the sample data acquired by terminal 120, thereby generating a first answer sequence for responding to the sample data through the first model, and a second answer sequence for responding to the sample data through the second model. For example, if the inference task corresponding to the sample task is to perform addition, i.e., the sample data is the text prompt "Give the answer 2+3.", the first answer sequence generated by the first model is "2+3=5", and the second answer sequence generated by the second model is "The answer to 2+3 is 1".

[0033] In the example above, the first answer sequence and the second answer sequence are implemented as text sequences. The first character position of the first answer sequence corresponds to multiple first text words as candidate characters, for example, the next character position of "2+3=" corresponds to the numbers "0" to "9". The first character of the second answer sequence corresponds to multiple second text words as candidate characters, for example, the next character position of "2+3 is" corresponds to the numbers "0" to "9". Server 140, based on the prediction probabilities corresponding to the multiple first text words (e.g., the prediction probability of the number "5" is 0.8, the prediction probability of the number "1" is 0.05, the prediction probability of the number "4" is 0.03, etc.), obtains at least two selected text words from the multiple first text words, for example, using the numbers "5" and "1" as selected text words.

[0034] In the example above, server 140 trains a second model based on the prediction probabilities corresponding to at least two selected text words and the prediction probabilities corresponding to multiple second text words, and obtains the trained second model, which is used to perform inference tasks.

[0035] Optionally, server 140 sends the trained second model to terminal 120, and terminal 120 performs an inference task using the trained second model to obtain the generated result corresponding to the inference task; or, server 140 performs an inference task using the trained second model, obtains the generated result corresponding to the inference task, and sends the generated result to terminal 120, which receives the generated result. The trained second model can undergo further training processing, such as training with reinforcement learning (RL) to improve its generalization ability in the vertical domain.

[0036] It is worth noting that the aforementioned terminal 120 refers to an electronic device with output display capabilities and input control capabilities. The aforementioned server 140 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0037] Cloud technology refers to a managed technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0038] In some embodiments, the server 140 described above can also be implemented as a node in a blockchain system.

[0039] Please refer to Figure 2 This document illustrates a flowchart of a training method for an inference model provided in one embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 1 The method is executed by either the terminal 120 or the server 140 shown, or it is executed jointly by the terminal and the server. In this embodiment, the method is executed by the server as an example. Figure 2 As shown, the method may include at least one of the following steps 210 to 240.

[0040] Step 210: Obtain sample data.

[0041] Sample data is used to describe the task requirements of the inference task. Sample data is the training data required by the inference model during the training process, and it corresponds to the inference task used for training. The inference task is given in the form of a question or task requirement, which the inference model needs to generate a corresponding result, i.e., a sequence of answers. The generated result needs to be able to answer the corresponding question, or satisfy the task requirements. Illustratively, the inference task includes, but is not limited to, at least one of the following tasks.

[0042] 1. Logical Reasoning Tasks: These tasks involve deriving logically valid conclusions based on pre-defined premises and rules, using methods such as deduction, induction, or analogy. Examples include common-sense reasoning, causal reasoning, multi-step knowledge reasoning, truth-or-false judgment reasoning, and symbolic and rule-based reasoning. Correspondingly, the logical reasoning process includes, but is not limited to, proving specific propositions, judging the validity of inferences based on premises and assumptions, judging whether hypotheses are supported based on experimental conditions and data, explaining the causes of specific phenomena, and applying abstract rules to specific scenarios to solve problems.

[0043] 2. Mathematical Reasoning Tasks: These refer to solving problems or verifying conjectures based on numerical values, symbols, and mathematical relationships, using methods such as algebraic transformations, geometric deduction, or numerical analysis to arrive at accurate mathematical results. Examples include solving mathematical word problems, solving equations, proving theorems, and solving complex mathematical problems step-by-step. Correspondingly, the aforementioned mathematical reasoning process includes, but is not limited to, deriving unknowns from known conditions, proving the truth or falsity of mathematical propositions, seeking optimal solutions based on constraints, and simulating and predicting mathematical models.

[0044] 3. Code Generation Tasks: This refers to converting task requirements, such as natural language descriptions, formal specifications, or high-level design intentions, into executable code programs or code snippets, following the syntax and semantic rules of a specific programming language. Examples include generating functions based on functional descriptions, troubleshooting code errors, completing code based on test cases, translating or refactoring code between different syntaxes based on specific code snippets, etc. Accordingly, the above code generation process includes, but is not limited to, parsing task requirements into algorithmic steps, establishing control and data structures based on logical relationships, establishing secure operation mechanisms, reducing code complexity, and improving code readability.

[0045] 4. Affective Reasoning Tasks: These tasks involve identifying, analyzing, and inferring the emotional states, intentions, or subjective evaluations represented or implied by multimodal input data, such as text, speech, or vision. Examples include tasks in human-computer dialogue and public opinion analysis, such as affective classification, stance detection, understanding irony or metaphor, and emotion analysis. Accordingly, the affective reasoning process includes, but is not limited to, extracting implicit affective keywords from contextual expressions, inferring implicit emotions by combining context and common sense, analyzing the causes and intensity changes of emotions or attitudes, predicting potential subsequent behaviors triggered by emotions, or providing decision-making suggestions.

[0046] It is worth noting that the above classification of reasoning tasks is merely illustrative, and the embodiments of this application do not limit the form and content of reasoning tasks. Optionally, the task types of the different reasoning tasks mentioned above may overlap. For example, a problem in the field of mathematics that needs to be proven through logical deduction is both a logical reasoning task and a mathematical reasoning task.

[0047] Sample data is used to describe the problem or task requirements corresponding to the reasoning task. The methods and approaches for obtaining sample data vary in different application scenarios. Optionally, the methods for obtaining sample data include, but are not limited to, at least one of the following.

[0048] 1. Data acquisition modules or device sensors directly collect the user's physical state or required real-time information, such as data collected from real-world scenarios, manually collected or labeled input data, and real-time data acquired via the network. For example, if the purpose of training the inference model is to solve inference tasks related to autonomous driving, during test driving, real-time camera image data or radar point cloud data around the vehicle are collected as sample data to train and update the inference model in real time to adapt to changing vehicle driving environments.

[0049] 2. Existing data files are actively provided or selected by the user or the system; such as historical model-generated data, open-source data, etc. For example, sample data can be uploaded to the task prediction system from local or network storage; historical sample data, including but not limited to historical inference tasks, user information, and operation logs, can be automatically loaded by the system. For instance, if the purpose of training the inference model is to solve inference tasks related to solving mathematical problems, the sample data is used to describe the problem corresponding to the mathematical inference task. Existing, open-source sample datasets are obtained through network downloads or other means as sample data, and the inference model is trained based on this sample data.

[0050] In some embodiments, the sample data is implemented as multimodal data, where multimodal refers to multiple data types. Multimodal data describes the task requirements of the reasoning task based on different physical sources and perceptual dimensions. Multimodal data includes data fragments corresponding to multiple data modalities, which can be implemented as image, audio, text, video, etc. For example, the sample data includes a geometric diagram and the text prompt "Based on the conditions shown in the diagram, solve for the degree of angle 1 in the diagram". In the above case, the sample data combines image data with text data to describe the task requirements of the reasoning task. Optionally, the multimodal data includes, but is not limited to, at least two of the following: 1. Visual sample data: including image data, video data, 3D model data, etc.; 2. Text sample data: including natural language text, code, structured data, etc., presented in the form of symbolic sequences; 3. Auditory sample data: including waveform audio such as speech and voiceprints, and symbolic audio such as musical scores; 4. Physical sample data: including sensor signal data such as pressure and texture, and motion data such as acceleration; 5. Spatiotemporal sample data: including time series, spatial coordinates, etc.; Optionally, the sample data includes at least two of the above-mentioned multimodal data.

[0051] In some embodiments, the sample data is implemented in the form of a sample dataset. The sample dataset includes the task requirements and expected execution results corresponding to at least two reasoning tasks; that is, the sample dataset includes the questions corresponding to at least two reasoning tasks and the sequence of answers to those questions. Illustratively, when the reasoning task is implemented as a mathematical reasoning task, the sample dataset can be implemented as a high school math competition (MATH) dataset, or an 8,000-question elementary school math dataset (GSM8K).

[0052] It is worth noting that the above-mentioned implementation of the sample data is merely an illustrative example, and the specific implementation of the sample data in this application embodiment is not limited.

[0053] Step 220: Generate a first answer sequence for responding to the sample data using the first model, and generate a second answer sequence for responding to the sample data using the second model.

[0054] In this sequence, the first sequence position of the first answer sequence corresponds to multiple first text words, and each first text word is associated with a predicted probability. Similarly, the first sequence position of the second answer sequence corresponds to multiple second text words, and each second text word is associated with a predicted probability. The predicted probability of the first text word is obtained during the generation of the first answer sequence and represents the confidence that the first text word is the result word at the first sequence position of the first answer sequence. The predicted probability of the second text word is obtained during the generation of the second answer sequence and represents the confidence that the second text word is the result word at the first sequence position of the second answer sequence. Schematic, the first text word is a candidate word at the first sequence position of the first answer sequence, and the second text word is a candidate word at the first sequence position of the second answer sequence. In the process of generating the answer sequence using the inference model, the first sequence position of the answer sequence corresponds to multiple text words as candidate words. In the above process, the inference model is implemented as a first model, the answer sequence as a first answer sequence, and the text words as first text words; or, the inference model is implemented as a second model, the answer sequence as a second answer sequence, and the text words as second text words. In this embodiment, the process of generating the first answer sequence using the first model is used as an example for explanation.

[0055] In some embodiments, the answer sequence is implemented in text form and is used to respond to the sample data corresponding to the reasoning task. Based on obtaining the answer sequence, it is converted into natural language text output. Text units are the smallest data processing units in the model processing process and are the basic input and output units of large language models. The implementation forms of text units include, but are not limited to, at least one of the following: single characters, words, phrases, word groups, numbers, or symbols.

[0056] The answer sequence includes at least one sequence position, and each sequence position corresponds to multiple text words as candidate words. The sequence position refers to the index of the text word corresponding to the answer sequence within the answer sequence; that is, the sequence position indicates the order of the text word in the answer sequence. Illustratively, in the final output of the inference model to be trained, the generated answer sequence is a sequence of words consisting of at least one result word. That is, there is a one-to-one correspondence between the at least one sequence position corresponding to the generated result and the at least one result word. The result word is determined from multiple text words; optionally, the result word is the word with the highest predicted probability among the multiple text words.

[0057] For example, taking a mathematical reasoning task, if the sample data is implemented as the text prompt "Xiaoming has 5 apples, ate 2, and then bought 3 more, how many apples does he have now?", the generated result obtained by the reasoning model is the answer sequence "Xiaoming now has 6 apples." The multiple result words corresponding to the answer sequence include "Xiaoming," "now," "has," "6," "apples," and ".". Correspondingly, the sequence positions include positions 1 to 7, each corresponding to a specific result word. During the process of generating the answer sequence by the reasoning model, multiple text words corresponding to each sequence position are used as candidate words; for example, the multiple text words corresponding to position 4 include, but are not limited to, "6," "4," and "8."

[0058] In some embodiments, generating a sequence of answers to respond to sample data using a reasoning model to be trained includes, but is not limited to, at least one of the following steps.

[0059] 1. Input sample data into the inference model to be trained, and encode the sample data to obtain an input sequence. The sample data describes the task requirements of the inference task, and can be implemented as data in at least one modality. Illustratively, if the sample data includes text data, the text data is converted into a text input sequence; if the sample data includes image data, the image data is converted into an image input sequence. Optionally, if the sample data includes both text and image data, the image input sequence is concatenated with the text input sequence to obtain a multimodal fusion sequence as the input sequence.

[0060] In some embodiments, text data is divided using a word segmentation algorithm to obtain at least two text words; the at least two text words are encoded using a text encoder to obtain text feature vectors; the at least two text feature vectors form the aforementioned text input sequence. Optionally, the aforementioned word segmentation algorithm includes, but is not limited to, the maximum matching algorithm, the Jieba word segmentation algorithm, etc.; the aforementioned text encoder includes, but is not limited to, the Word to Vector (Word2Vec) model, the Global Vectors (GloVe) model, etc.

[0061] The process of obtaining the text feature vector corresponding to the text data includes, but is not limited to: preprocessing the text data to obtain at least two text words, including, but not limited to, word segmentation and normalization (such as unifying capitalization); mapping the text words to discrete values, such as establishing a vocabulary, which is used to store the mapping relationship between text words and indices; and converting the discrete values ​​corresponding to the text words into text feature vectors through mapping methods such as word embedding matrices.

[0062] In some embodiments, an image encoder is used to segment the image data in the sample data to obtain at least two image patches; based on the at least two image patches, the image encoder obtains the image feature vectors corresponding to the at least two image patches respectively, and the at least two image feature vectors constitute the above-mentioned image input sequence. Optionally, the image encoder is implemented as a pre-trained Transformer visual backbone, such as a Vision Transformer (ViT).

[0063] The process of obtaining image feature vectors corresponding to at least two image patches includes, but is not limited to, color normalization of the image patches, such as adjusting the pixel values ​​corresponding to the image patches to eliminate systematic deviations such as device differences; converting the pixel values ​​of each pixel in the image patch into numerical vectors; obtaining the overall feature vector corresponding to the image patch through nonlinear transformation based on the numerical vectors corresponding to each pixel in the image patch and the positional relationship between pixels; such as merging the numerical vectors corresponding to each pixel in the image patch into a one-dimensional vector, and adding positional information to the one-dimensional vector corresponding to the image patch. This application does not limit the process of obtaining image feature vectors corresponding to at least two image patches in its embodiments.

[0064] It is worth noting that the above-described encoding process for different modal sample data does not limit the temporal order of the processing steps. Optionally, when the sample data includes both text and image data, the image input sequence is projected onto the text vector space and concatenated with the text input sequence to obtain a multimodal fusion sequence as the input sequence.

[0065] 2. The inference model generates multiple result words based on the input sequence using autoregression, and obtains the answer sequence based on these multiple result words. Illustratively, the process of generating multiple result words is as follows: the inference model generates multiple text words as candidate words, each corresponding to a first sequence position in the answer sequence; the Transformer decoder determines the first result word among the multiple text words corresponding to the first sequence position, which corresponds to the first sequence position in the answer sequence; based on the first result word, the logical values ​​corresponding to the multiple text words at the next sequence position are calculated, and the next result word is determined based on these logical values. The logical values ​​indicate the prediction result for the text word, and are the original prediction scores generated by the inference model corresponding to the prediction probability. The prediction probability is the confidence level converted from the original prediction scores to the range [0, 1]. The above steps are repeated to obtain the next result word, which is then added to the next sequence position in the answer sequence until a complete answer sequence is generated. Optionally, the convergence condition for generating a complete answer sequence includes, but is not limited to, the answer sequence reaching a preset length threshold, or the generation result corresponding to the next result word indicating that the generation process has stopped.

[0066] In some embodiments, the process of determining the next result word based on logical values ​​includes, but is not limited to: obtaining the predicted probabilities of multiple text words based on the logical values ​​corresponding to the multiple text words corresponding to the next sequence position through normalization processing, such as the softmax function; and determining the next result word among the multiple text words corresponding to the next sequence position through a decoding strategy.

[0067] It is worth noting that the above process for obtaining the answer sequence is only an illustrative example, and the specific method of generating the answer sequence is not limited in the embodiments of this application.

[0068] Step 230: Based on the predicted probabilities corresponding to the multiple first text words, at least two selected text words are obtained from the multiple first text words.

[0069] A first answer sequence for responding to sample data is generated using a first model, wherein a first sequence position in the first answer sequence corresponds to multiple first text words as candidate words. In some embodiments, multiple first text words are generated synchronously using the first model, each first text word including its corresponding text content and a corresponding predicted probability. The predicted probability indicates the likelihood of the first text word appearing at a first sequence position in the answer sequence. Optionally, the predicted probability is obtained based on the logical value corresponding to the first text word.

[0070] In some embodiments, at least two selected text words are those that best meet the task requirements of the reasoning task corresponding to the sample data, which are the first text words that may appear at the first sequence position of the first answer sequence. This reflects the generation preference of the first model for the first sequence position during the generation of the first answer sequence. Based on the prediction probabilities corresponding to multiple first text words, at least two selected text words are obtained from the multiple first text words. The prediction probabilities corresponding to the at least two selected text words satisfy a preset requirement, which indicates the selection strategy among the multiple first text words for the first sequence position of the first answer sequence. Optionally, obtaining at least two selected text words whose prediction probabilities satisfy the preset requirement from the multiple first text words includes, but is not limited to, at least one of the following methods.

[0071] 1. Based on multiple first text words, obtain a word sequence, which includes multiple first text words; obtain K selected text words from the word sequence, where K is a positive integer, and the prediction probability corresponding to each of the K selected text words is higher than or equal to the prediction probability corresponding to any other text word in the word sequence.

[0072] Obtaining the K selected text words includes, but is not limited to, at least one of the following: Arranging multiple first text words in descending order of predicted probability to obtain a word word sequence; obtaining the first K selected text words of the word word sequence, where K is a positive integer. Alternatively, arranging multiple first text words in ascending order of predicted probability to obtain a word word sequence; obtaining the last K selected text words of the word word sequence, where K is a positive integer. Alternatively, traversing the word word sequence including multiple first text words and obtaining the K selected text words with the highest predicted probability, where K is a positive integer.

[0073] For illustrative purposes, please refer to the following: Figure 3 , Figure 3 This is a schematic diagram illustrating the acquisition of the Top-K selected terms according to an exemplary embodiment of this application, such as... Figure 3 As shown, sample data is input into the first model 310, which generates a first answer sequence for responding to the sample data. The first sequence position of the first answer sequence corresponds to a first text word set 320, which includes multiple first text words as candidate words. For example, the number of first text words in the first text word set 320 is n, where n is a positive integer. That is, the first sequence position corresponds to first text word 1 to first text word n, a total of n first text words. For example, the value of K can be an integer in the range [3, 20].

[0074] like Figure 3 As shown, based on the predicted probabilities corresponding to the n first text words in the first text word set 320, at least two selected text words are obtained from the first text word set 320. A word word sequence is obtained based on multiple first text words; optionally, the n first text words in the first text word set 320 are arranged in descending order of predicted probability to obtain a word word sequence. For example, the word word sequence arranged in order is implemented as first text word 21, first text word 32…first text word 6…first text word n. Based on the word word sequence, a selected text word set 330 is obtained, which includes K selected text words in the word word sequence, where K is a positive integer, and the predicted probability corresponding to each of the K selected text words is higher than or equal to the predicted probability corresponding to any other text word in the word word sequence. For example, the selected text word set 330 includes the first K selected text words from the word word sequences of first text word 21, first text word 32, and first text word 6, i.e., the Top-K selected text words.

[0075] It is worth noting that the above-mentioned sequential sequence of lexical elements, namely first text lexical element 21, first text lexical element 32... first text lexical element 6... first text lexical element n, is merely an illustrative example. This application does not limit the numbering and order of the multiple first text lexical elements.

[0076] The second model is trained by obtaining a word sequence based on multiple first text words, for example, by sorting the multiple first text words according to their respective predicted probabilities. K first text words from this sequence are selected as chosen words, meaning at least two chosen words are selected from the multiple first text words. This process is repeated to determine at least two chosen words corresponding to each position in the answer sequence generated by the first model. Based on the predicted probabilities of the Top-K chosen words, model distillation is used to train the second model. This retains the preferences of the first model when generating the answer sequence while reducing computational complexity, storage space, and computational resources, thus improving the training efficiency of the inference model.

[0077] In some embodiments, before obtaining K selected text words in the word sequence, the method further includes: obtaining the task type of the inference task; and obtaining a value of K based on the task type. The value of K can be flexibly adjusted according to the inference task. Different values ​​of K can be set according to the exploration breadth requirements of different inference tasks, adapting to the preset requirements of multiple types and multi-path inference tasks in the process of determining selected text words, thus improving the flexibility of the training method and improving the training efficiency of the inference model in different application scenarios. Illustratively, different values ​​of K are obtained according to different task types, including but not limited to at least one of the following methods.

[0078] 1.1 Obtain the mapping relationship between task types and K values ​​in advance. For example, store a mapping table between task types and K values ​​in advance; or, obtain the K values ​​corresponding to different task types through historical training data, and generate a mapping relationship between task types and K values ​​that meet the training requirements in advance based on the historical training data. Optionally, the pre-generated mapping relationship can be implemented as a fitting function, etc.

[0079] 1.2 Real-time acquisition of the value of K corresponding to the task type. For example, during model training, the task type of the inference task corresponding to the sample data used for training is determined, and the value of K is set by the tester; or, the value of K is adaptively adjusted during model training through statistical analysis or other methods; or, historical training data is acquired, which includes different task types and the value of K corresponding to each task type, and a corresponding large language model is pre-trained based on the historical training data, and the value of K corresponding to the task type of the inference task is generated through the large language model during model training.

[0080] It is worth noting that the above process for obtaining the value of K is only an illustrative example, and the embodiments of this application do not limit the method of obtaining the value of K corresponding to the reasoning task.

[0081] To illustrate, the value of K varies depending on the task type. When the task type of a reasoning task indicates that the solution path is broad, the number of answer sequences that can be used to respond to the sample data is large, or the evaluation of the generated results is more subjective, that is, the reasoning task has characteristics such as openness and multiple solutions, then the value of K is larger. Conversely, when the task type of a reasoning task indicates that the solution path is simple, the answer sequence has a more precise standard answer, or the evaluation of the generated results is more objective, that is, the reasoning task has characteristics such as closedness and unique solution, then the value of K is smaller.

[0082] Optionally, when the task type based on reasoning is a logical reasoning task, the logical chain corresponding to the task is clear, the corresponding reasoning answer is clearly right or wrong, and under the constraints of the premises and logical rules, the answer path is clear, and the generated result can be verified. Therefore, the value of K is relatively small, such as K=3. When the task type based on reasoning is a mathematical reasoning task, there may be multiple answers that meet the requirements for a mathematical problem. Therefore, the value of K is relatively large, such as K=5. When the task type based on reasoning is a code generation task, there is no unique answer path, the solution space is wide, and the number of candidate answers is huge. While ensuring the correctness of the code, the evaluation standard of the generated result of the code generation task is subjective. Therefore, the value of K is even larger, such as K=10. Similarly, reasoning tasks can also include, but are not limited to, open-domain question answering, creative writing, brainstorming, and other tasks.

[0083] The answer path includes the reasoning process and answer sequence corresponding to the reasoning task. Optionally, the answer sequence is the generated result corresponding to the reasoning task.

[0084] It is worth noting that the task type of the above reasoning task is subjectively preset. That is, when different reasoning tasks correspond to the same task type, the actual value of K for the reasoning task is different.

[0085] In some embodiments, the value of K is determined based on the breadth of exploration required by the reasoning task. For example, when the reasoning task is implemented as a mathematical reasoning task, the sample dataset can be implemented as the MATH dataset or the GSM8K dataset. The GSM8K dataset includes mathematical problems in different languages ​​and with different content. Each mathematical problem requires at least one basic calculation using basic arithmetic operations to obtain the answer, and the answers to these mathematical problems are implemented in natural language form. Compared to the GSM8K dataset, the mathematical problems in the MATH dataset are more complex and have more diverse forms. For example, solving mathematical problems requires more reasoning steps, and the problem stem may include graphical descriptions, constraints, symbolic formulas, etc., or the process of solving the mathematical problem may require derivation based on pre-defined knowledge such as mathematical formulas. Therefore, the breadth of exploration required for the inference task corresponding to the sample data in the MATH dataset is higher than that required for the inference task corresponding to the sample data in the GSM8K dataset. Accordingly, during the training of the model based on the sample data in the MATH dataset, the value of K is larger than that in the GSM8K dataset. The training process includes, but is not limited to, obtaining at least two selected text words from multiple first text words based on the prediction probabilities corresponding to multiple first text words.

[0086] 2. Obtain a pre-set probability threshold; determine at least two selected text words from multiple first text words based on the probability threshold. The probability threshold is determined based on the predicted probability distribution corresponding to the first text words. Illustratively, obtain the predicted probabilities corresponding to multiple first text words; obtain the statistical distribution characteristics corresponding to the multiple first text words; determine the probability threshold based on the statistical distribution characteristics, thereby obtaining at least two selected text words whose predicted probabilities reach the probability threshold. Optionally, the statistical distribution characteristics include variance and mean.

[0087] It is worth noting that the above method of determining at least two selected text words from multiple first text words is only an illustrative example, and the specific method of obtaining selected text words is not limited in the embodiments of this application.

[0088] Step 240: Based on the predicted probabilities corresponding to at least two selected text words and the predicted probabilities corresponding to multiple second text words, train the second model to obtain the trained second model.

[0089] The trained second model is used to perform inference tasks. In some embodiments, the above-mentioned model distillation is used to learn the prediction probabilities corresponding to at least two selected text words corresponding to the first model, thereby training the second model and obtaining the trained second model.

[0090] The following explanation uses the example of implementing the first model as the teacher model and the second model as the student model. Before training the student model, the student model is first obtained. Optionally, the student model is implemented as an open-source inference model; or, the student model is implemented as a lightweight inference model obtained through pre-trained general training over a historical time period; or, the student model is obtained based on the teacher model.

[0091] Schematic representation: The teacher model's structure includes an input layer that acquires and preprocesses input data, an intermediate processing layer that acquires feature vectors based on the input data, and an output layer that generates the final result based on the feature vectors. The process of acquiring the student model from the teacher model includes, but is not limited to: copying the backbone network responsible for core feature extraction from the intermediate processing layer of the teacher model, and the corresponding model parameters of the backbone network, from the student model; constructing the output layer corresponding to the student model, connecting it to the backbone network, and randomly initializing the model parameters corresponding to the output layer.

[0092] Secondly, the student model is trained to obtain a trained student model capable of performing inference tasks. The process of training the student model is implemented as a supervised fine-tuning (SFT) process; wherein, SFT includes, but is not limited to, at least one of the following steps.

[0093] 1. Input sample data into the teacher model and obtain its output; input sample data into the student model and obtain its output. Illustratively, using the teacher model, based on the sample data, generate the predicted probabilities of at least two generated results corresponding to the teacher model, forming a teacher probability distribution, which serves as the output of the teacher model; optionally, the at least two generated results are selected results that meet preset requirements. Using the student model, based on the sample data, generate the predicted probabilities of at least two generated results corresponding to the student model, forming a student probability distribution, which serves as the output of the student model.

[0094] 2. Based on the outputs of the teacher model and the student model, the model parameters of the student model are fine-tuned multiple times to obtain the trained student model. The fine-tuning process can be implemented in steps 210 to 240 above. Illustratively, the student model is trained on sample data using the model parameters corresponding to the backbone network in the student model and the model parameters corresponding to the randomly initialized output layer; wherein the sample data corresponds to the inference task to be solved by the student model.

[0095] Optionally, during the fine-tuning process, some or all of the model parameters of the student model can be fine-tuned.

[0096] In some embodiments, the above-described multi-round fine-tuning process can be implemented as follows: Based on the teacher's probability distribution and the student's probability distribution, obtain the distribution loss value for the current round, where the distribution loss value indicates the difference between the teacher's probability distribution and the student's probability distribution; train the student model for the current round based on the distribution loss value, and obtain the trained student model when the current training round number corresponding to the second model reaches a preset first training round number threshold. The training of the student model for the current round based on the distribution loss value includes, but is not limited to: obtaining the first gradient value of the student model for the current round using the backpropagation algorithm based on the distribution loss value; adjusting the model parameters of the student model during the current round of training based on the first gradient value, and obtaining the trained student model when the current training round number corresponding to the second model reaches a preset first training round number threshold. The first training round number threshold is an upper limit on the number of rounds for training the student model, used to indicate that the model training process has converged and multi-round training has stopped.

[0097] In some embodiments, based on the trained second model, further RL training is performed on the trained second model. Based on test sample data, at least one answer path is generated using the trained second model. Each answer path includes the reasoning process corresponding to the reasoning task and a third answer sequence for responding to the test sample data. Based on the at least one answer path, reward signals corresponding to each answer path are obtained through a reward model or human feedback. These reward signals indicate the accuracy of the answer path. The trained second model is then trained multiple times based on the reward signals until the current training round reaches a preset second training round threshold, thus obtaining the reasoning model. The second training round threshold is an upper limit on the number of training rounds performed on the trained second model, used to indicate that the model training process has converged and multi-round training has stopped. Optionally, the reward signal can be implemented as a numerical reward score.

[0098] In some embodiments, after the training process of the inference model described above is completed, test data is acquired, and the performance of the trained inference model is evaluated. The object of the comparative evaluation can be a training method in related technologies that uses hard labels for parameter fine-tuning, or a training method that performs model distillation based on the full generation results. Performance evaluation metrics include, but are not limited to, model training time and the accuracy of sampling results during training.

[0099] Schematic, the process of performing a reasoning task based on a reasoning model includes, but is not limited to, obtaining descriptive data corresponding to the reasoning task, the descriptive data being used to express the task requirements of the reasoning task, and the descriptive data may include text prompts, schematic images, etc.; inputting the descriptive data into the reasoning model, and encoding the descriptive data to obtain an input sequence; and generating an answer sequence through the reasoning model to respond to the descriptive data.

[0100] In summary, the method provided in this application determines at least two selected text words based on the predicted probabilities of multiple first text words generated by the first model during the generation of the first answer sequence. A second model is then trained based on the predicted probabilities of these two selected text words and the predicted probabilities of multiple second text words. This preserves the preferences of the first model when generating the answer sequence, improves the basic accuracy of the trained second model in subsequent training processes, and thus improves the execution accuracy of the reasoning task. Furthermore, by extracting selected text words, the amount of data to be processed during the training of the second model is reduced, computational resource consumption is decreased, and the training efficiency of the reasoning model is improved.

[0101] Please refer to Figure 4 This illustrates a flowchart of a training method for an inference model provided in another embodiment of this application. The method is performed by a computer device (which can be implemented as...) Figure 1 The method is executed by either the terminal 120 or the server 140 shown, or it is executed jointly by the terminal and the server. In this embodiment, the method is executed by the server as an example. Figure 4 As shown, step 240 above can also be implemented as steps 242 and 244 below.

[0102] Step 242: Obtain the first probability distribution corresponding to at least two selected text words, and the second probability distribution corresponding to multiple second text words.

[0103] Based on the predicted probabilities corresponding to at least two selected text words, a first probability distribution corresponding to at least two selected text words is obtained; and based on the predicted probabilities corresponding to multiple second text words, a second probability distribution corresponding to multiple second text words is obtained. Specifically, in the process of generating a first answer sequence for responding to sample data using a first model, and generating a second answer sequence for responding to sample data using a second model, the first sequence position of the first answer sequence corresponds to multiple first text words as candidate words, and the first model also generates predicted probabilities corresponding to multiple first text words, with each first text word associated with its corresponding predicted probability; similarly, the first sequence position of the second answer sequence corresponds to multiple second text words as candidate words, and the second model also generates predicted probabilities corresponding to multiple second text words, with each second text word associated with its corresponding predicted probability.

[0104] In some embodiments, during the generation of the first answer sequence, a first logical value corresponding to a plurality of first text words is obtained through a first model; and during the generation of the second answer sequence, a second logical value corresponding to a plurality of second text words is obtained through a second model. Normalization is performed on the plurality of first logical values ​​to obtain the prediction probabilities corresponding to the plurality of first text words, and normalization is performed on the plurality of second logical values ​​to obtain the prediction probabilities corresponding to the plurality of second text words. Illustratively, the prediction probability corresponding to the first text word is a confidence level in the range [0, 1] obtained by converting the original prediction score indicated by the first logical value; the prediction probability corresponding to the second text word is a confidence level in the range [0, 1] obtained by converting the original prediction score indicated by the second logical value.

[0105] The logit value indicates the original prediction score for each text word generated by the model's output layer. The logit value is an evaluation of the matching degree between the sequence position and the text word by the inference model, including word selection preference, model prediction uncertainty, generalization tendency, and other model structural information. The role of the logit value in the inference model is to provide a linear output value representing the inference model's prediction for each text word; that is, it is a measure of the inference model's confidence or prediction probability for each text word, rather than directly representing the prediction probability. The prediction probability for each text word is obtained by normalizing the logit value, such as by applying an activation function.

[0106] Optionally, the activation function mentioned above includes, but is not limited to, the softmax function. The softmax function can convert logistic values ​​into probability values, such that the prediction score corresponding to each text word is between 0 and 1, and the sum of the prediction probabilities corresponding to multiple text words is 1. The output after softmax processing can more intuitively explain the probability of each text word appearing in the answer sequence.

[0107] For example, the output layer of the inference model corresponds to the logical values ​​of multiple text words at the first sequence position in the generated answer sequence, which are 1.2, 3.1, -0.5, 4.8, and 2.3, respectively. A larger logical value corresponds to a higher prediction probability; that is, a positive value indicates that the text word tends to appear at the first sequence position, while a negative value indicates that the text word tends not to appear at the first sequence position. Normalizing the logical values ​​of the multiple text words using the softmax function yields prediction probabilities of 0.021, 0.141, 0.004, 0.771, and 0.063; that is, the probability distribution of multiple text word pairs is p = [0.021, 0.141, 0.004, 0.771, 0.063].

[0108] By converting the logical values ​​indicating the confidence level of the prediction results into standard form of prediction probabilities, it is easier to obtain the probability distribution corresponding to text words, so that the model output results have clear statistical significance, eliminate the scale difference of logical values, and improve the stability, smoothness and boundedness of prediction probabilities through normalization processing, under the constraints that the prediction probabilities satisfy non-negativity and the sum of the prediction probabilities corresponding to multiple text words is 1, thereby reducing data processing errors and improving the accuracy of subsequent training.

[0109] In some embodiments, a first probability distribution corresponding to at least two selected text words is obtained based on their predicted probabilities. This can be achieved through any of the following methods: 1. Obtaining the predicted probabilities corresponding to at least two selected text words, and retaining only the probability distribution corresponding to the at least two selected text words as the first probability distribution; 2. When at least two selected text words are determined from multiple first text words, setting the predicted probabilities corresponding to at least two non-selected text words to 0, and using the changed probability distribution of the multiple first text words as the first probability distribution. The first probability distribution includes the predicted probabilities corresponding to the at least two selected text words, and the predicted probabilities that have been set to 0 for the at least two non-selected text words. Optionally, the above two methods of obtaining the first probability distribution are equivalent during training.

[0110] Step 244: Based on the first probability distribution and the second probability distribution, the second model is trained in multiple rounds to obtain the trained second model.

[0111] Training a second model based on a first and second probability distribution, compared to training the second model using hard labels generated by the first model corresponding to sample data, can convey richer knowledge from the first model and retain the generation preferences learned by the first model on large-scale training data, thereby improving the training efficiency of the inference model. Here, a hard label refers to the first text word with the highest predicted probability. Illustratively, the aforementioned first probability distribution includes the predicted probabilities corresponding to at least two selected words, which can be considered as soft labels. Training the second model based on these soft labels reflects the similarity between the resulting word and other candidate words, including the degree of certainty of the first model's prediction. It can be understood that soft labels are used to represent the implicit associations and feature similarities between the first text words within the cognitive range of the first model.

[0112] For example, when the result word corresponding to the first sequence position in the answer sequence generated by the first model is the number "3", the multiple first text words corresponding to the first sequence position are implemented as the numbers "0" to "9". The prediction probabilities corresponding to the multiple first text words are 0.005, 0.003, 0.020, 0.800, 0.070, 0.010, 0.002, 0.080, 0.008, and 0.002, respectively. That is, the soft label corresponding to the first sequence position is [0.005, 0.003, 0.020, 0.800, 0.070, 0.010, 0.002, 0.080, 0.008, and 0.002]. The soft label reflects that the similarity between the number "3" and the number "8" is higher than the similarity between the number "3" and the number "1", conveying the error that is more likely to occur in the first sequence position of the prediction task.

[0113] In some embodiments, a distribution loss value for the current round is obtained based on a first probability distribution and a second probability distribution. This distribution loss value indicates the difference between the first and second probability distributions. The second model is then trained on the current round based on this distribution loss value. If the current training round number for the second model reaches a preset first training round number threshold, the trained second model is obtained. The distribution loss value for the current round is obtained based on the first and second probability distributions using a preset distribution loss function. This distribution loss function quantifies the difference in the correspondence between the first and second probability distributions to obtain the distribution loss value.

[0114] Schematic, the second model is trained based on the distributed loss value of the current epoch, and the number of training epochs is recorded during the training process, for example, by using a counter to increment the corresponding number of training epochs at the end of each training epoch. When the current number of training epochs for the second model reaches a preset first training epoch threshold, training of the second model stops, that is, the training process of the second model satisfies the first convergence condition, and the trained second model is obtained. The first convergence condition is the convergence condition for the training of the second model, including but not limited to the current number of training epochs reaching the first training epoch threshold, and the difference between the model parameters of the current training epoch and the model parameters of the previous epoch being within a preset first convergence range; this application does not limit this.

[0115] Optionally, the methods for obtaining the distribution loss value for the current round include, but are not limited to: obtaining the cross-entropy between the first probability distribution and the second probability distribution as the distribution loss value for the current round; or, obtaining the Kullback-Leibler divergence (KL) between the first probability distribution and the second probability distribution as the distribution loss value for the current round. That is, the distribution loss function includes, but is not limited to, the cross-entropy function or the KL divergence function.

[0116] Schematic, the formula (1) corresponding to obtaining the KL divergence corresponding to the first probability distribution and the second probability distribution as the distribution loss value L of the current round is shown below.

[0117] (1) in, It is the second probability distribution, which includes the predicted probabilities corresponding to multiple second text terms; It is the first probability distribution, which includes the predicted probabilities corresponding to at least two selected text terms; the KL divergence function is implemented as shown in the following formula (2).

[0118] (2) Where X is a random variable, x is the possible result corresponding to X, and X is used to indicate multiple text terms.

[0119] Alternatively, the formula (3) corresponding to the above method of obtaining the cross-entropy between the first probability distribution and the second probability distribution as the distribution loss value L of the current round is shown below.

[0120] (3) in, It is the second probability distribution, which includes the predicted probabilities corresponding to multiple second text terms; is the first probability distribution, which includes the predicted probabilities corresponding to at least two selected text words; x is the possible outcome of the random variable, used to indicate any one of the multiple text words.

[0121] By obtaining the distribution loss value of the current round, the training direction of the second model in the process of multi-round training based on the first probability distribution and the second probability distribution can be clarified with continuous and differentiable quantitative indicators. This provides a basis for the second model to learn the generalization ability of the first model pre-trained on large-scale training data, thereby improving the training efficiency of the inference model.

[0122] Optionally, the model parameters of the second model are adjusted based on the first gradient value through multiple iterations. The multiple iterations are controlled by training parameters, which include, but are not limited to, at least one of the following parameters.

[0123] 1. Learning rate: Used to control the step size of parameter updates in each round. The size of the learning rate affects the efficiency of training. For example, if the learning rate is too large, the parameter iteration trajectory will be unstable, making it difficult to reach the convergence condition of the model training process and ending the iteration convergence. If the learning rate is too small, it will lead to slow convergence or cause the iteration process to stop prematurely in the suboptimal solution region. For example, the range of the learning rate can be [1e-5, 5e-5].

[0124] 2. Batch size: This indicates the size of the subset of sample data input to the inference model at one time during each round of iterative training; that is, the number of sample data contained in the batch. Batch size affects the stability of the iterative training process. For example, a larger batch size results in a higher accuracy for the first gradient. Furthermore, batch size affects the processor's memory requirements; a larger batch size requires more storage space during training. Additionally, batch size affects the convergence speed during iterative training. A larger batch size allows for more data to be processed in parallel during each round, resulting in a higher iteration speed per round, but more iterations are needed to reach the convergence condition of the model training process, thus leading to slower convergence. For example, the batch size can range from [8, 32], and is a positive integer. The batch size can be adjusted according to the available storage space.

[0125] 3. The training round threshold refers to the upper limit of the number of iterations during training, and is an optional implementation method for the convergence condition of the model training process. The number of training rounds is determined based on the complexity of the inference task corresponding to the sample data; illustratively, complex inference tasks, such as those with many steps in the inference process and long answer sequences, correspond to a larger decision space, that is, a wider range of candidate words for each sequence position, and correspondingly, the iterative training process needs to traverse more rounds.

[0126] The number of training epochs affects the computational complexity and model training quality during training. Illustratively, too many training epochs can lead to overfitting of the second model and lower accuracy in inference tasks. Excessive training epochs can also result in lower generalization performance of the second model and lower accuracy in inference tasks. For example, the training epoch threshold can be in the range [1, 3], and the training epoch threshold is a positive integer.

[0127] It is worth noting that the above-mentioned method of setting training parameters is only an illustrative example, and the specific process of adjusting the parameters of the second model based on the first gradient value through multiple rounds of iteration is not limited in the embodiments of this application.

[0128] In some embodiments, based on the distributed loss value of the current round, the first gradient value of the second model corresponding to the current round is obtained through backpropagation algorithm; based on the first gradient value of the current round, the model parameters of the second model are adjusted during the training of the current round, and the trained second model is obtained when the current training round number of the second model reaches a preset first training round number. Optionally, based on the distributed loss value of the current round, the direction and magnitude of the adjustment processing corresponding to the model parameters of the second model, i.e., the first gradient value of the current round, are obtained through backpropagation algorithm.

[0129] Indicatively, the model parameters of the second model are adjusted during the training of the current round based on the first gradient value of the current round, and the number of training rounds is recorded during the training of the current round, for example, by using a counter to increment the number of training rounds corresponding to the counter at the end of the current round of training; when the current training round number corresponding to the second model reaches the preset first training round number threshold, the training of the second model is stopped, that is, the training process of the second model satisfies the first convergence condition, and the trained second model is obtained.

[0130] Schematic, the partial derivative of the distribution loss function with respect to the model parameters of the second model in the current round is obtained and used as the first gradient value of the model parameters of the second model in the current round. After updating the model parameters of the second model based on the first gradient value of the current round, the distribution loss value in the next round decreases accordingly, thereby reducing the difference between the first probability distribution and the second probability distribution.

[0131] In some embodiments, the model parameters of the second model are updated using an optimization algorithm based on the first gradient value of the current round. The optimization algorithm includes, but is not limited to, gradient descent. Through backpropagation, the first gradient value of the preset distributed loss function relative to the model parameters of the second model in the current round is automatically calculated. This allows for the determination of the adjustment process that should be performed on the model parameters of the second model in the current round, as well as the urgency of the adjustment, thereby adjusting the model parameters of the second model and improving the training efficiency of the model parameters. It also ensures that the distributed loss value is reduced in each iteration of multi-round training. Through the distributed loss function, the first probability distribution and the second probability distribution are made to approximate each other, enabling the second model to learn the preferences of the first model in generating text terms, thus improving the accuracy of the second model in performing inference tasks.

[0132] For illustrative purposes, please refer to the following: Figure 5 , Figure 5 This is a schematic diagram of a training process based on the probability distributions corresponding to the first model and the second model, respectively, provided in an exemplary embodiment of this application. Figure 5As shown, when sample data is acquired, a first answer sequence for responding to the sample data is generated by a first model 310. The first sequence position of the first answer sequence corresponds to multiple first text words as candidate words, and the first text words are associated with the predicted probabilities corresponding to the first text words. Based on the predicted probabilities corresponding to the multiple first text words, at least two selected text words 521 are obtained from the multiple first text words. A second answer sequence for responding to the sample data is generated by a second model 510. The first sequence position of the second answer sequence corresponds to multiple second text words 522 as candidate words, and the second text words are associated with the predicted probabilities corresponding to the second text words. Based on the predicted probabilities corresponding to the at least two selected text words 521, a first probability distribution 531 is obtained, and based on the predicted probabilities corresponding to the multiple second text words 522, a second probability distribution 532 is obtained.

[0133] like Figure 5 As shown, based on the first probability distribution 531 and the second probability distribution 532, the distribution loss value 540 for the current round is obtained. The distribution loss value 540 is used to indicate the difference between the first probability distribution 531 and the second probability distribution 532. Optionally, the distribution loss value 540 for the current round can be realized as the cross-entropy between the first probability distribution 531 and the second probability distribution 532 or the KL divergence corresponding to the first probability distribution 531 and the second probability distribution 532. The second model 510 is trained for the current round based on the distribution loss value 540 for the current round, including but not limited to: obtaining the first gradient value 550 of the second model 510 for the current round through the backpropagation algorithm based on the distribution loss value 540 for the current round; adjusting the model parameters of the second model 510 during the training of the current round based on the first gradient value 550, and obtaining the trained second model 560 when the current training round number of the second model 510 reaches the first training round number threshold.

[0134] In summary, the method provided in this application determines at least two selected text words based on the predicted probabilities of multiple first text words generated by the first model during the generation of the first answer sequence. A second model is then trained based on the predicted probabilities of these two selected text words and the predicted probabilities of multiple second text words. This preserves the preferences of the first model when generating the answer sequence, improves the basic accuracy of the trained second model in subsequent training processes, and thus improves the execution accuracy of the reasoning task. Furthermore, by extracting selected text words, the amount of data to be processed during the training of the second model is reduced, computational resource consumption is decreased, and the training efficiency of the reasoning model is improved.

[0135] The method provided in this application training method trains a second model based on at least two first probability distributions corresponding to selected text words and multiple second probability distributions corresponding to second text words. The first probability distribution includes more gradient information. During training, by comparing the first probability distribution and the second probability distribution, richer knowledge in the first model is obtained, and the generation preferences learned by the first model on large-scale training data are retained, thereby improving the training stability and training efficiency of the inference model.

[0136] Please refer to Figure 6 This illustrates a flowchart of a training method for an inference model provided in yet another embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 1 The method is executed by either the terminal 120 or the server 140 shown, or it is executed jointly by the terminal and the server. In this embodiment, the method is executed by the server as an example. Figure 6 As shown, the method may further include the following steps; optionally, after step 240 above, it may further include steps 610 to 630.

[0137] Step 610: Based on the test sample data, generate at least one answer path using the trained second model.

[0138] The answer path includes the reasoning process corresponding to the reasoning task, and a third answer sequence used to respond to the test sample data. That is, the answer path is used to respond to the reasoning task. Optionally, the reasoning process is used to express the thought process of obtaining the answer sequence, and can be implemented as a natural language text sequence, or a flowchart image, etc. The answer path is the result generated by the trained second model for the reasoning task, including the reasoning process and the third answer sequence. Illustratively, generating the reasoning process is a sub-task of the reasoning task. The second model improves the interpretability and comprehensibility of the third answer sequence through the reasoning process, thus providing a supplementary response to the test sample data. Optionally, the process of the second model generating the second answer sequence can be implemented as generating the answer path corresponding to the second model, including but not limited to the reasoning process and second answer sequence generated by the second model corresponding to the reasoning task.

[0139] It is worth noting that the test sample data is the training data from the RL training phase, and the test sample data may be the same as or different from the sample data. Optionally, the aforementioned test sample data may be sample data used for iterative training of the second model, or the test sample data may be data used only for iterative training of the second model after training to obtain the inference model.

[0140] In some embodiments, the sampling results obtained from sampling test sample data form the basis for further RL training of the trained second model. Here, the sampling results refer to the answer paths generated by the trained second model based on the test sample data. For example, sampling strategies such as sampling strategies and bundle search strategies are used to generate the sampling results corresponding to the test sample data.

[0141] In some embodiments, when test sample data is obtained, a third answer sequence for responding to the test sample data is generated by a trained second model. At the same time, an explanation is generated to describe the reasoning process of deriving the third answer sequence based on the reasoning task description. Accordingly, at least one answer path corresponding to the test sample data is obtained.

[0142] In the reasoning process corresponding to the reasoning task, prompting techniques, such as the thought chain prompting algorithm, can be used to decompose and output the reasoning steps corresponding to the generation of the third answer sequence by the trained second model into a coherent natural language text sequence. The process of generating the third answer sequence by the trained second model includes, but is not limited to: inputting test sample data into the trained second model and encoding the test sample data to obtain the input sequence; and the trained second model generating at least one third answer sequence by autoregression based on the input sequence.

[0143] Step 620: Based on at least one answer path, obtain the reward signal corresponding to each of the at least one answer path through the reward model.

[0144] The reward signal is used to indicate the accuracy of the answer path. In some embodiments, the reward model is an independent large language model pre-trained based on test sample data with human preferences, used to evaluate the quality of the sampling results; wherein, the quality of the sampling results includes evaluation dimensions such as the coverage, flexibility, and accuracy of the answer path.

[0145] Optionally, the reward model can be obtained in ways including but not limited to at least one of the following methods.

[0146] 1. Obtain a pre-defined reward model. This can be illustrated by downloading a reward model that meets the requirements of the reasoning task from the internet, such as a reward model from an open-source platform or an officially released reward model; alternatively, a reward model can be obtained through a cloud-based Application Programming Interface (API), and the reward signals corresponding to at least one answer path can be directly obtained from the reward model via the API, without needing to obtain the model file.

[0147] 2. Obtain reward sample data representing human preferences; pre-train a reward model that meets the requirements of the reasoning task based on the reward sample data. For example, obtain historical data or open-source reward sample data, start training the model from scratch, and obtain the reward model; or, obtain a pre-set base model by downloading or uploading, and obtain reward sample data corresponding to the reasoning task, update the base model based on the reward sample data, and obtain the reward model.

[0148] In some embodiments, a reward model receives at least one answer path generated by a trained second model, and test sample data corresponding to at least one answer path, and outputs a reward signal as a scalar value. The reward signal quantifies the overall quality of the answer path in dimensions such as its conformity to human preferences, its matching degree with the reasoning task, its safety, and its factuality; a higher reward signal indicates a better quality answer path. Optionally, the reward signal can be obtained through manual annotation.

[0149] Step 630: Based on the reward signal, the trained second model is trained for multiple rounds until the current training round number reaches the preset second training round number threshold to obtain the inference model.

[0150] In some embodiments, training the trained second model multiple times based on reward signals is implemented as a Reinforcement Learning from Human Feedback (RLHF) process. RLHF incorporates human feedback into the training process, providing a natural and human-like interactive learning process for machines to perform inference tasks through the inference model. Optionally, RLHF uses reward signals to guide further reinforcement learning training of the trained second model, improving the inference model's generalization ability in vertical domains.

[0151] Schematic, the trained second model is trained multiple times based on the reward signal; the number of training rounds is recorded during each round, for example, by using a counter, and the number of training rounds corresponding to the counter is incremented at the end of each round; when the current training round number of the second model reaches a preset second training round number threshold, training of the trained second model is stopped, that is, the training process of the trained second model satisfies the second convergence condition, and the inference model is obtained. The second convergence condition is the convergence condition for training the trained second model, including but not limited to the current training round number reaching the second training round number threshold, and the difference between the model parameters of the current training round and the model parameters of the previous round being within a preset second convergence range; this application does not limit this.

[0152] In some embodiments, based on the reward signal, the policy loss value of the current round is obtained, which is used to indicate the difference between the answer path and the target answer path corresponding to the inference task; based on the policy loss value of the current round, the second gradient value of the trained second model corresponding to the current round is obtained through the backpropagation algorithm; based on the second gradient value of the current round, the model parameters of the trained second model are adjusted in the training of the current round until the current training round reaches the second training round threshold to obtain the inference model.

[0153] Schematic, the policy loss value for the current round is obtained based on the reward signal; the second gradient value for the current round is obtained based on the policy loss value for the current round; the trained second model is trained for the current round based on the second gradient value for the current round; the number of training rounds is recorded during the training process of the current round, for example, by using a counter, and the number of training rounds corresponding to the counter is incremented at the end of the current round of training; when the number of training rounds corresponding to the trained second model reaches the preset second training round threshold, the training of the trained second model is stopped, that is, the training process of the trained second model satisfies the second convergence condition, and the inference model is obtained.

[0154] The second model is trained multiple times based on the reward signal until the current training round reaches the preset second training round threshold to obtain the inference model; the above method for obtaining the inference model includes, but is not limited to, at least one of the following training methods.

[0155] 1. Proximal Policy Optimization (PPO) Algorithm: In the sampling phase, the trained second model generates at least one answer path based on the test sample data, that is, interacts with the environment corresponding to the reasoning task to obtain the sampling results; the reward signal corresponding to at least one answer path is obtained based on the reward model. Optionally, the reward model is implemented as a value model (Critic).

[0156] Furthermore, through multiple rounds of iterative training, the probability ratio of the predicted probability generated by the inference model in the current round to that of the inference model in the previous round for the same sequence position in the answer path is calculated. The PPO algorithm introduces a pruning mechanism to forcibly limit the probability ratio within a preset range, thereby constraining the adjustment range of the model parameters of the trained second model in each round and improving training stability. Based on the pruned probability ratio and the corresponding reward signal, the policy loss value for the current round is calculated. This policy loss value indicates the difference in the generation strategy of the trained second model in the current round compared to the previous round. Optionally, a value loss value is calculated using a value loss function, and combined with other loss values, a policy loss value that integrates different measurement dimensions is obtained. Based on the policy loss value of the current round, the second gradient value of the trained second model for the current round is obtained through backpropagation. Based on the second gradient value of the current round, the model parameters of the trained second model are adjusted, and the iterative training process, taking the current round as an example, is repeated for multiple rounds until the current training round number reaches a preset second training round number threshold, thus obtaining the inference model.

[0157] 2. Group Relative Policy Optimization (GRPO) Algorithm: First, based on the same test sample data, at least one answer path is generated in parallel by the trained second model to obtain a sampling group. After obtaining the reward signal corresponding to each answer path through the reward model, the average reward signal corresponding to at least one answer path in the sampling group is obtained, and this average reward signal is used as the evaluation baseline. The average reward signal of each answer path is subtracted from the corresponding average reward signal to obtain the relative reward signal of each answer path. Based on the relative reward signal of each answer path, the policy loss value of the current round with different weights is obtained. The policy loss value is used to indicate the difference between the answer path and the target answer path corresponding to the inference task. Based on the policy loss value of the current round, the second gradient value of the trained second model corresponding to the current round is obtained through backpropagation. Based on the second gradient value of the current round, the model parameters of the trained second model are adjusted in the current round of training. When the current training round number of the trained second model reaches the second training round number threshold, the inference model is obtained.

[0158] In some embodiments, based on the policy loss value of the current round, the second gradient value of the trained second model corresponding to the current round is obtained through backpropagation. Based on the second gradient value of the current round, the model parameters of the trained second model are adjusted during the current round of training. When the current training round number corresponding to the trained second model reaches a second training round number threshold, the inference model is obtained. Optionally, based on the policy loss value of the current round, the direction and magnitude of the adjustment processing of the model parameters of the trained second model corresponding to the current round are obtained through backpropagation, i.e., the second gradient value of the current round. Schematically, the partial derivative of the policy loss function with respect to the model parameters of the trained second model in the current round is obtained as the second gradient value of the model parameters of the trained second model corresponding to the current round. After updating the model parameters of the trained second model based on the second gradient value of the current round, the policy loss value of the next round decreases accordingly, thereby reducing the difference between the answer path and the target answer path corresponding to the inference task.

[0159] In some embodiments, the model parameters of the trained second model are updated using an optimization algorithm based on the second gradient value of the current iteration. Through backpropagation, the second gradient value of the preset policy loss function relative to the model parameters of the trained second model in the current iteration is automatically calculated. This allows for the determination of the necessary adjustments to the model parameters in the current iteration, as well as the urgency of these adjustments, thereby adjusting the model parameters and improving the training efficiency of the trained second model. Furthermore, it ensures that the policy loss value is reduced in each iteration. The policy loss function, with its continuous and differentiable quantitative indicators, clarifies the direction of RL training, providing a basis for further training of the trained second model and improving the accuracy of the inference model in performing inference tasks.

[0160] In some embodiments, a third answer sequence and a reasoning process are obtained based on the answer path; a first reward signal corresponding to the third answer sequence and a second reward signal corresponding to the reasoning process are obtained through a reward model; and a reward signal corresponding to the answer path is obtained based on the first and second reward signals. Schematic, based on test sample data, at least one answer path is generated through a trained second model, wherein each answer path includes a third answer sequence and a set of reasoning processes; based on the third answer sequence corresponding to each answer path, a first reward signal corresponding to each answer path is obtained through a reward model, and based on the reasoning process corresponding to each answer path, a second reward signal corresponding to each answer path is obtained through a reward model; thereby, the first reward signal and the second reward signal corresponding to each answer path are combined to obtain the reward signal corresponding to each answer path.

[0161] For example, when the reasoning task is implemented as a mathematical reasoning task, test sample data is obtained, such as the text prompt "Find the number of positive integers less than 1000 that are divisible by 7 but not by 3." At least one answer path is generated through the trained second model, including an example answer path. The example answer path includes an example third answer sequence and an example reasoning process. For example, the example third answer sequence is "285," and the example reasoning process is the text sequence "Obtaining the number of positive integers that meet the requirements includes the following two steps of reasoning: counting the multiples of 7 within 1000; and removing positive integers that are multiples of both 7 and 3 from the positive integers corresponding to the multiples of 7 within 1000 (i.e., removing positive integers that are multiples of 21)." Based on at least one answer path, a reward model is used to obtain the first reward signal corresponding to the third answer sequence and the second reward signal corresponding to the reasoning process. The reward signal corresponding to the answer path is obtained by combining the first and second reward signals.

[0162] The first reward signal indicates the reward signal for the result of the third answer sequence, ensuring the accuracy of the final answer. The second reward signal indicates the reward signal for the reasoning process, guiding the trained second model to learn logical and interpretable reasoning steps. This allows for a more comprehensive and refined evaluation of the output results of the trained second model. The comprehensive reward signal directly aligns with complex objectives, such as human preferences or reasoning task scores, while simultaneously evaluating different evaluation metrics of the sampled results. This avoids overfitting during training and improves the logical consistency and generalization ability of the reasoning model after iterative training, thereby enhancing the training efficiency of the reasoning model.

[0163] In summary, the method provided in this application determines at least two selected text words based on the predicted probabilities of multiple first text words generated by the first model during the generation of the first answer sequence. A second model is then trained based on the predicted probabilities of these two selected text words and the predicted probabilities of multiple second text words. This preserves the preferences of the first model when generating the answer sequence, improves the basic accuracy of the trained second model in subsequent training processes, and thus improves the execution accuracy of the reasoning task. Furthermore, by extracting selected text words, the amount of data to be processed during the training of the second model is reduced, computational resource consumption is decreased, and the training efficiency of the reasoning model is improved.

[0164] The method provided in this application embodiment uses a reward model to enable the trained second model to continuously interact with the environment corresponding to the reasoning task, autonomously explore and learn the target answer path that conforms to the reasoning task; guided by reward signals, it can discover solutions other than the preset answer path in complex reasoning tasks with multiple steps, multiple solution paths, and interpretability, thereby enhancing the reasoning model's understanding and satisfaction of human intentions and improving the efficiency and accuracy of the reasoning model in performing reasoning tasks.

[0165] The overall training process of the inference model provided in this application includes, but is not limited to: 1. Top-K text token extraction from the teacher model; 2. Top-K distribution distillation SFT processing; 3. RL training of the distilled small model (i.e., the student model). The specific implementation process of the above overall training process of the inference model includes at least one of the following steps.

[0166] Step 1. Extract Top-K Tokens and Predicted Probabilities for the Teacher Model: For the training data of the target task (such as mathematical reasoning task, code generation task, etc.), for each token position of the teacher model (such as GPT-5, DeepSeek-R1-0528, etc., the structure and parameters of the above models can be obtained from public channels), output the training data for the student model, including but not limited to: the Top-K candidate tokens for each token position, where K is a predefined integer, and the value of K can be adjusted according to the task. For example, the value range is [3, 20]; the normalized predicted probability or the unnormalized Logit corresponding to each candidate token.

[0167] To illustrate, if the teacher model outputs Logit, it needs to be converted into predicted probability through softmax processing. Then, the tokens are sorted from high to low according to the predicted probability to obtain the Top-K candidate tokens. For each token position, the predicted probability of non-Top-K tokens can be set to 0, or only the probability distribution of the Top-K candidate tokens can be retained, thereby reducing the amount of computation.

[0168] Step 2. Distillation SFT processing based on Top-K distribution: The Top-K distribution refers to the probability distribution of the Top-K candidate tokens at each token position. The training loss function of the student model (i.e., the small model, such as GPT-2, Qwen3-8B, and other open-source models; the structure and parameters of these models can be obtained from public channels) can be the distribution matching loss function.

[0169] Indicatively, the student model outputs Logit for each token position; the KL divergence (or cross-entropy) between the student model's Logit and the teacher model's Top-K distribution is calculated as the loss value L, as shown in formula (1) below.

[0170] (1) in, It represents the probability distribution of the current token position in the student model; It is the probability distribution of the Top-K tokens at this position in the teacher model; optionally, the predicted probability of non-Top-K tokens is 0.

[0171] The above-mentioned distillation SFT processing is implemented as iterative training. The training parameters can include at least one of the following implementation methods: the learning rate ranges from [1e-5, 5e-5]; the batch size ranges from [8, 32], and the batch size can be adjusted according to the memory size of the training computing card, such as the graphics processing unit (GPU); the number of training rounds ranges from [1, 3], and the number of training rounds is adapted to the task complexity.

[0172] Step 3. RL Training on the Distilled Small Model: The small model after the SFT processing described above is trained using reinforcement learning (e.g., the PPO algorithm). The training process is consistent with the regular RLHF training process, but since the small model has already learned the Top-K distribution of the teacher model, the efficiency of sampling high-quality paths is improved. Illustratively, the small model samples multiple answer paths; the path quality is evaluated using a reward model or human feedback; and the small model parameters are updated based on PPO with the aim of maximizing rewards.

[0173] In some embodiments, the training method for the inference model includes at least one of the following steps: a. For each token position in the teacher model, extract the Top-K candidate tokens and the predicted probabilities corresponding to the Top-K candidate tokens respectively; b. Use the student model to learn the Top-K Token probability distribution of the teacher model, with the loss function being distribution matching loss; c. Use the distilled student model for reinforcement learning training.

[0174] Optionally, the Top-K values ​​range from [3, 20], and the distribution matching loss is either KL divergence or cross-entropy. Optionally, the inference task to be performed by the student model is a mathematical inference task or a code generation task.

[0175] For illustrative purposes, please refer to the following: Figure 7 , Figure 7 This is a schematic diagram of the inference model training process provided in an exemplary embodiment of this application, as shown below. Figure 7 As shown, the first model and the second model are obtained, and step 710, data preparation, is performed. Sample data is obtained, which describes the task requirements of the inference task; optionally, the sample data can be implemented as a sample dataset, which includes multiple task requirement descriptions corresponding to different inference tasks, and the task types of the inference tasks in the same sample dataset are consistent; the sample dataset is the dataset corresponding to the target inference task, and the inference task includes, but is not limited to, mathematical inference tasks or code generation tasks. For example, the sample dataset includes, but is not limited to, the GSM8K (primary school math problems) dataset, the MATH (high school math problems) dataset, etc.; the teacher model, i.e., the first model, can be implemented as the DeepSeek-R1-0528 model; since there are few candidate tokens for the inference path in the mathematical task corresponding to the sample dataset, the Top-K value is K=5.

[0176] like Figure 7 As shown, step 720 involves selecting text words for extraction. A first answer sequence for responding to sample data is generated using a first model. The first sequence position of the first answer sequence corresponds to multiple first text words as candidate words, and each first text word is associated with its corresponding predicted probability. Based on the predicted probabilities of the multiple first text words, at least two selected text words are extracted from the multiple first text words. The process of extracting at least two selected text words includes: obtaining a word word sequence based on the multiple first text words, the word word sequence including the multiple first text words; obtaining K selected text words from the word word sequence, where K is a positive integer, and the predicted probability corresponding to each of the K selected text words is higher than or equal to the predicted probability corresponding to any other text word in the word word sequence. Illustratively, the multiple first text words are arranged in descending order of predicted probability to obtain the word word sequence; the first K selected text words of the word word sequence are obtained, the value of K being determined based on the task type of the reasoning task. Optionally, the value of K is predetermined in step 710. For example, the value of K can be an integer in the range [3, 20].

[0177] For example, step 720 is to extract the Top-K distribution of the teacher model: for each math problem, DeepSeek-R1-0528 generates the reasoning process, and outputs the Top-5 tokens and the prediction probabilities corresponding to the Top-5 tokens at each token position. For example, for the position "2+3=", the Top-5 tokens and prediction probabilities may be "5" (0.8), "1" (0.05), "4" (0.03), etc.

[0178] Furthermore, before executing step 730, a second answer sequence for responding to the sample data is generated through the second model; the first sequence position of the second answer sequence corresponds to multiple second text words as candidate words, and the second text words are associated with the predicted probabilities corresponding to the second text words; optionally, the acquisition of multiple second text words can be performed before the extraction of selected text words, or the acquisition of multiple second text words can be performed after the extraction of selected text words, or the acquisition of multiple second text words can be performed in parallel with the extraction of selected text words.

[0179] like Figure 7 As shown, step 730, model distillation, is performed. Based on the predicted probabilities corresponding to at least two selected text words and the predicted probabilities corresponding to multiple second text words, a second model is trained to obtain the trained second model, which is used to perform the inference task. Based on the predicted probabilities corresponding to at least two selected text words, a first probability distribution corresponding to at least two selected text words is obtained; and based on the predicted probabilities corresponding to multiple second text words, a second probability distribution corresponding to multiple second text words is obtained. Based on the first and second probability distributions, the second model is trained multiple times to obtain the trained second model, which is the distilled inference model.

[0180] The training of the second model involves obtaining the distribution loss value for the current round, which indicates the difference between the first probability distribution and the second probability distribution. Based on the distribution loss value for the current round, the second model is trained for the current round. When the current training round number for the second model reaches a preset first training round number threshold, the trained second model is obtained. The methods for obtaining the distribution loss value for the current round include, but are not limited to, obtaining the cross-entropy between the first probability distribution and the second probability distribution as the distribution loss value for the current round; or, obtaining the KL divergence corresponding to the first probability distribution and the second probability distribution as the distribution loss value for the current round.

[0181] For example, step 730, also known as the Top-K distillation SFT process, includes at least one of the following implementation details: the student model can be Qwen3-8B, the loss function corresponds to KL divergence, and the training parameters include a learning rate of 2e-5, a batch size of 16, and a training epoch of 3.

[0182] like Figure 7As shown, based on the obtained trained second model, step 740, RL training, is executed. Based on the test sample data, at least one answer path is generated through the trained second model. The answer path includes the reasoning process corresponding to the reasoning task, and a third answer sequence used to respond to the test sample data. Based on at least one answer path, a reward signal corresponding to each answer path is obtained through a reward model. The reward signal is used to indicate the accuracy of the answer path, i.e., the correctness of the answer. Based on the reward signal, the trained second model is trained multiple times using an RL algorithm (such as the GRPO algorithm) until the current training round reaches a preset second training round threshold, thus obtaining the reasoning model.

[0183] Specifically, the reward signal obtained through the reward model for at least one answer path includes, but is not limited to, obtaining the policy loss value for the current round based on the reward signal, whereby the policy loss value is used to indicate the difference between the answer path and the target answer path corresponding to the inference task; obtaining the second gradient value for the current round of the trained second model through the backpropagation algorithm based on the policy loss value for the current round; adjusting the model parameters of the trained second model in the current round of training based on the second gradient value for the current round; and obtaining the inference model when the current training round number corresponding to the trained second model reaches the second training round number threshold.

[0184] For example, step 740 above can be trained using the GRPO algorithm, with the reward model being a reward function that evaluates the sampling results by combining the correctness of the answer and the rationality of the reasoning steps, and the number of training rounds being 3.

[0185] like Figure 7 As shown, based on the inference model obtained through the above training process, step 750 is executed for performance evaluation. Test data is obtained to evaluate the performance of the trained inference model. The comparison and evaluation can be against training methods that fine-tune parameters using hard labels, or training methods that perform model distillation based on the full generated results. Performance evaluation metrics include, but are not limited to, model training time (used to indicate training efficiency) and the accuracy of sampling results during training. For example, compared to traditional SFT processing (using hard labels as training data) and distillation using the probability distribution corresponding to the complete Logit, the method provided in this application, namely the Top-K distillation small model training method, has higher sampling accuracy and shorter training time on GSM8K.

[0186] Through the training method of the above inference model, the Top-K distribution retains the "multi-candidate token preference" of the teacher model, making the small model sample closer to the high-quality answer path. For example, it can improve the answer accuracy in mathematical reasoning tasks, thereby improving the sampling quality in the RL training process. Furthermore, by processing only the Top-K tokens, the amount of training data is reduced, the distillation training speed is improved, and the memory usage is reduced. In some model training examples, the Top-K tokens account for less than 0.1% of the total tokens, thereby reducing the computational cost. Moreover, the above training method can flexibly adjust the value of Top-K. For example, K=10 is used for coding tasks, and K=5 is used for mathematical tasks. The value of K can be set according to the exploration breadth requirements of different tasks, adapting to multiple types of multi-path inference tasks, thereby enhancing the task adaptability of the training process and the inference model.

[0187] Figure 8 This is a structural block diagram of a training apparatus for an inference model provided in an exemplary embodiment of this application, as shown below. Figure 8 As shown, the device includes at least one of the following modules.

[0188] The acquisition module 810 is configured to acquire sample data, which is used to describe the task requirements of the inference task. The generation module 820 is configured to generate a first answer sequence for responding to sample data through a first model, and to generate a second answer sequence for responding to sample data through a second model; wherein, the first sequence position of the first answer sequence corresponds to multiple first text words, and the first text words are associated with the predicted probabilities corresponding to the first text words; the first sequence position of the second answer sequence corresponds to multiple second text words, and the second text words are associated with the predicted probabilities corresponding to the second text words. The acquisition module 810 is also configured to acquire at least two selected text words from the multiple first text words based on the predicted probabilities corresponding to the multiple first text words respectively; The training module 830 is configured to train a second model based on the prediction probabilities corresponding to at least two selected text words and the prediction probabilities corresponding to multiple second text words, thereby obtaining a trained second model, which is used to perform inference tasks.

[0189] In an optional embodiment, the acquisition module 810 is further configured to acquire a word sequence based on a plurality of first text words, the word sequence including a plurality of first text words; The acquisition module 810 is also configured to acquire K selected text words in the word word sequence, where K is a positive integer, and the prediction probability corresponding to each of the K selected text words is higher than or equal to the prediction probability corresponding to any other text word in the word word sequence.

[0190] In an optional embodiment, before the acquisition module 810 acquires the first K selected text words of the word sequence, the acquisition module 810 is also configured to acquire the task type of the reasoning task. The acquisition module 810 is also configured to acquire the value of K based on the task type.

[0191] In an optional embodiment, the generation module 820 is further configured to obtain first logical values ​​corresponding to multiple first text words through a first model during the generation of the first answer sequence, and to obtain second logical values ​​corresponding to multiple second text words through a second model during the generation of the second answer sequence. The acquisition module 810 is also configured to perform normalization processing on multiple first logical values ​​to obtain the prediction probabilities corresponding to multiple first text words respectively, and to perform normalization processing on multiple second logical values ​​to obtain the prediction probabilities corresponding to multiple second text words respectively.

[0192] In an optional embodiment, the acquisition module 810 is further configured to acquire a first probability distribution corresponding to at least two selected text words based on the prediction probabilities corresponding to at least two selected text words respectively, and to acquire a second probability distribution corresponding to multiple second text words based on the prediction probabilities corresponding to multiple second text words respectively. The training module 830 is also configured to perform multiple rounds of training on the second model based on the first probability distribution and the second probability distribution to obtain the trained second model.

[0193] In an optional embodiment, the training module 830 is further configured to obtain a distribution loss value for the current round based on a first probability distribution and a second probability distribution, the distribution loss value being used to indicate the difference between the first probability distribution and the second probability distribution; The training module 830 is also configured to train the second model for the current round based on the distribution loss value of the current round, and to obtain the trained second model when the current training round number of the second model reaches the preset first training round number threshold.

[0194] In an optional embodiment, the training module 830 is further configured to obtain the first gradient value of the second model for the current round based on the distribution loss value of the current round through the backpropagation algorithm. The training module 830 is also configured to adjust the model parameters of the second model in the current training round based on the first gradient value of the current round, and obtain the trained second model when the current training round number corresponding to the second model reaches the first training round number threshold.

[0195] In an optional embodiment, the generation module 820 is further configured to generate at least one answer path based on the test sample data using a trained second model, the answer path indicating the reasoning process corresponding to the reasoning task, and a third answer sequence for responding to the test sample data. The acquisition module 810 is also configured to acquire reward signals corresponding to at least one answer path based on at least one answer path through a reward model. The reward signals are used to indicate the accuracy of the answer path. The training module 830 is also configured to perform multiple rounds of training on the trained second model based on the reward signal until the current training round number reaches the preset second training round number threshold to obtain the inference model.

[0196] In an optional embodiment, the training module 830 is further configured to obtain a policy loss value for the current round based on a reward signal, the policy loss value being used to indicate the difference between the answer path and the target answer path corresponding to the inference task; The training module 830 is also configured to obtain the second gradient value of the current round corresponding to the current round of the trained second model through the backpropagation algorithm based on the policy loss value of the current round. The training module 830 is also configured to adjust the model parameters of the trained second model in the current training round based on the second gradient value of the current round, and obtain the inference model when the current training round number corresponding to the trained second model reaches the second training round number threshold.

[0197] In an optional embodiment, the generation module 820 is further configured to obtain a third answer sequence and a reasoning process based on the answer path; The acquisition module 810 is also configured to acquire the first reward signal corresponding to the third answer sequence and the second reward signal corresponding to the reasoning process through the reward model; The acquisition module 810 is also configured to acquire the reward signal corresponding to the answer path based on the first reward signal and the second reward signal.

[0198] In summary, the apparatus provided in this application determines at least two selected text words based on the predicted probabilities corresponding to multiple first text words generated by the first model during the generation of the first answer sequence. It then trains a second model based on the predicted probabilities corresponding to the at least two selected text words and the predicted probabilities corresponding to multiple second text words. This preserves the preferences of the first model when generating the answer sequence, improves the basic accuracy of the trained second model in subsequent training processes, and thus improves the execution accuracy of the reasoning task. Furthermore, by extracting selected text words, it reduces the amount of data to be processed during the training of the second model, lowers computational resource consumption, and improves the training efficiency of the reasoning model.

[0199] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0200] Please refer to Figure 9 This diagram illustrates a structural block diagram of a computer device 900 provided in one embodiment of this application. The computer device 900 may be... Figure 1 The terminal 120 or server 140 in the computer system shown is used to implement the training method of the inference model provided in the above embodiments. Specifically: The computer device 900 can be a portable mobile terminal, also referred to as a mobile terminal in this embodiment. Examples include: smartphones, tablets, Moving Picture Experts Group Audio Layer III (MP3) players, and Moving Picture Experts Group Audio Layer IV (MP4) players. The computer device 900 may also be referred to as a user device, portable terminal, or other names.

[0201] Typically, computer device 900 includes a processor 901 and a memory 902.

[0202] Processor 901 may include one or more processing cores, such as a 4-core processor, a 9-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU, which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 901 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0203] The memory 902 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 902 may also include high-speed random access memory devices and non-volatile storage devices, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 902 to implement the training method of the inference model provided in the various method embodiments of this application.

[0204] In some embodiments, the computer device 900 may also optionally include: a peripheral device interface 903 and at least one peripheral device.

[0205] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0206] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the computer device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0207] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements the training method for the inference model described above. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0208] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the training method of the inference model described above.

[0209] On the other hand, embodiments of this application provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the training method of the inference model provided in the embodiments of this application above.

[0210] On the other hand, embodiments of this application provide a computer device including the processor described above. Optionally, the processor is a GPU. The computer device can be at least one of a portable computer, a desktop computer, a server, a server cluster, an AI computing cluster, and a cloud computing cluster. The AI ​​computing cluster can also be simply referred to as an intelligent computing cluster or a smart computing cluster.

[0211] It should be noted that the collection and processing of relevant data (such as sample data) in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0212] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0213] Furthermore, the step numbers described herein are merely illustrative of one possible execution order between steps. In some other embodiments, the steps may not be executed in the order of their numbers, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0214] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a reasoning model, characterized in that, The method includes: Acquire sample data, which is used to describe the task requirements of the reasoning task; A first answer sequence for responding to the sample data is generated by a first model, and a second answer sequence for responding to the sample data is generated by a second model; wherein, the first sequence position of the first answer sequence corresponds to a plurality of first text words, and the first text words are associated with the predicted probabilities corresponding to the first text words; the first sequence position of the second answer sequence corresponds to a plurality of second text words, and the second text words are associated with the predicted probabilities corresponding to the second text words. Based on the predicted probabilities corresponding to the plurality of first text words, at least two selected text words are obtained from the plurality of first text words; Based on the predicted probabilities corresponding to the at least two selected text words and the predicted probabilities corresponding to the plurality of second text words, the second model is trained to obtain the trained second model, which is used to perform inference tasks.

2. The method according to claim 1, characterized in that, The step of obtaining at least two selected text words from the plurality of first text words based on the predicted probabilities corresponding to the plurality of first text words includes: Based on the plurality of first text words, a word sequence is obtained, wherein the word sequence includes the plurality of first text words; Obtain K selected text words from the word word sequence, where K is a positive integer, and the prediction probability corresponding to each of the K selected text words is higher than or equal to the prediction probability corresponding to any other text word in the word word sequence.

3. The method according to claim 2, characterized in that, Before obtaining the K selected text words in the word sequence, the method further includes: Obtain the task type of the inference task; Based on the task type, obtain the value of K.

4. The method according to claim 1, characterized in that, The method further includes: In the process of generating the first answer sequence, the first logical value corresponding to each of the plurality of first text words is obtained through the first model; and in the process of generating the second answer sequence, the second logical value corresponding to each of the plurality of second text words is obtained through the second model. Normalization is performed on multiple first logical values ​​to obtain the prediction probabilities corresponding to the multiple first text words respectively; and normalization is performed on multiple second logical values ​​to obtain the prediction probabilities corresponding to the multiple second text words respectively.

5. The method according to claim 1, characterized in that, The step of training the second model based on the predicted probabilities corresponding to the at least two selected text words and the predicted probabilities corresponding to the plurality of second text words to obtain the trained second model includes: Based on the predicted probabilities corresponding to the at least two selected text words, a first probability distribution corresponding to the at least two selected text words is obtained; and based on the predicted probabilities corresponding to the plurality of second text words, a second probability distribution corresponding to the plurality of second text words is obtained. Based on the first probability distribution and the second probability distribution, the second model is trained in multiple rounds to obtain the trained second model.

6. The method according to claim 5, characterized in that, The step of training the second model multiple times based on the first probability distribution and the second probability distribution to obtain the trained second model includes: Based on the first probability distribution and the second probability distribution, the distribution loss value for the current round is obtained, and the distribution loss value is used to indicate the difference between the first probability distribution and the second probability distribution; The second model is trained based on the distribution loss value of the current round. When the number of training rounds corresponding to the second model reaches a preset first training round threshold, the trained second model is obtained.

7. The method according to claim 6, characterized in that, The step of training the second model based on the distribution loss value of the current round, and obtaining the trained second model when the current training round number of the second model reaches a preset first training round number threshold, includes: Based on the distribution loss value of the current round, the first gradient value of the second model corresponding to the current round is obtained through the backpropagation algorithm; Based on the first gradient value of the current round, the model parameters of the second model are adjusted during the training of the current round. When the current training round number corresponding to the second model reaches the first training round number threshold, the trained second model is obtained.

8. The method according to claim 1, characterized in that, The method further includes: Based on the test sample data, at least one answer path is generated by the trained second model. The answer path includes the reasoning process corresponding to the reasoning task, and a third answer sequence for responding to the test sample data. Based on the at least one answer path, a reward signal corresponding to each of the at least one answer path is obtained through a reward model, and the reward signal is used to indicate the accuracy of the answer path; The trained second model is trained multiple times based on the reward signal until the current training round reaches a preset second training round threshold, thus obtaining the inference model.

9. The method according to claim 8, characterized in that, The step of training the trained second model multiple times based on the reward signal until the current training round number reaches a preset second training round number threshold to obtain the inference model includes: Based on the reward signal, the strategy loss value for the current round is obtained, and the strategy loss value is used to indicate the difference between the answer path and the target answer path corresponding to the reasoning task; Based on the policy loss value of the current round, the second gradient value of the trained second model for the current round is obtained through the backpropagation algorithm; Based on the second gradient value of the current round, the model parameters of the trained second model are adjusted in the training of the current round. When the current training round number corresponding to the trained second model reaches the second training round number threshold, the inference model is obtained.

10. The method according to claim 8, characterized in that, The step of obtaining the reward signal corresponding to each of the at least one answer path through a reward model includes: Based on the answer path, the third answer sequence and the reasoning process are obtained; The reward model is used to obtain the first reward signal corresponding to the third answer sequence and the second reward signal corresponding to the reasoning process. Based on the first reward signal and the second reward signal, obtain the reward signal corresponding to the answer path.

11. A training device for a reasoning model, characterized in that, The device includes: The acquisition module is configured to acquire sample data, which is used to describe the task requirements of the inference task. The generation module is configured to generate a first answer sequence for responding to the sample data using a first model, and to generate a second answer sequence for responding to the sample data using a second model; wherein, the first sequence position of the first answer sequence corresponds to a plurality of first text words, and the first text words are associated with the predicted probabilities corresponding to the first text words; the first sequence position of the second answer sequence corresponds to a plurality of second text words, and the second text words are associated with the predicted probabilities corresponding to the second text words. The acquisition module is further configured to acquire at least two selected text words from the plurality of first text words based on the predicted probabilities corresponding to the plurality of first text words respectively; The training module is configured to train the second model based on the prediction probabilities corresponding to the at least two selected text words and the prediction probabilities corresponding to the plurality of second text words, thereby obtaining the trained second model, which is used to perform inference tasks.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the training method for the inference model as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the training method for the inference model as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor reads from and executes the computer program to implement the training method of the inference model as described in any one of claims 1 to 10.

Images