Large language model reasoning data generation method based on reasoning probability distribution consistency

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a big language model reasoning data generation method based on reasoning probability distribution consistency, which comprises the following steps: S1, determining scenes or question types contained in a reasoning data set needing to be generated and an output format of reasoning data; s2, constructing a reference reasoning distribution sequence; s3, referring to the reference reasoning distribution sequence, guiding a large language model to generate reasoning data, and introducing an online consistency constraint when the reasoning data is generated; s4, taking the generated reasoning data as a training sample; and S5, respectively executing the steps S2-S4 for all scenes or question types. The method comprises the following steps: introducing a reference reasoning distribution time sequence as a typical reasoning habit standard in a reasoning data generation stage, extracting conditional probability distribution in each reasoning time step generated by a reasoning chain, and carrying out online constraint and dynamic deviation correction on a generation process through a consistency deviation degree; therefore, the reasoning data for training can be directly generated without post screening, and the generation efficiency and quality of the reasoning data are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a method for generating reasoning data for large language models, particularly a method for generating reasoning data for large language models based on the consistency of reasoning probability distribution. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have been widely used in natural language understanding, content generation, code generation, and complex reasoning. To improve the performance of large language models in specific tasks or application scenarios, model fine-tuning has become one of the mainstream model optimization methods. In practical applications, model fine-tuning not only relies on high-quality training data but also increasingly depends on high-quality, controllable, and reproducible inference data to guide the model to continuously improve inference chain generation, logical consistency, and stability.

[0003] In an ideal scenario for enhancing reasoning ability, training data should not only contain structured information such as "question-reasoning chain-answer," but also reflect probability distribution characteristics consistent with the model's current reasoning behavior during the reasoning process. This would enable the model to obtain stable and continuous gradient signals during training and improve the controllability of its reasoning behavior. However, in current large language model training practices, the generation and construction of reasoning data remains relatively crude, generally employing a method of "generating candidate samples on a large scale first, followed by posterior screening or manual sampling." This approach makes it difficult to constrain the quality and consistency of the reasoning chain during the generation stage, leading to problems such as high data generation costs, low availability, and reasoning chain style drift, thereby affecting the stability of fine-tuning effects and the efficiency of training resource utilization.

[0004] Especially with the increasing size of models and the growing complexity of inference processes, the quality differences in inference data are amplified during training: if the inference chain deviates, skips steps, or becomes redundant at certain key time steps, subsequent generation often accumulates along the wrong direction, eventually forming unusable samples. If posterior screening is still relied upon, it not only wastes a lot of computing power and time, but also makes it difficult to guarantee the consistency and controllability of different batches of data, thus restricting the efficiency improvement of inference data generation and model iteration.

[0005] In recent years, research and engineering practices related to the generation and construction of inference data can be mainly summarized into the following technical approaches: 1. Reasoning data generation scheme based on prompt word templates or rule constraints.

[0006] These approaches typically guide the model to generate structured "question-reasoning chain-answer" samples by manually writing prompt templates, specifying output formats, and setting lengths or key fields. While these methods are simple to implement and easy to deploy, their constraints are mostly limited to the output format or surface-level rules. They struggle to stably constrain the reasoning chain generation process at the reasoning mechanism level, leading to issues such as reasoning style drift, step jumps, redundant descriptions, or logical inconsistencies. This results in insufficient training usability and consistency of the generated data.

[0007] 2. Scheme for constructing inference data based on post-generation scoring or posterior screening.

[0008] Some solutions introduce rule-based scoring, discriminant models, reward models, or manual sampling after data generation to filter and clean candidate samples, thereby improving data quality. These solutions focus on post-generation quality control and can eliminate obvious errors to some extent, but they are essentially post-processing: it is difficult to detect and correct deviations in a timely manner during the generation process; moreover, post-processing often requires a large amount of redundant generation, resulting in low inference data generation efficiency, high computational consumption, and the selection thresholds and strategies are highly dependent on experience, making it difficult to guarantee stable consistency across batches.

[0009] 3. Reasoning data organization scheme based on difficulty level or phased generation.

[0010] To improve training efficiency, some existing technologies attempt to generate reasoning questions in stages or by grading their difficulty. For example, simple questions are generated first, followed by more complex ones, or the generation difficulty is dynamically adjusted based on model performance. However, these methods are mostly organized based on heuristic rules or results, lacking the ability to analyze the distribution and evolution of conditional probability during the reasoning process. This makes it difficult to ensure consistency between the "generation process" and the model's typical reasoning behavior, and thus may still produce a large number of samples with unstable reasoning chains or style drift.

[0011] 4. A scheme that separates the generation of inference data from the model's inference behavior.

[0012] In most existing inference data generation processes, data generation is primarily driven by templates or rules, while model inference behavior (such as the conditional probability distribution characteristics of each generation time step) is not explicitly incorporated into the generation control mechanism. In other words, data generation proceeds according to a predetermined method, while the model passively outputs results during the generation process. Due to the lack of a mechanism to introduce model inference distribution characteristics into the generation stage and apply them for online constraints and corrections, it is difficult to maintain consistency in inference data based on the model's current typical inference habits, thus limiting the controllability and specificity of the inference data generation process.

[0013] In summary, while existing inference data generation and construction schemes have supported model fine-tuning training to some extent, they still generally suffer from the following key technical deficiencies: (1) The inference data generation process lacks explicit perception and online constraint capabilities regarding the probability distribution characteristics of model inference; (2) It is difficult to characterize and control the consistency relationship between the inference chain and typical inference habits during the generation stage; (3) Over-reliance on post-generation screening, cleaning, or manual sampling leads to wasted computing power and low data generation efficiency; (4) It is difficult to guarantee the consistency and stability of data generated in different batches and scenarios, affecting the continuity and reproducibility of training; (5) It is difficult to provide refined, controllable, and sustainably iterative inference data support for enhancing the inference capabilities of large language models.

[0014] Based on the above analysis, there is an urgent need for a reasoning data generation technology that can constrain and regulate the generation process online from the perspective of the reasoning mechanism during the reasoning data generation stage. Summary of the Invention

[0015] To overcome the shortcomings of existing technologies, this invention proposes a method for generating reasoning data for large language models based on the consistency of reasoning probability distribution. This method constructs a reference reasoning probability distribution time series of the model as a standard for typical reasoning habits. At each reasoning time step in the generated reasoning chain, the model's output distribution is extracted to measure its degree of consistency deviation from the reference reasoning probability distribution. Based on this, the generation direction is adjusted online or the deviation steps are resampled, ensuring that the generated data meets the consistency requirements upon completion. This achieves a "generated and qualified" reasoning dataset construction method, providing a reliable data foundation for the efficient and stable training of large language models.

[0016] To achieve the above objectives, the present invention provides the following technical solution: A method for generating inference data for a large language model based on the consistency of inference probability distributions, characterized by the following steps: S1: Determine the scenarios or question types included in the reasoning dataset to be generated, as well as the output format of the reasoning data; S2: Randomly select a set of reference samples from the pre-selected inference data sources, and use a large language model to perform inference on each reference sample in the reference sample set to obtain a reference inference distribution sequence; S3: Based on the scenario or question type and the output format of the reasoning data, and referring to the reference reasoning distribution sequence, guide the large language model to generate reasoning data and introduce online consistency constraints when generating reasoning data to ensure that the generation process does not deviate from the reference reasoning habits; S4: Write the generated inference data as training samples into the inference dataset; S5: For each scenario or question type contained in the reasoning dataset to be generated, execute S2-S4 respectively to obtain the final reasoning dataset.

[0017] Preferably, the output format of the inference data is: , , In the formula, Indicates the first The first scenario or question type One piece of inference data; Indicates the first The first scenario or question type Input of inference data; Indicates the first The first scenario or question type The reasoning chain of the reasoning data; Represents the chain of reasoning The content of each inference time step in the process, among which, Represents the chain of reasoning The number of inference time steps, i.e., the inference chain. Length; Indicates the first The first scenario or question type The final answer to the inference data; Indicates the number of the inference data; Indicates the number of the scenario or question type.

[0018] Preferably, step S2 specifically includes: S21: Randomly select a set of reference samples from the pre-selected inference data sources: , In the formula, For reference sample set; For the first in the reference sample set One reference sample; This represents the number of reference samples in the reference sample set. S22: The model parameters of the large language model are At the current stage, inference is performed on each reference sample, in the... Each inference time step yields the conditional probability distribution corresponding to each reference sample: , In the formula, Indicates the first Each inference time step, the large language model reads the reference sample. The conditional probability distribution of the output at that time; The model parameters are: The conditional probability distribution function of a large language model; Indicates the first Context information that has been generated or aligned prior to each inference time step; S23: Regarding the first The conditional probability distributions corresponding to each reference sample obtained at each inference time step are aggregated to obtain the first... Reference inference probability distribution for each inference time step : , S24: Based on the reference inference probability distribution at each inference time step, obtain the reference inference distribution sequence. : , In the formula, This indicates the maximum length of the inference time step.

[0019] Preferably, in step S3, guiding the large language model to generate inference data based on the scenario or question type and the output format of the inference data, and referring to the reference inference distribution sequence, specifically includes: S31: Obtain the basic conditional probability distribution of the large language model itself at each inference time step: , In the formula, This indicates that the large language model itself is generating the first... The first scenario or question type When inferring data, in the first... The basic conditional probability distribution of the output at each inference time step; Indicates the first The first scenario or question type Inference chains in inference data In the first The contents of each inference time step preceding the current inference time step; S32: Based on the reference inference probability distribution and the basic conditional probability distribution for each inference time step, generate the guided inference probability distribution for each inference time step: , In the formula, Indicates the generation of the first The first scenario or question type When inferring data, the first The probability distribution of guided inference at the inference time step; Indicates the guiding coefficient; S33: At each inference time step, reason step by step according to the guided inference probability distribution to generate the inference chain and the answer.

[0020] Preferably, in step S3, introducing online consistency constraints when generating inference data to ensure that the generation process does not deviate from the reference inference habits specifically involves: when generating inference data, calculating the deviation between the reference inference probability distribution and the guided inference probability distribution at each inference time step, and making online adjustments based on the deviation to ensure that the generation process does not deviate from the reference inference habits.

[0021] Preferably, in step S3, the deviation between the reference inference probability distribution and the guided inference probability distribution for each inference time step is calculated based on KL divergence: , In the formula, Indicates the generation of the first The first scenario or question type When inferring data, the first Deviation at inference time step; This represents the KL divergence.

[0022] Preferably, in step S3, the online adjustment based on the deviation degree specifically involves: comparing the deviation degree with a deviation threshold; when the deviation degree is greater than the deviation threshold, increasing the guidance coefficient until the deviation degree is no greater than the deviation threshold.

[0023] Furthermore, this invention also provides a large language model reasoning data generation system based on the consistency of reasoning probability distribution, characterized in that it includes: The module for determining the pattern of the inference data to be generated is used to determine the scenarios or question types contained in the inference dataset to be generated and the output format of the inference data; The reference reasoning distribution construction module is used to randomly select a set of reference samples corresponding to the scenario or question type from a pre-selected reasoning data source, and perform reasoning on each reference sample in the reference sample set using a large language model to obtain a reference reasoning distribution sequence. The distribution similarity-guided inference data generation module is used to guide the large language model to generate inference data based on the scenario or question type and the output format of the inference data, and with reference to the reference inference distribution sequence. Online consistency constraints are introduced when generating inference data to ensure that the generation process does not deviate from the reference inference habits. The inference dataset generation module is used to write the generated inference data as training samples into the inference dataset.

[0024] Furthermore, the present invention also provides a large language model inference data generation device based on the consistency of inference probability distribution, characterized in that it includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the large language model inference data generation method based on the consistency of inference probability distribution as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the large language model inference data generation method based on the consistency of inference probability distribution as described above.

[0025] Compared with existing methods that mainly rely on prompt word templates, manual rule constraints, post-generation scoring and screening, or manual sampling to construct inference data, the large language model inference data generation method based on the consistency of inference probability distribution proposed in this invention has the following significant advantages in terms of inference data quality, data generation efficiency, controllability of the generation process, and stability of model fine-tuning: (1) The “Generation is Qualified” mechanism based on the consistency constraint of inference probability distribution improves the quality and availability of inference data from the source.

[0026] Existing inference data construction typically employs a "generate a large number of candidate samples first, then perform posterior screening" approach. This lack of distributional constraints on the inference chain generation behavior during the generation phase leads to frequent issues such as inference chain style drift, skipping steps, redundancy, or logical inconsistencies, resulting in low usability of the generated data. This invention extracts the conditional probability distribution of each inference time step during the generation process and uses a reference inference distribution time series as a standard for typical inference habits to apply consistency guidance and constraints to the generation process, ensuring that the inference chain maintains distributional consistency during the generation phase. Through this mechanism, the generated data can be directly used for training without relying on large-scale posterior screening, significantly improving the quality, stability, and training usability of the inference data from the source.

[0027] (2) Online consistency deviation assessment and dynamic correction avoid invalid generation caused by the accumulation of deviations, and significantly improve data generation efficiency.

[0028] In existing technologies, once the inference chain deviates from certain critical steps, subsequent generation often accumulates along the wrong direction, ultimately producing unusable data. Such errors are typically only detected after generation through screening, resulting in significant waste of computing power and time. This invention calculates the consistency deviation at each inference time step and adjusts the generation direction online based on the degree of deviation (e.g., dynamically adjusting the guidance strength), thereby achieving closed-loop control of "generating, quality checking, and correcting deviations simultaneously." This approach can correct deviations in a timely manner during the generation process, reduce invalid data output, and significantly improve the efficiency of inference data generation and the utilization rate of computing power.

[0029] (3) The generation control method does not require the introduction of additional teacher models or complex training frameworks, reducing system implementation and deployment costs. Unlike solutions that require the introduction of additional teacher models, reward models, or complex generation-screening pipelines, this invention obtains the time-step probability distribution based on the logits output by the large language model itself, and uses softmax to obtain the conditional probability distribution for consistency guidance and deviation evaluation. This method does not require modification of the large language model structure, nor does it rely on additional model structures or complex training scheduling strategies. It only adds lightweight distribution statistics and online control logic in the inference data generation stage to achieve the direct generation of high-quality inference data, with low engineering implementation costs and good deployability.

[0030] (4) Distributed control makes the generation process interpretable and controllable, which facilitates debugging and quality management.

[0031] Existing inference data generation often relies on empirical templates or posterior scoring, making it difficult to explain the root causes of inconsistencies in inference chain quality and to accurately pinpoint "at which step it starts to deviate" during the generation phase. This invention uses the consistency of the inference probability distribution as the basis for generation control, ensuring that each time step in the generation process has a clear metric (deviation rate) and corresponding corrective actions (parameter tuning / resampling). Through this approach, changes in the quality of the generated inference chain can be directly correlated with the degree of distribution deviation, thereby significantly enhancing the interpretability and controllability of the generation process at the inference mechanism level, facilitating engineering parameter tuning, quality regression, and data governance.

[0032] (5) Supports stable and consistent generation in multiple scenarios and batches, providing a unified and scalable data foundation for continuous training and phased optimization.

[0033] In practical applications, inference data often needs to be generated in batches for different question types / scenarios and continuously updated with model version iterations. Existing posterior-filtering solutions struggle to guarantee consistency in inference chain style across batches and scenarios, leading to data distribution drift and impacting training continuity and reproducibility. This invention characterizes typical inference habits of the model by referencing inference distribution time series and applies consistency guidance during the generation phase. This ensures that data generated in different batches and scenarios maintains consistency in inference style and distribution, providing a unified and scalable inference data foundation for continuous model training, phased inference enhancement, and version iteration.

[0034] In summary, this invention introduces an online generation control mechanism guided by the consistency of inference probability distribution. Without increasing the complexity of the model structure or relying on large-scale posterior screening, it achieves a high-quality construction method where inference data is "qualified upon generation". This significantly improves the efficiency of inference data generation, the controllability of the generation process, and the stability and performance improvement of subsequent model fine-tuning training. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution according to the present invention.

[0036] Figure 2 This is a schematic diagram of the structure of the large language model reasoning data generation system based on the consistency of reasoning probability distribution of the present invention.

[0037] Figure 3 This is a structural block diagram of the large language model inference data generation device based on the consistency of inference probability distribution of the present invention. Detailed Implementation

[0038] Before detailing any embodiment of the invention, it should be understood that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the following description or illustrated in the following figures. The invention can have other embodiments and can be practiced or carried out in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising” or “having” and variations thereof in this invention is intended to cover the items set forth below and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “linkage,” and variations thereof are used broadly and cover both direct and indirect installation, connection, support, and linking. Moreover, “connection” and “linkage” are not limited to physical or mechanical connections or links. Furthermore, firstly, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention. Secondly, the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity.

[0039] With the widespread application of generative large language models in scenarios such as intelligent question answering, automated programming, content generation, and complex reasoning, model fine-tuning and continuous training have become key means to improve the inference ability and stability of models. However, existing methods of constructing inference data are still relatively crude at the "data generation mechanism" level, often adopting a process of "generating a large amount first and then filtering afterwards." This makes it difficult to constrain the quality and consistency of the inference process during the generation stage, resulting in problems such as high data generation costs, low availability, and unstable inference chain styles, which in turn affect the efficiency of fine-tuning and the controllability of performance improvement. Specifically, current technical solutions mainly have the following prominent technical problems: (1) Existing inference data generation lacks the ability to constrain the distribution characteristics of model inference online.

[0040] Existing methods typically generate inference data through prompt word templates, length limits, or simple rules. However, the generation process itself lacks constraints and control over the "conditional probability distribution of the model's output at each inference time step." This leads to phenomena such as inference style drift, skipping steps, redundancy, or instability when the model generates inference chains. Consequently, the generated data is difficult to stably conform to the typical inference habits of the current model, thereby reducing the training usability of the generated data.

[0041] (2) Existing solutions cannot guarantee that the reasoning chain is consistent with the “typical reasoning habits” during the generation stage.

[0042] While some schemes can filter out unqualified samples after generation using scorers or manual rules, they are essentially post-generation screening methods. They cannot detect and correct time steps in the inference chain that deviate from the typical inference distribution during the generation process. Once the inference chain deviates, subsequent steps often accumulate in the wrong direction, eventually producing unusable data, resulting in a significant waste of computing power and time, and making the quality of the inference dataset highly dependent on the screening strategy and threshold experience.

[0043] (3) The lack of a reasoning data generation mechanism that ensures "qualification upon generation" leads to high data generation costs and low efficiency.

[0044] Current inference data generation often relies on a process of "generating a large number of candidates first, then screening / cleaning, and then manually sampling," which results in high generation redundancy, high screening costs, and uncontrollable quality. Especially when generating inference data continuously in multiple scenarios and batches, posterior screening cannot guarantee consistency and stability across batches, thus affecting the continuity and repeatability of subsequent fine-tuning training.

[0045] To address the aforementioned issues, this invention proposes a method for generating inference data for large language models based on the consistency of inference probability distributions. This invention introduces a "reference inference distribution time series" as a standard for typical inference habits during the inference data generation stage. Furthermore, it extracts the conditional probability distribution of the model output at each inference time step in the inference chain generation. The generation process is then constrained online and dynamically corrected through a consistency deviation metric. This allows for the direct generation of training-ready inference datasets without post-processing screening, improving the efficiency and quality stability of inference data generation.

[0046] Specifically, this invention improves the inference data generation mechanism in the following ways: when the generation distribution of the model at a certain inference time step is highly consistent with the reference inference distribution, the inference chain continues to be generated according to the current guidance strategy to maintain the stability of the inference style and ensure the usability of the generated samples; when the generation distribution of the model at a certain inference time step deviates significantly from the reference inference distribution, the guidance intensity is adjusted online (e.g., the guidance coefficient is dynamically adjusted) or resampling is performed on that time step to correct the generation direction in real time, avoid error accumulation, and improve the data output rate of "generated data is qualified".

[0047] The main technological breakthroughs achieved by this invention are reflected in the following three aspects: (1) Introduce an online generation guidance mechanism based on the consistency of inference probability distribution.

[0048] During the inference chain generation process, the conditional probability distribution of each inference time step of the model is extracted and constrained to be consistent with the reference inference distribution, so that the generation process has distribution-level controllability and ensures the stability and consistency of the generated samples from the inference level.

[0049] (2) Establish a reasoning data construction method of "generating is qualified" to reduce the dependence on posterior screening.

[0050] By calculating the consistency deviation and dynamically correcting it during the generation stage, the output data meets the preset consistency requirements when the generation is completed, which significantly reduces the waste of computing power and manual costs of "generating first and then filtering", and improves the efficiency of inference data generation.

[0051] (3) Implement a controllable reasoning data generation framework oriented towards typical reasoning habits of the model.

[0052] By referencing the time series of inference distributions to characterize the typical inference habits of the model and using them as a generation guide standard, the inference style of data generated in different scenarios and batches remains consistent, improving the stability, continuity, and repeatability of subsequent fine-tuning training, and providing a reliable data foundation for enhancing the inference capabilities of large language models.

[0053] Figure 1A flowchart of the method for generating reasoning data for large language models based on the consistency of reasoning probability distributions according to the present invention is shown. Figure 1 As shown, the method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution according to the present invention includes the following steps: S1: The format of the inference data to be generated is determined.

[0054] Determine the scenarios or question types included in the inference dataset to be generated, as well as the output format of the inference data.

[0055] In this invention, it is necessary to first prepare the scenario constraints and output format that enable large language models to generate inference data in batches, so as to ensure that the subsequently generated inference data is naturally trainable and contains the inference process. Specifically, the first step is to generate the seed set and select the batch scenario: .

[0056] In the formula, This represents the seed set for generating reasoning scenarios or question templates that can be generated (such as "two cars meeting problem / logical reasoning / common sense reasoning"). Indicates the first A seed generator (a description of a specific scenario or question type template); Indicates the number of seeds generated; This indicates the batch index, which is the scene or question type number. It only indicates "which scene or question type is being generated at the moment". Except for different scenes or question types, the generation process of each batch is the same. Indicates the first A subset of scenarios or question types used for a given scenario or question type.

[0057] Then, determine the output structure (question + reasoning chain + answer) of the generated samples (that is, the generated inference data): , .

[0058] In the formula, Indicates the first The first scenario or question type One piece of inference data; Indicates the first The first scenario or question type Input of inference data; Indicates the first The first scenario or question type The reasoning chain of the reasoning data; Represents the chain of reasoning The content of each inference time step in the process, among which, Represents the chain of reasoning The number of inference time steps, i.e., the inference chain. Length; Indicates the first The first scenario or question type The final answer to the inference data; Indicates the number of the inference data.

[0059] For any inference data that needs to be generated Its input All of this is known; the purpose of generating inference data is based on the input. Generate the corresponding inference chain and the final answer .

[0060] S2: Construction of reference inference distribution.

[0061] A set of reference samples is randomly selected from a pre-selected inference data source. Inference is then performed on each reference sample in the reference sample set using a large language model to obtain a reference inference distribution sequence, which specifically includes: 1. Pre-selected reference sample set.

[0062] In this invention, firstly, a set of reference samples is randomly selected from a pre-selected inference data source. The pre-selected inference data source may include at least one of the following: (1) A pre-built public reasoning task dataset; (2) Inference input data used in the training or evaluation of historical models; (3) A set of benchmark reasoning prompts that are designed manually or generated automatically.

[0063] The data in these pre-selected reasoning data sources are all data that can reflect the reasoning behavior characteristics of the large language model at the current stage, similar to "reference questions". Their reasoning behavior is used as the standard of "typical reasoning habits". The purpose of generating reasoning data is to generate reasoning data that is as similar as possible to the reasoning behavior characteristics of the "reference questions".

[0064] A set of reference samples randomly selected from the pre-selected inference data sources: .

[0065] In the formula, This represents the reference sample set, used to statistically analyze the reference inference distribution. It is generally decoupled from the generated inference data to ensure the stability of the baseline. Represents the first in the reference sample set One reference sample; This indicates the number of reference samples in the reference sample set, used to control the stability and representativeness of the reference sample distribution.

[0066] 2. Reference inference distribution collection.

[0067] The model parameters of the large language model are: At the current stage, inference is performed on each reference sample in the reference sample set, in the first... Each inference time step yields the conditional probability distribution corresponding to each reference sample: .

[0068] In the formula, Indicates the first Each inference time step, the large language model reads the reference sample. The conditional probability distribution of the output at that time; The model parameters are: The conditional probability distribution function of a large language model; This represents the model parameters of the large language model at the current stage, used to characterize the current rational ability state of the model; Indicates the first One reference sample; Indicates the first Contextual information that has been generated or aligned prior to each inference time step, including historical generated results or given inference chain prefixes.

[0069] In this context, the generative lexicon of the large language model is assumed to be a set of discrete symbols: .in, This represents the size of the generatrix that can be generated; that is, a large language model can generate... Each word element.

[0070] For the reference sample Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The above shows the predicted probability distribution for the next word: , , …, , ..., And satisfy: .

[0071] in, Indicates for reference sample Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The next word is given above. The predicted probability; Indicates for reference sample Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The next word is given above. The predicted probability; Indicates for reference sample Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The next word is given above. The predicted probability.

[0072] therefore, .

[0073] This invention extracts probability distribution features from the model inference process itself. By performing inference step by step on reference samples and recording the conditional probability distribution of the output at each time step, it can fully characterize the inference behavior of a large language model under the current parameter state, rather than focusing only on the final generated result.

[0074] 3. Aggregate to obtain the reference inference distribution.

[0075] For the first The conditional probability distributions corresponding to each reference sample obtained at each inference time step are aggregated to obtain the first... Reference inference probability distribution for each inference time step : .

[0076] In the formula, Indicates the first The reference inference probability distribution for each inference time step, i.e., the aggregated result of the inference distribution of multiple (K) reference samples at that time step; Indicates the first The reference sample at the ... The conditional probability distribution corresponding to each inference time step; This indicates the number of reference samples in the reference sample set, used to average the inference distribution of multiple reference samples.

[0077] 4. Reference inference distribution sequence.

[0078] Based on the reference inference probability distribution at each inference time step, the reference inference distribution sequence is obtained. : .

[0079] In the formula, It represents the complete reference reasoning distribution sequence, used to characterize the distribution evolution trajectory of the large language model throughout the reasoning process, and can guide subsequent generation; This represents the maximum length of the inference time step, and the number of steps in which the model is generated or aligned during a single inference process.

[0080] Therefore, this step is equivalent to selecting a batch of reference questions and having the large language model perform inference generation in an autoregressive manner; at each inference time step (i.e., the first) (Second generation step) Extract the logits from the model output and obtain the conditional probability distribution for that time step using softmax. The probability distributions of all reference questions at the same time step are aggregated to form a reference reasoning distribution time series. This serves as a guiding standard for subsequent generation.

[0081] S3: Generation of inference data guided by distribution similarity.

[0082] Based on the scenario or question type and the output format of the reasoning data, and referring to the reference reasoning distribution sequence, the large language model is guided to generate reasoning data, and online consistency constraints are introduced when generating reasoning data to ensure that the generation process does not deviate from the reference reasoning habits.

[0083] In this invention, if in the first... Batch scenarios Next, inference data is generated. When generating the inference chain, each step uses a mixture of "model's own distribution + reference distribution" to guide the large language model in generating an inference process that better conforms to typical inference habits. Specifically, this includes: 1. Obtain the basic conditional probability distribution of the large language model itself at each inference time step: .

[0084] In the formula, This indicates that the large language model itself is generating the first... The first scenario or question type When inferring data, in the first... The basic conditional probability distribution of the output at each inference time step; Indicates the first The first scenario or question type Input of inference data; Indicates the first The first scenario or question type Reasoning chain of reasoning data In the first The content of each inference time step before the first inference time step, that is, the first inference time step. Contextual information during each inference time step.

[0085] Similarly, suppose the generative lexicon of a large language model is a set of discrete symbols: .in, This represents the size of the generatrix that can be generated; that is, a large language model can generate... Each word element.

[0086] For the The first scenario or question type Input of inference data Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The above shows the predicted probability distribution for the next word: , , ..., , ..., And satisfy: .

[0087] in, Indicates for the first The first scenario or question type Input of inference data Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The next word is given above. The predicted probability; Indicates for the first The first scenario or question type Input of inference data Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The next word is given above. The predicted probability; Indicates for the first The first scenario or question type Input of inference data Large language models in the first Each inference time step is based on context. and model parameters In a generative lexicon The next word is given above. The predicted probability.

[0088] therefore, .

[0089] 2. Based on the reference inference probability distribution and the basic conditional probability distribution for each inference time step, generate the guided inference probability distribution for each inference time step: .

[0090] In the formula, Indicates the generation of the first The first scenario or question type When inferring data, the first The probability distribution of guided inference at the inference time step; This represents the guiding coefficient, which controls the degree to which the reference inference probability distribution participates. It is generated entirely according to the model's own distribution. It is generated entirely according to the reference inference probability distribution. The larger the value, the closer the generation process is to the reference reasoning habits.

[0091] 3. At each inference time step, reason step by step according to the guided inference probability distribution to generate the inference chain and the answer.

[0092] That is, at the t-th inference time step, according to Generate the reasoning chain and answer: at the t-th time step, from Sample or select the next generated word and append it to the inference prefix; repeat the above process until a complete inference chain is generated. And further generate the corresponding final answer. .

[0093] To ensure that the generation process does not deviate from the reference reasoning conventions, this invention introduces online constraints during generation: if a step deviates too much, the constraint is increased. This allows the generated output inference data to be used immediately.

[0094] In this invention, when generating inference data, the deviation between the reference inference probability distribution and the guided inference probability distribution at each inference time step is calculated, and online adjustments are made based on the deviation to ensure that the generation process does not deviate from the reference inference habits.

[0095] Specifically, the deviation between the reference inference probability distribution and the guided inference probability distribution at each inference time step is calculated based on KL divergence: , In the formula, Indicates the generation of the first The first scenario or question type When inferring data, the first The deviation at the inference time step is used to measure the magnitude of the difference between the current guiding distribution and the reference distribution; This represents the KL divergence, used to measure the difference between the reference inference probability distribution and the guiding inference probability distribution.

[0096] Online adjustment is performed based on the aforementioned deviation; if... Then improve until That is, the deviation. and deviation threshold In comparison, when the deviation is... Greater than the deviation threshold At the same time, increase the guiding coefficient. Until the deviation is no greater than the deviation threshold. This indicates a deviation from the threshold. If the deviation exceeds the threshold, it is considered that the step deviates too much from typical reasoning habits. It can be set as needed. This indicates the adjustment step size of the guiding coefficient.

[0097] S4: Inference dataset generation.

[0098] The generated inference data is written into the inference dataset as training samples.

[0099] In this invention, by deviation The generation process can be adjusted online (e.g., the guiding coefficient can be dynamically adjusted). This ensures that the generated inference chain satisfies the reference inference probability distribution at each inference time step. Consistency constraints. Therefore, this invention outputs each inference data after completing online adjustment. All of them can be directly used as training samples and written into the final generated inference dataset. .

[0100] S5: For each scenario or question type contained in the reasoning dataset to be generated, execute S2-S4 respectively to obtain the final reasoning dataset.

[0101] Through different scenarios or question types Repeat the above generation process and accumulate inference data to obtain the final inference dataset that covers multiple scenarios or problems and can be directly used for training / fine-tuning.

[0102] This invention can be widely applied to training and fine-tuning scenarios of large language models that require high levels of inference ability, stability of inference chains, and efficiency of data generation, including but not limited to fine-tuning of inference instructions, enhanced training of inference chains, continuous learning across multiple scenarios, domain-adaptive training, and construction of regression evaluation data for inference ability.

[0103] In the above applications, the present invention can complete consistency constraints and online correction during the inference data generation stage, significantly reducing the reliance on post-generation screening, manual sampling and over-generation, and improving the efficiency of inference data generation and cross-batch consistency. At the same time, since the reference inference distribution can be dynamically updated with model iteration, the present invention can support a long-term continuous generation and continuous training closed loop, providing a controllable and reproducible data foundation for the steady improvement of the inference capabilities of large language models.

[0104] Furthermore, this invention can be integrated into existing LLMOps / training platforms as the core generation control and online quality inspection module of a large model inference data generator, possessing good engineering feasibility and broad industrial application prospects. It is particularly suitable for application scenarios that require the continuous generation of high-quality inference training data but are sensitive to computing power and labor costs.

[0105] Figure 2 A schematic diagram of the structure of the large language model reasoning data generation system based on the consistency of reasoning probability distribution according to the present invention is shown. Figure 2 As shown, the large language model reasoning data generation system based on the consistency of reasoning probability distribution of the present invention includes: 1. Module for determining the pattern of inference data to be generated.

[0106] The module for determining the style of the inference data to be generated is used to determine the scenarios or question types contained in the inference dataset to be generated and the output format of the inference data.

[0107] 2. Reference inference distribution construction module.

[0108] The reference reasoning distribution construction module is used to randomly select a set of reference samples corresponding to the scenario or question type from a pre-selected reasoning data source, and use a large language model to perform reasoning on each reference sample in the reference sample set to obtain a reference reasoning distribution sequence.

[0109] 3. Distribution similarity-guided inference data generation module.

[0110] The distribution similarity-guided inference data generation module is used to guide the large language model to generate inference data based on the scenario or question type and the output format of the inference data, and with reference to the reference inference distribution sequence. Online consistency constraints are introduced when generating inference data to ensure that the generation process does not deviate from the reference inference habits.

[0111] 4. Inference Dataset Generation Module.

[0112] The inference dataset generation module is used to write the generated inference data as training samples into the inference dataset.

[0113] Furthermore, this invention also provides a device for generating reasoning data for large language models based on the consistency of reasoning probability distributions. For example... Figure 3 As shown, the large language model inference data generation device based on the consistency of inference probability distribution of the present invention includes: a memory 11 for storing one or more programs; one or more processors 12; when the one or more programs are executed by the one or more processors 12, the one or more processors 12 implement the large language model inference data generation method based on the consistency of inference probability distribution of the present invention. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for generating large language model inference data based on the consistency of inference probability distribution in the present invention.

[0114] The computer-readable storage medium includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this invention, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0115] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this invention can be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution, characterized in that, Includes the following steps: S1: Determine the scenarios or question types included in the reasoning dataset to be generated, as well as the output format of the reasoning data; S2: Randomly select a set of reference samples from the pre-selected inference data sources, and use a large language model to perform inference on each reference sample in the reference sample set to obtain a reference inference distribution sequence; S3: Based on the scenario or question type and the output format of the reasoning data, and referring to the reference reasoning distribution sequence, guide the large language model to generate reasoning data and introduce online consistency constraints when generating reasoning data to ensure that the generation process does not deviate from the reference reasoning habits; S4: Write the generated inference data as training samples into the inference dataset; S5: For each scenario or question type contained in the reasoning dataset to be generated, execute S2-S4 respectively to obtain the final reasoning dataset.

2. The method for generating reasoning data for a large language model based on the consistency of inference probability distribution according to claim 1, characterized in that, The output format of the inference data is: ， ， In the formula, Indicates the first The first scenario or question type One piece of inference data; Indicates the first The first scenario or question type Input of inference data; Indicates the first The first scenario or question type The reasoning chain of the reasoning data; Represents the chain of reasoning The content of each inference time step in the process, among which, Represents the chain of reasoning The number of inference time steps, i.e., the inference chain. Length; Indicates the first The first scenario or question type The final answer to the inference data; Indicates the number of the inference data; Indicates the number of the scenario or question type.

3. The method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution according to claim 2, characterized in that, Step S2 specifically includes: S21: Randomly select a set of reference samples from the pre-selected inference data sources: ， In the formula, For reference sample set; For the first in the reference sample set One reference sample; This represents the number of reference samples in the reference sample set. S22: The model parameters of the large language model are At the current stage, inference is performed on each reference sample, in the... Each inference time step yields the conditional probability distribution corresponding to each reference sample: ， In the formula, Indicates the first Each inference time step, the large language model reads the reference sample. The conditional probability distribution of the output at that time; The model parameters are: The conditional probability distribution function of a large language model; Indicates the first Context information that has been generated or aligned prior to each inference time step; S23: Regarding the first The conditional probability distributions corresponding to each reference sample obtained at each inference time step are aggregated to obtain the first... Reference inference probability distribution for each inference time step : ， S24: Based on the reference inference probability distribution at each inference time step, obtain the reference inference distribution sequence. : ， In the formula, This indicates the maximum length of the inference time step.

4. The method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution according to claim 3, characterized in that, In step S3, guiding the large language model to generate inference data based on the scenario or question type and the output format of the inference data, and referring to the reference inference distribution sequence, specifically includes: S31: Obtain the basic conditional probability distribution of the large language model itself at each inference time step: ， In the formula, This indicates that the large language model itself is generating the first... The first scenario or question type When inferring data, in the first... The basic conditional probability distribution of the output at each inference time step; Indicates the first The first scenario or question type Reasoning chain of reasoning data In the first The contents of each inference time step preceding the current inference time step; S32: Based on the reference inference probability distribution and the basic conditional probability distribution for each inference time step, generate the guided inference probability distribution for each inference time step: ， In the formula, Indicates the generation of the first The first scenario or question type When inferring data, the first The probability distribution of guided inference at the inference time step; Indicates the guiding coefficient; S33: At each inference time step, reason step by step according to the guided inference probability distribution to generate the inference chain and the answer.

5. The method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution according to claim 4, characterized in that, In step S3, introducing online consistency constraints when generating inference data to ensure that the generation process does not deviate from the reference inference habits specifically involves: when generating inference data, calculating the deviation between the reference inference probability distribution and the guided inference probability distribution at each inference time step, and making online adjustments based on the deviation to ensure that the generation process does not deviate from the reference inference habits.

6. The method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution according to claim 5, characterized in that, In step S3, the deviation between the reference inference probability distribution and the guided inference probability distribution at each inference time step is calculated based on KL divergence: ， In the formula, Indicates the generation of the first The first scenario or question type When inferring data, the first Deviation at inference time step; This represents the KL divergence.

7. The method for generating reasoning data for a large language model based on the consistency of reasoning probability distribution according to claim 6, characterized in that, In step S3, the online adjustment based on the deviation degree specifically involves comparing the deviation degree with a deviation threshold. When the deviation degree is greater than the deviation threshold, the guiding coefficient is increased until the deviation degree is no greater than the deviation threshold.

8. A large language model reasoning data generation system based on the consistency of reasoning probability distribution, characterized in that, include: The module for determining the pattern of the inference data to be generated is used to determine the scenarios or question types contained in the inference dataset to be generated and the output format of the inference data; The reference reasoning distribution construction module is used to randomly select a set of reference samples corresponding to the scenario or question type from a pre-selected reasoning data source, and perform reasoning on each reference sample in the reference sample set using a large language model to obtain a reference reasoning distribution sequence. The distribution similarity-guided inference data generation module is used to guide the large language model to generate inference data based on the scenario or question type and the output format of the inference data, and with reference to the reference inference distribution sequence. Online consistency constraints are introduced when generating inference data to ensure that the generation process does not deviate from the reference inference habits. The inference dataset generation module is used to write the generated inference data as training samples into the inference dataset.

9. A device for generating reasoning data for a large language model based on the consistency of reasoning probability distribution, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating large language model inference data based on the consistency of inference probability distribution as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method for generating large language model inference data based on the consistency of inference probability distribution as described in any one of claims 1-7.

Images