Formatted knowledge reasoning method and system based on synergy of big and small language models

Abstract

The invention provides a formatted knowledge reasoning method and system based on synergy of big and small language models, and the method comprises the steps: generating a basic answer in a natural language form on an original reasoning task for any big language model; generating a correct reasoning path according to the reasoning problem and the basic answer by using an iteration method through a small language model; according to a specific format instruction, the reasoning question, the basic answer, the reasoning step and the standard answer are reconstructed into a novel reasoning data set; carrying out primary training on any open source small language model; and constructing a reward function to score the output of the small language model after the primary training, and performing secondary training according to the score to weaken the influence of secondary factors in the original training target on the training result. According to the method, the accuracy of utilizing the knowledge of the large language model by the small language model is remarkably improved, and the ability to follow user instruction requirements is enhanced, so that the high performance of the language model in complex reasoning and user instruction following is realized.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically, to a formatted knowledge reasoning method and system based on the collaboration of large and small language models. Background Technology

[0002] With the rapid development of large-scale pre-trained language models, significant progress has been made in both complex reasoning and instruction compliance, demonstrating strong performance in tasks such as general question answering and text generation. However, in real-world applications, there is often a need to simultaneously handle complex reasoning and strictly adhere to user instructions. For example, in clinical case generation, the system not only needs to perform complex diagnostic reasoning based on patient information but also must output results according to the standardized format of medical documents. In educational scenarios, the demonstration of problem-solving processes requires not only accurate calculations but also strict adherence to user-defined expressions or formatting requirements. However, when directly solving these tasks using existing large language models, there is often a tendency to focus on the reasoning behind complex problems, while insufficient adherence to user instructions results in logically correct outputs that fail to meet practical requirements in terms of format or structure. This imbalance limits the application value of large language models in scenarios requiring high reliability and standardization.

[0003] An intuitive solution is to fine-tune the large language model for specific tasks and instruction requirements. However, due to the massive number of parameters in large language models, the fine-tuning process often requires extremely high computational resources and time costs. Even with lightweight parameter-efficient fine-tuning methods, it is difficult to avoid interference with the original knowledge reserves of the large language model, potentially leading to catastrophic amnesia. Furthermore, for some closed-source commercial large language models, developers cannot directly obtain their parameter information, thus preventing any form of fine-tuning training, which further limits the feasibility of this approach.

[0004] To address these challenges, the research community has proposed a collaborative approach between large and small language models. These methods aim to leverage the powerful knowledge and reasoning capabilities of the large language model while enhancing the adaptability of the small language model to specific tasks through efficient fine-tuning. However, existing collaborative methods still have shortcomings. They fail to effectively guide the small language model to fully utilize the foundational knowledge provided by the large language model, leading to new and more serious errors in the small language model when correcting mistakes or optimizing output, ultimately harming overall performance.

[0005] Therefore, there is an urgent need in the market for a new method for the collaboration of large and small language models, which can fully utilize the powerful reasoning ability of large language models, efficiently improve the compliance of small language models with user instructions, and avoid introducing new errors. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a formatted knowledge reasoning method and system based on the collaboration of large and small language models.

[0007] A formatted knowledge reasoning method based on the collaboration of large and small language models, provided by the present invention, includes:

[0008] Step S1: For any large language model, generate a basic answer in natural language form on the original inference task;

[0009] Step S2: Using a small language model, generate the correct reasoning path based on the reasoning question and the basic answer using an iterative method;

[0010] Step S3: Based on specific format instructions, reconstruct the reasoning questions, basic answers, reasoning steps, and standard answers into a new reasoning dataset with specific format output and thought chain;

[0011] Step S4: Train any open-source small language model once using the novel inference dataset;

[0012] Step S5: Construct a reward function to score the output of the small language model after one training, and perform secondary training based on the score to reduce the influence of secondary factors in the original training objective on the training results.

[0013] Preferably, the arbitrary large language model includes all open-source large language models and closed-source large language models.

[0014] Preferably, the generation of the basic answer includes processing the original dataset X = {x1, x2, ... x...} N} and the corresponding standard answer Y={y1,y2,...,y N Large Language Model The basic response in natural language form is as follows:

[0015]

[0016] Preferably, step S2 includes:

[0017] Step S2.1: For the original problem x i Standard answer y i The basic answer of the large language model Small language model The initial inference path and prediction result are generated as follows:

[0018]

[0019] In the formula, r i Indicates the reasoning path, a i Indicates the prediction result;

[0020] Step S2.2: Analyze the model's prediction results a i Make a judgment if a i With y i If they are equal, then the corresponding reasoning path r is considered to be... i Correct and retain the reasoning path r i If a i With y i If they are not equal, then the standard answer y will be... i The input is given to the small language model as a reference for execution step S2.3;

[0021] Step S2.3: Generate the inference path and answer for the small language model for the second time, as follows:

[0022]

[0023] Step S2.4: Again, for the current a i Judge whether a is correct or incorrect. i If correct, then retain the corresponding reasoning path r. i Otherwise, delete the corresponding sample directly.

[0024] Preferably, the novel inference dataset As shown in the following formula:

[0025]

[0026] t i =(y f ,r i ,y i )

[0027] Where N′ represents the length of the dataset after the inference path is generated, t i This indicates that the correct inference path r is coupled. i and the standard answer y i Formatted output, y f This indicates a specific output format.

[0028] Preferably, the objective of a single training iteration is to minimize the cross-entropy loss function;

[0029] The training session includes using the original problem x i The basic answer of the large language model As input for training the small language model, the output t is formatted. i To achieve the training objective, the cross-entropy loss function is used for training. For a small language model during training, the training loss function is as follows:

[0030]

[0031] in, Let π represent the first-order training loss function of the small language model. θ t represents the small language model during training. i This indicates that the correct inference path r is coupled. i and the standard answer y i Formatted output, x i This represents the original question. Representing problem x i The basic answer to the corresponding large model.

[0032] Preferably, the construction of the reward function to score the output of the small language model after one training session includes:

[0033] For the standard output t = (y) in the dataset f Model output of (r,y) and small language model The reward function formula is as follows:

[0034]

[0035] Preferably, the secondary training employs a group relative policy optimization algorithm. A reward function `reward()` is constructed to perform a second training on the small language model, and the training loss function is:

[0036]

[0037] in, Let G represent the quadratic training loss function of the small language model, and let G represent the number of answers sampled from the old policy for each question. ρ i ε represents the ratio of the current policy to the old policy; ε is a hyperparameter of the shearing function, used to control the maximum magnitude of each policy update; β is a hyperparameter used to adjust the population difference penalty term in the loss function. A represents the KL divergence between the current policy model and the reference model. i This represents the group relative advantage estimate for each sample.

[0038] Preferably, the formula for estimating the group relative advantage of each sample is as follows:

[0039]

[0040] Where r1,…,r G r represents the reward value calculated using the reward function reward() for the answer sampled from the old strategy. i This represents the reward value calculated using the reward function reward() for the current policy response;

[0041] The formula for the ratio of the current strategy to the old strategy is as follows:

[0042]

[0043] The formula for calculating the KL divergence between the current policy model and the reference model is as follows:

[0044]

[0045] Where, π θ Represents the current policy model, π ref This represents the reference model.

[0046] A formatted knowledge reasoning system based on big-small language model collaboration, provided by the present invention, includes:

[0047] Basic Response Generation Module: For any large language model, generate basic responses in natural language form on the original inference task;

[0048] Data Reconstruction Module: Using a small language model, an iterative method is used to generate the correct reasoning path based on the reasoning question and the basic answer; according to specific format instructions, the reasoning question, basic answer, reasoning steps and standard answer are reconstructed into a new reasoning dataset with specific format output and thought chain;

[0049] One-time training module: Train any open-source small language model once based on the novel inference dataset;

[0050] Secondary training module: Construct a reward function to score the output of the small language model after the first training, and perform secondary training based on the score to reduce the influence of secondary factors in the original training objective on the training results.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. This invention significantly improves the accuracy of small language models in utilizing the knowledge of large language models without destroying the original knowledge reserves of large language models by integrating the natural language response and reasoning thought chain of large language models in the input and output respectively, and adopting a two-stage training mode.

[0053] 2. This invention enhances the ability to follow user instructions, thereby achieving high performance of the language model in complex reasoning and following user instructions. Attached Figure Description

[0054] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0055] Figure 1 This is a schematic diagram of the formatted knowledge reasoning method based on the collaboration of large and small language models according to the present invention;

[0056] Figure 2 This is a schematic diagram of the mathematical knowledge reasoning network of the present invention. Detailed Implementation

[0057] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0058] This invention generates basic answers in natural language form for any large language model on reasoning problems; through iterative steps, it enables a small language model to generate the correct knowledge reasoning path based on the question and basic answers; according to user instructions, it reconstructs the question, basic answers, reasoning path, and standard answer into a new reasoning dataset with specific output format and thought chain; it trains the small language model once using a new training set; and it designs a reward function to train the small language model a second time.

[0059] Example 1

[0060] According to the present invention, a formatted knowledge reasoning method based on the collaboration of large and small language models is provided, such as... Figure 1 As shown, it includes:

[0061] Step S1: For any large language model, generate a basic answer in natural language form on the original inference task. The arbitrary large language model includes all open-source and closed-source large language models. The generation of the basic answer includes: on the original dataset X = {x1, x2, ... x...} N} and the corresponding standard answer Y={y1,y2,...,y N Large Language Model The basic response in natural language form is as follows:

[0062]

[0063] Step S2: Using a small language model, an iterative method is employed to generate the correct reasoning path based on the reasoning question and the basic answer. Step S2 includes:

[0064] Step S2.1: For the original problem x i Standard answer y i The basic answer of the large language model Small language model The initial inference path and prediction result are generated as follows:

[0065]

[0066] In the formula, r i Indicates the reasoning path, a i This indicates the prediction result.

[0067] Step S2.2: Analyze the model's prediction results a i Make a judgment if a i With y i If they are equal, then the corresponding reasoning path r is considered to be... i Correct and retain the reasoning path r i If a i With y i If they are not equal, then the standard answer y will be... i The input is given to the small language model as a reference for execution step S2.3.

[0068] Step S2.3: Generate the inference path and answer for the small language model for the second time, as follows:

[0069]

[0070] Step S2.4: Again, for the current a i Judge whether a is correct or incorrect. i If correct, then retain the corresponding reasoning path r. i Otherwise, delete the corresponding sample directly.

[0071] Step S3: Based on specific format instructions, reconstruct the reasoning question, basic answer, reasoning steps, and standard answer into a new reasoning dataset with a specific format output and thought chain. The new reasoning dataset... As shown in the following formula:

[0072]

[0073] t i =(y f ,r i ,y i )

[0074] Where N′ represents the length of the dataset after the inference path is generated, t i This indicates that the correct inference path r is coupled. i and the standard answer y i Formatted output, y f This indicates a specific output format.

[0075] Step S4: Train any open-source small language model once using the novel inference dataset. The training objective is to minimize the cross-entropy loss function. This single training step includes: using the original problem x... i The basic answer of the large language model As input for training the small language model, the output t is formatted. i For training purposes, a cross-entropy loss function is used. This allows the small language model to learn specific output format information and the mapping between the question and the natural language responses from the large language model to the standard answer during training. The training loss function for the small language model is as follows:

[0076]

[0077] in, Let π represent the first-order training loss function of the small language model. θ t represents the small language model during training. i This indicates that the correct inference path r is coupled. i and the standard answer y i Formatted output, x i This represents the original question. Representing problem x i The basic answer to the corresponding large model.

[0078] Step S5: Construct a reward function to score the output of the small language model after one training iteration. Perform secondary training based on the score to reduce the influence of secondary factors in the original training objective on the training results. This is to avoid the small language model focusing excessively on the inference path (r) and neglecting the more important format information (y) during the learning process due to excessively long inference path text length. f In learning the standard answer y, this invention eliminates the influence of the inference path on the training results during the second training process by constructing a reward function reward(). The construction of the reward function to score the output of the small language model after one training iteration includes: [the following is a separate section, likely related to the previous sentence about the standard output t = (y) in the dataset]. f Model output of (r,y) and small language model The reward function formula is as follows:

[0079]

[0080] The secondary training employs a group relative policy optimization algorithm. A reward function `reward()` is constructed to perform a second training on the small language model. The training loss function is:

[0081]

[0082] in, Let G represent the quadratic training loss function of the small language model, and let G represent the number of answers sampled from the old policy for each question. ρ iε represents the ratio of the current policy to the old policy; ε is a hyperparameter of the shearing function, used to control the maximum magnitude of each policy update; β is a hyperparameter used to adjust the population difference penalty term in the loss function. A represents the KL divergence between the current policy model and the reference model. i The formula for estimating the relative group advantage for each sample is as follows:

[0083]

[0084] Where r1,…,r G r represents the reward value calculated using the reward function reward() for the answer sampled from the old strategy. i This represents the reward value calculated using the reward function reward() for the current policy response.

[0085] The formula for the ratio of the current strategy to the old strategy is as follows:

[0086]

[0087] The formula for calculating the KL divergence between the current policy model and the reference model is as follows:

[0088]

[0089] Where, π θ Represents the current policy model, π ref This represents the reference model.

[0090] This invention reconstructs a novel reasoning dataset from the original dataset by generating natural language responses from a large language model and correct thought processes from a small language model. This dataset lacks specific output formats and thought processes. Furthermore, by training the small language model in two stages, high performance is achieved on reasoning tasks with specific output format requirements, avoiding the high overhead of directly training the large language model.

[0091] The present invention aims to solve the problem that large language models have difficulty simultaneously following user instructions when handling complex reasoning tasks.

[0092] Example 2

[0093] The present invention also provides a formatted knowledge reasoning system based on big-small language model collaboration. The formatted knowledge reasoning system based on big-small language model collaboration can be implemented by executing the process steps of the formatted knowledge reasoning method based on big-small language model collaboration. That is, those skilled in the art can understand the formatted knowledge reasoning method based on big-small language model collaboration as a preferred embodiment of the formatted knowledge reasoning system based on big-small language model collaboration.

[0094] According to the present invention, a formatted knowledge reasoning system based on the collaboration of large and small language models is provided, such as... Figure 2 As shown, it includes:

[0095] Basic Response Generation Module: For any large language model, generate basic responses in natural language form on the original inference task. The "arbitrary large language model" includes all open-source and closed-source large language models. The generation of the basic responses involves processing the original dataset X = {x1, x2, ... x...}. N} and the corresponding standard answer Y={y1,y2,...,y N Large Language Model The basic response in natural language form is as follows:

[0096]

[0097] The dataset reconstruction module uses a small language model and an iterative method to generate the correct reasoning path based on the reasoning question and the basic answer generated by the large language model. First, iteratively, using an untrained small language model, the correct reasoning path is generated based on the natural language answers from the original dataset and the large language model. Then, according to specific format instructions, the reasoning question, basic answer, reasoning module, and standard answer are reconstructed into a new reasoning dataset with specific format output and thought chain. Module M2 includes: Module M2.1: For the original question x... i Standard answer y i The basic answer of the large language model Small language model The initial inference path and prediction result are generated as follows:

[0098]

[0099] In the formula, r i Indicates the reasoning path, a i Indicates the prediction result; Module M2.2: Prediction result a of the model. i Make a judgment if a i With y i If they are equal, then the corresponding reasoning path r is considered to be... i Correct and retain the reasoning path r i If a i With y i If they are not equal, then the standard answer y will be... i The input is given to the small language model as a reference for execution module M2.3; module M2.3 generates the inference path and answer for the small language model a second time, as shown in the following formula: Module M2.4: Again, for the current a i Judge whether a is correct or incorrect. iIf correct, then retain the corresponding reasoning path r. i Otherwise, the corresponding sample is deleted directly. The novel inference dataset... As shown in the following formula:

[0100]

[0101] t i =(y f ,r i ,y i )

[0102] Where N′ represents the length of the dataset after the inference path is generated, t i This indicates that the correct inference path r is coupled. i and the standard answer y i Formatted output, y f This indicates a specific output format.

[0103] One training module: Supervised training is performed on any open-source small language model on a newly reconstructed inference dataset with specific output formats and thought chains. The training objective is to minimize the cross-entropy loss function. The objective of one training module is to minimize the cross-entropy loss function; this one training includes using the original problem x. i The basic answer of the large language model As input for training the small language model, the output t is formatted. i To achieve the training objective, the cross-entropy loss function is used for training. For a small language model during training, the training loss function is as follows:

[0104]

[0105] in, Let π represent the first-order training loss function of the small language model. θ t represents the small language model during training. i This indicates that the correct inference path r is coupled. i and the standard answer y i Formatted output, x i This represents the original question. Representing problem x i The basic answer to the corresponding large model.

[0106] The secondary training module: By constructing a reward function, a population relative strategy optimization algorithm is used to perform secondary training on the small language model, reducing the influence of secondary factors in the original training objective on the training results. The construction of the reward function to score the output of the small language model after the first training includes: [the following is a separate section, likely related to a specific dataset or algorithm, and is not directly related to the module description]. f Model output of (r,y) and small language model The reward function formula is as follows:

[0107]

[0108] The secondary training employs a group relative policy optimization algorithm. A reward function `reward()` is constructed to perform a second training on the small language model. The training loss function is:

[0109]

[0110] in, Let G represent the quadratic training loss function of the small language model, and let G represent the number of answers sampled from the old policy for each question. ρ i ε represents the ratio of the current policy to the old policy; ε is a hyperparameter of the shearing function, used to control the maximum magnitude of each policy update; β is a hyperparameter used to adjust the population difference penalty term in the loss function. A represents the KL divergence between the current policy model and the reference model. i This represents the group relative advantage estimate for each sample. The formula for estimating the group relative advantage for each sample is as follows:

[0111]

[0112] Where r1,…,r G r represents the reward value calculated using the reward function reward() for the answer sampled from the old strategy. i This represents the reward value calculated using the reward function reward() for the current policy response; the formula for the ratio of the current policy to the old policy is as follows:

[0113]

[0114] The formula for calculating the KL divergence between the current policy model and the reference model is as follows:

[0115]

[0116] Where, π θ Represents the current policy model, π ref This represents the reference model.

[0117] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0118] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A format knowledge reasoning method based on size language model collaboration, characterized in that, The method comprises the following steps: Step S1: generating a basic answer in natural language form on an original reasoning task for any large language model; Step S2: generating a correct reasoning path according to the reasoning question and the basic answer by using a small language model through an iterative method; Step S3: reconstructing the reasoning question, the basic answer, the reasoning steps and the standard answer into a new reasoning dataset with a specific format output and a thinking chain according to specific format instructions; Step S4: training any open-source small language model according to the new reasoning dataset; Step S5: constructing a reward function to score the output of the small language model after the first training, and performing secondary training according to the score to weaken the influence of secondary factors in the original training target on the training result.

2. The format knowledge reasoning method based on size language model collaboration according to claim 1, characterized in that, The any large language model comprises all open-source large language models and closed-source large language models.

3. The format knowledge reasoning method based on size language model collaboration according to claim 1, characterized in that, The generation of the basic answer includes an original data set X = {x1, x2, … x n} and a corresponding standard answer Y = {y1, y2, …, y N}, a large language model The basic answer in the form of natural language is as follows:

4. The format knowledge reasoning method based on size language model collaboration according to claim 1, characterized in that, The step S2 comprises the following steps: Step S2.1: For the original problem x i , the standard answer y i , and the base answer of the large language model , the small language model generates the inference path and the prediction result as follows: In the formula, r i represents the inference path, a i represents the prediction result; Step S2.2: input the prediction result a of the model to the small language model as a reference to perform step S2.3 i If a i is equal to y i , it is considered that the corresponding reasoning path r i is correct and the reasoning path r i is retained; if a i is not equal to y i , the standard answer y i is input to the small language model as a reference to perform step S2.3; Step S2.3: generating a reasoning path and an answer by the small language model for the second time as follows: Step S2.4: Again, judge the correctness of a i , if a i is correct, keep the corresponding reasoning path r i , otherwise, directly delete the corresponding sample.

5. The format knowledge reasoning method based on size language model collaboration according to claim 1, characterized in that, The novel inference dataset The following formula: t i = (y f , r i , y i ) where N' denotes the length of the dataset after the inference path is generated, t i denotes the formatted output coupled with the correct inference path r i and the standard answer y i y f denotes the specified special output format.

6. The format knowledge reasoning method based on size language model collaboration according to claim 1, characterized in that, The first training target is to minimize the cross-entropy loss function; The one-time training includes using the original question x i and the base answer of the large language model as input for the small language model training to format the output t i The training target is to use the cross-entropy loss function, and the training loss function for the small language model in training is as follows: in, Let π represent the first-order training loss function of the small language model. θ t represents the small language model during training. i This indicates that the correct inference path r is coupled. i and the standard answer y o Formatted output, x i This represents the original question. Representing problem x i The basic answer to the corresponding large model.

7. The format knowledge reasoning method based on size language model collaboration according to claim 1, characterized in that, The step of constructing a reward function to score the output of the small language model after the first training comprises the following steps: For a standard output t = (y f , r, y) in the dataset and a model output The reward function is given by:

8. The format knowledge reasoning method based on size language model collaboration according to claim 1, characterized in that, The secondary training adopts a group relative strategy optimization algorithm, and the small language model is trained for the second time by constructing a reward function reward(), and the loss function of the training is as follows: where, denotes the quadratic training loss function of the small language model, G denotes the number of answers sampled from the old policy for each question, and i denotes the ratio of the current policy and the old policy; ε is the hyperparameter of the clipping function, which is used to control the maximum amplitude of each policy update; β is the hyperparameter, which is used to adjust the group difference penalty term in the loss function, denotes the KL divergence between the current policy model and the reference model, A i denotes the group relative advantage estimate of each sample.

9. The format knowledge reasoning method based on size language model collaboration according to claim 8, characterized in that, The group relative advantage estimation formula of each sample is as follows: where r1,..., r G represents a reward value calculated by the reward function reward() for an answer sampled from the old policy, r i represents a reward value calculated by the reward function reward() for an answer from the current policy; The ratio formula of the current strategy and the old strategy is as follows: The KL divergence calculation formula between the current strategy model and the reference model is as follows: wherein, π θ represents the current policy model, π ref represents the reference model. 10.A formatting knowledge reasoning system based on size language model collaboration, characterized in that, The method comprises the following steps: The basic answer generation module: generating a basic answer in natural language form on an original reasoning task for any large language model; The data reconstruction module: generating a correct reasoning path according to the reasoning question and the basic answer by using a small language model through an iterative method; and reconstructing the reasoning question, the basic answer, the reasoning steps and the standard answer into a new reasoning dataset with a specific format output and a thinking chain according to specific format instructions; The first training module: training any open-source small language model according to the new reasoning dataset; The secondary training module: constructing a reward function to score the output of the small language model after the first training, and performing secondary training according to the score to weaken the influence of secondary factors in the original training target on the training result.

Images