Knowledge internalization progressive method and device for small language model reasoning

Abstract

The invention provides a knowledge internalization progressive method and device for small language model reasoning. The method comprises: knowledge generation: utilizing a large language model (LLMs) to generate a reasoning process and symbolic knowledge for specific training data, including learning abstracts and supplementary materials; secondly, knowledge compression and example screening are carried out, and symbol knowledge and training examples are compressed and simplified step by step based on a preset linear attenuation time table so as to optimize knowledge representation; and finally, progressive fine tuning is performed, compressed knowledge and screened examples are combined for fine tuning of the SLMs, so that the SLMs only depend on the problem during reasoning, and the efficiency is improved. And along with fine tuning, knowledge is gradually internalized into model parameters, so that the model can be inferred without additional document retrieval or multi-stage generation. According to the method, symbol knowledge is internalized into a small language model through a progressive fine tuning process, so that the performance of the method on a complex reasoning task is improved.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and large model reasoning, and specifically relates to a knowledge internalization progressive method and device for small language model reasoning. Background Art

[0002] In recent years, large language models (LLMs), such as ChatGPT and Llama3, have demonstrated excellent performance on a variety of complex reasoning tasks. These models are trained with massive amounts of data and are able to capture the deep semantics and patterns in language, thus achieving remarkable achievements in mathematical reasoning, common sense reasoning, and symbolic reasoning. However, large language models are computationally expensive and their requirements for environmental resources limit their large-scale deployment and practical applications. In addition, their complexity and large number of parameters also pose challenges to the interpretability and maintainability of the models. To address these issues, researchers have begun to explore how to migrate the advanced reasoning capabilities of large language models to smaller and more efficient models, namely small language models (SLMs).

[0003] SLMs aim to achieve similar performance with lower computational and data costs. One approach is to compress and transfer the knowledge of LLMs to SLMs through knowledge distillation techniques, which usually involves using the output of LLMs (e.g., the reasoning process of question answering) as training data to fine-tune SLMs.

[0004] Although knowledge distillation technology has achieved the improvement of model performance to a certain extent, existing methods still have some limitations. For example, some methods require additional document retrieval or multi-stage generation during reasoning, which increases the computational burden and delay. In addition, the introduced external knowledge may increase noise and affect model performance. Therefore, how to effectively internalize the knowledge of LLMs into SLMs while reducing computational overhead and improving reasoning efficiency has become a research hotspot in this field.

[0005] The SKIntern method of the present invention is proposed just in this context. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a progressive method and device for knowledge internalization of small language model reasoning, which is used to improve the performance of small language models (SLMs) when performing complex reasoning tasks while reducing the demand for computing resources. The method is called SKIntern, which imitates the learning process of human interns and effectively internalizes symbolic knowledge and a small number of examples into model parameters through progressive fine-tuning. SKIntern is implemented through three main steps: first, knowledge generation, using large language models (LLMs) to generate reasoning processes (Rationale) and symbolic knowledge for specific training data, including learning summaries and supplementary knowledge. Secondly, knowledge compression and example screening, based on a predetermined linear decay schedule, symbolic knowledge and training examples are gradually compressed and streamlined to optimize knowledge representation. According to the predefined linear decay table, the training examples are gradually reduced, for example, a total of 5 steps of decay (100%, 80%, 60%, ..., 0%), the initial examples are the 5 examples closest to the problem, and then one example is reduced at each time step according to the predefined decay compression rate. As for symbolic knowledge, the existing prompt word compression algorithm is used. The present invention uses the currently optimal LLMLingua2 to compress according to the compression rate. Finally, progressive fine-tuning is performed to combine the compressed symbolic knowledge with the screened training examples to fine-tune the SLMs so that it only relies on the problem itself during reasoning, thereby improving efficiency. As fine-tuning proceeds, the symbolic knowledge is gradually internalized into the SLMs model parameters, so that the SLMs model can be reasoned without additional document retrieval or multi-stage generation. After fine-tuning is completed, SLMs can perform effective reasoning independently without increasing additional computing costs.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A gradual method for knowledge internalization for small language model reasoning, the method comprising:

[0009] Step 1: Use a large language model to generate reasoning procedures and symbolic knowledge for specific training data, where the symbolic knowledge includes learning summaries and supplementary knowledge;

[0010] Step 2: Based on a predefined linear decay schedule, the generated symbolic knowledge is gradually compressed to prune the set of training examples selected from the specific training data;

[0011] Step 3: Combine the compressed symbolic knowledge with the reduced set of training examples to progressively fine-tune the small language model, where the progressive fine-tuning takes minimizing the negative log-likelihood as the optimization objective.

[0012] Furthermore, the training example set in step 2 is a most relevant example set selected from specific training data based on relevance score or cosine similarity.

[0013] Furthermore, the compression of symbolic knowledge and the simplification of the training example set in step 2 are performed according to a predefined linear decay schedule, which specifies the rates of symbolic knowledge compression and example simplification.

[0014] Furthermore, the step 3 includes, in a progressive fine-tuning process, the small language model parameters gradually absorb the compressed symbolic knowledge and streamlined training example information to achieve knowledge internalization, wherein the internalization means gradually reducing the number of symbolic knowledge and training examples in the input to zero under the guidance of a predefined linear decay schedule.

[0015] In another aspect, the present invention provides a knowledge internalization progressive device for small language model reasoning, comprising:

[0016] A symbolic knowledge generation unit, which is used to generate reasoning procedures and symbolic knowledge for specific training data using a large language model, where the symbolic knowledge includes learning summaries and supplementary knowledge;

[0017] A compression and refinement element for gradually compressing the generated symbolic knowledge based on a predefined linear decay schedule to refine the set of training examples selected from the specific training data;

[0018] A fine-tuning unit is used to combine the compressed symbolic knowledge with the streamlined set of training examples for progressively fine-tuning the small language model, wherein the progressive fine-tuning takes minimizing the negative log-likelihood as an optimization goal.

[0019] In a third aspect, the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned progressive method of knowledge internalization for small language model reasoning.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned knowledge internalization progressive method for small language model reasoning.

[0021] The beneficial effects of the present invention are:

[0022] The present invention gradually internalizes symbolic knowledge into SLMs through an innovative incremental fine-tuning process, thereby improving the performance of SLMs in complex reasoning tasks without incurring additional computational costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram for comparing knowledge utilization between SKIntern and other typical CoT distillation methods;

[0024] Figure 2 Schematic diagram of the progressive method for knowledge internalization for small language model reasoning of the present invention;

[0025] Figure 3 This is a schematic diagram of FLOPs-accuracy (%) for different model sizes according to an embodiment of the present invention;

[0026] Figure 4 It is an efficiency curve diagram of training data and model size according to an embodiment of the present invention;

[0027] Figure 5 This is a comparison chart of ablation experiments using different methods;

[0028] Figure 6 The impact of model size changes on the performance of different methods;

[0029] Figure 7 The performance impact of different methods of qwen2-7b with different data amounts (100%->1%). DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0031] like Figure 1 As shown in the figure, the knowledge utilization of our proposed SKIntern is compared with other typical CoT distillation methods. (i) Std-CoT: SLMs are fine-tuned to generate reasoning and answers (Q → R + A) for questions. However, due to the limited parameter size of SLMs, they cannot effectively memorize all knowledge and model reasoning capabilities, and are difficult to generalize to out-of-domain (OOD) tasks. (ii) KARD: SLMs are fine-tuned to generate reasoning and answers (Q + K → R + A) based on questions and retrieved symbolic knowledge. Redundant and noisy symbolic knowledge reduces the effectiveness of CoT distillation. (iii) CasCoD: The single CoT learning step is decomposed into two integrated learning steps: reasoning generation (Q → R) and reasoning utilization (Q + R → A). Long input and multi-level generation reduce the reasoning efficiency of CoT distillation (iv) {SKIntern}: Similar to human interns, SLMs gradually absorb and internalize the symbolic knowledge provided by LLMs during the step-by-step fine-tuning process, resulting in efficient reasoning (Q → R + A) and effective knowledge modeling (K is incorporated into parameters).

[0032] like Figure 2 As shown, this is a schematic diagram of the progressive method of knowledge internalization for small language model reasoning of the present invention. First, a large language model (LLM) is selected as a teacher model, which has the ability to handle complex reasoning tasks, and a small language model (SLM) is selected, which will receive knowledge transfer as a student model.

[0033] The method comprises three steps, specifically comprising:

[0034] Step 1: Generate reasoning and symbolic knowledge: Use the teacher model to process a set of training data to generate high-quality reasoning processes (rationales) and symbolic knowledge. Symbolic knowledge can include learning summaries, supplementary knowledge, and any relevant background information.

[0035] Step 2: Compress symbolic knowledge and simplify examples: According to a predetermined linear decay schedule, the generated symbolic knowledge is compressed using an existing prompt compression method. Based on the relevance score, the most relevant examples are selected from the training data as the initial example set, or the examples most relevant to the question are selected as the initial full-volume examples by cosine similarity score, and then the number of examples is reduced by synchronous compression with the symbolic knowledge, that is, a specified number of examples are selected from the initial example set as the fine-tuning example set for this time step, where symbolic knowledge compression is performed step by step according to a linear decay schedule using an existing prompt compression algorithm, and the simplification of the initial example set is also screened based on the selected examples according to the linear decay schedule. This schedule specifies the rate of symbolic knowledge compression and example simplification.

[0036] Step 3, progressive fine-tuning: Fine-tune the small language model (SLMs) by combining the compressed symbolic knowledge and the condensed initial example set, i.e., the fine-tuning example set. The fine-tuning process follows a predefined linear decay schedule, where the fine-tuning adopts the technique of minimizing the negative log-likelihood, and this is used as the optimization target at each time step. The following the predefined linear time decay schedule here means that the fine-tuning process is progressive, such as Figure 2 As shown in the figure, from the beginning of fine-tuning (i.e., step 0), all knowledge and examples are used, and the SLM is fine-tuned to generate rationales and answers by minimizing the negative log-likelihood as the optimization goal. Then step 1 compresses the knowledge and selects examples according to the defined compression rate, and then fine-tuning continues; and so on. It represents that the compression rate of the additional knowledge and examples used in the fine-tuning process follows a linear decay table. During the fine-tuning process, the model parameters gradually absorb symbolic knowledge and fine-tuning example information to achieve knowledge internalization. As training progresses, the symbolic knowledge and the number of fine-tuning examples in the input gradually decrease. Among them, the progressive fine-tuning process generates rationales and answers based on compressed symbolic knowledge and streamlined examples (100%->0%).

[0037] After fine-tuning, SLMs are able to perform efficient inference independently without relying on additional document retrieval or multi-stage generation.

[0038] To illustrate the effectiveness of the method, the method of the present invention was tested on multiple public multi-task datasets using open source SLMs such as TinyLLaMA and LLaMA2-7B. The method was verified, including in-domain (ID) and out-of-domain (OOD) datasets, covering mathematical reasoning, common sense reasoning, symbolic reasoning and other fields: BBH (BIG-Bench Hard), GSM8K, AGIEval, ARC, MetaMath, etc.

[0039] SKIntern significantly improves the complex reasoning capabilities of small language models (SLMs) on multiple benchmark test sets. Specifically, improvements were achieved on the test sets of mathematical reasoning (GSM8k), common sense reasoning (BBH), general reasoning (AGIEval), and factual knowledge (ARC). NesyCD was compared with other methods including Zero-shot, Fine-tuning, CoTdistillation, etc. Experimental results Figure 3-Figure 4 It is shown that SKIntern achieves better performance on multiple datasets with less computation (FLOPs). FLOPs are calculated based on processing all examples from the same task during inference. Figure 4 The results in report the mean and standard deviation of the accuracy of the CoT distillation methods for three different runs. The relative FLOPs cost is calculated relative to TinyLLaMA with Zero-shot, and the present invention calculates the FLOPs required for each method in the bbh test.

[0040] In the ID task, the SKIntern method improved the performance of the LLaMA2-7B model by 5.2% and the TinyLLaMA-1.1B model by 4.9%. In the OOD task, the performance of the LLaMA2-7B model improved by 6.1%, and the TinyLLaMA-1.1B model improved by 5.8%. The SKIntern method reduced the FLOPs of the LLaMA2-7B model by 3.9 times and the TinyLLaMA-1.1B model by 4.2 times. Compared with the baseline method, SKIntern significantly reduced the consumption of computing resources while maintaining performance. The efficiency of SKIntern in terms of training data and model size was analyzed, and experiments were conducted using Qwen2-0.5B, 1.5B, and 7B. It can be found that as the amount of training data increases, SKIntern always outperforms other baselines, and the improvement rate also increases. This shows that SKIntern performs best on different amounts of data and achieves excellent distillation of reasoning capabilities. Even with limited datasets, SKIntern outperforms other methods, demonstrating robustness and sample efficiency. As Figure 6 As shown, the model size is then considered, and the results highlight that SKIntern provides substantial advantages for models of different sizes, consistently outperforming other methods. These results emphasize the practical benefits of SKIntern in resource-limited settings, as it reduces the computational requirements of SLMs while providing performance comparable to or exceeding that of larger models. Notably, SKIntern enables Qwen2-7B to outperform the teacher model GPT-3.5 Turbo in both ID and OOD tasks, despite having fewer parameters.

[0041] Ablation experiment: Figure 5 and Figure 7 As shown, the present invention uses LLaMA2-7B to perform ablation experiments to evaluate the contribution of each part in the SKIntern model. Using LLaMA2-7B for ablation studies, three variants were created: (1) w / ok^m, removing the learning summary during fine-tuning; (2) w / ok^p, not including supplementary knowledge; (3) w / oe, omitting example pruning. Removing any of these components leads to a drop in performance, which highlights the key role of internalized knowledge and examples in enhancing the complex reasoning ability of SLM during gradual fine-tuning.

[0042] In addition, the effectiveness of the generated symbolic knowledge is studied, and adding the learned summary km and supplementary knowledge kp to the original zero-shot, zero-shot-cot, and few-shot-cot can significantly improve the performance. Notably, this improvement occurs without fine-tuning, demonstrating the practicality and generalization of symbolic knowledge in enhancing the model's reasoning ability.

[0043] In another aspect, the present invention provides a knowledge internalization progressive device for small language model reasoning, comprising:

[0044] A symbolic knowledge generation unit, which is used to generate reasoning procedures and symbolic knowledge for specific training data using a large language model, where the symbolic knowledge includes learning summaries and supplementary knowledge;

[0045] A compression and refinement element for gradually compressing the generated symbolic knowledge based on a predefined linear decay schedule to refine the set of training examples selected from the specific training data;

[0046] A fine-tuning unit is used to combine the compressed symbolic knowledge with the streamlined set of training examples for progressively fine-tuning the small language model, wherein the progressive fine-tuning takes minimizing the negative log-likelihood as an optimization goal.

[0047] In a third aspect, the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned progressive method of knowledge internalization for small language model reasoning.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned knowledge internalization progressive method for small language model reasoning.

[0049] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A knowledge internalization incremental method for small language model reasoning, characterized by: The method includes: Step 1: Use a large language model to generate reasoning procedures and symbolic knowledge for specific training data, where the symbolic knowledge includes learning summaries and supplementary knowledge; Step 2: Based on a predefined linear decay schedule, the generated symbolic knowledge is gradually compressed to prune the set of training examples selected from the specific training data; Step 3: Combine the compressed symbolic knowledge with the reduced set of training examples to progressively fine-tune the small language model, where the progressive fine-tuning takes minimizing the negative log-likelihood as the optimization objective.

2. The incremental method for knowledge internalization of small language model reasoning according to claim 1, characterized in that: The training example set in step 2 is the most relevant example set selected from the specific training data based on the relevance score or cosine similarity.

3. The incremental method for knowledge internalization of small language model reasoning according to claim 1, characterized in that: The compression of symbolic knowledge and the simplification of the training example set in step 2 are both performed according to a predefined linear decay schedule, which specifies the rates of symbolic knowledge compression and example simplification.

4. The incremental method for knowledge internalization of small language model reasoning according to claim 1, characterized in that: The step 3 includes, in a progressive fine-tuning process, the small language model parameters gradually absorb the compressed symbolic knowledge and streamlined training example information to achieve knowledge internalization, wherein the internalization means gradually reducing the number of symbolic knowledge and training examples in the input to zero under the guidance of a predefined linear decay schedule.

5. A knowledge internalization progressive device for small language model reasoning, characterized in that: include: A symbolic knowledge generation unit, which is used to generate reasoning procedures and symbolic knowledge for specific training data using a large language model, where the symbolic knowledge includes learning summaries and supplementary knowledge; A compression and refinement element for gradually compressing the generated symbolic knowledge based on a predefined linear decay schedule to refine the set of training examples selected from the specific training data; A fine-tuning unit is used to combine the compressed symbolic knowledge with the streamlined set of training examples for progressively fine-tuning the small language model, wherein the progressive fine-tuning takes minimizing the negative log-likelihood as an optimization goal.

6. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; Wherein, when one or more programs are executed by the one or more processors, the one or more processors implement the progressive method of knowledge internalization for small language model reasoning as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement the knowledge internalization progressive method for small language model reasoning as described in any one of claims 1-4.

Images