Small language model data adaptation method based on expert guidance and simulation fall detection

Abstract

The invention discloses a small language model data adaptation method based on expert guidance and simulation fall detection. The method comprises the steps of obtaining a multi-step reasoning track generated by an expert model; performing adaptability evaluation on each step in the trajectory, and calculating whether the student model can successfully complete reasoning after the step is adopted; according to the change of the adaptability score, detecting an imitation fall point and cutting off an expert path; guiding the student model to autonomously explore subsequent steps from the cut-off point until a correct answer consistent with the expert is obtained; and finally, taking a path formed by the expert prefix and the student suffix as a training sample to finely adjust the student model. By utilizing the method, the dynamic adaptation of the reasoning path to the student model ability can be realized, and the reasoning accuracy and generalization ability of the small model in the resource-constrained environment can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing and artificial intelligence model training, and particularly relates to a small language model data adaptation method based on expert guidance and imitation gap detection. BACKGROUND

[0002] In recent years, large language models (LLM) have shown excellent reasoning ability on complex tasks such as mathematical reasoning and code generation. However, there are still many challenges in transferring such reasoning ability to small language models. In the prior art, the common practice is to collect or generate chain-of-thought data containing step-by-step decomposition, and fine-tune smaller models. However, this static collection of expert reasoning demonstration data is often designed for powerful large models, and the effect is not good when directly used for weak small models.

[0003] The reason is that the pre-training corpus, parameter size and reasoning ability of small models are significantly different from those of large models, resulting in a mismatch between the distribution of training data and the distribution when small models actually reason. When small models mechanically imitate the long-chain reasoning steps of large models, distortion or reasoning failure often occurs, i.e., the model cannot correctly reproduce the expected reasoning process according to the input. This deviation will be amplified in subsequent steps of the reasoning process, seriously affecting the accuracy and robustness of the final answer of the model. In addition, it is costly to build a high-quality set of reasoning examples for each model, including fine logical decomposition, intermediate verification and teaching design, etc. An example template carefully designed for a large model is often difficult to directly apply to another model of different size or architecture, further exacerbating the inconsistency of data distribution.

[0004] For small models deployed on mobile devices, the above problems are more prominent. On the one hand, mobile end models are limited by computing and storage resources and cannot improve performance by simply stacking parameters; on the other hand, real-time applications require small models to complete reasoning independently, and cannot rely on cloud large models every time.

[0005] Therefore, the existing static fine-tuning method cannot fully tap the potential reasoning ability of small models, and a new solution is needed that can adaptively adjust the training process according to the student model's own ability to improve data efficiency and the practicality of the model in a mobile environment while ensuring reasoning accuracy. SUMMARY

[0006] In order to overcome the problems of imitation failure, unstable performance and high data adaptation cost of small language models in reasoning tasks in the prior art, the present application provides a small language model data adaptation method based on expert guidance and imitation gap detection, which can realize adaptive matching of reasoning trajectories to end-side small models, thereby significantly improving the reasoning ability and generalization performance of the small models in resource-constrained environments.

[0007] The method of the present application is suitable for enhancing the reasoning ability of small language models deployed on mobile devices such as mobile phones. Based on the high-quality solution demonstration path provided by the expert model, the method dynamically determines which steps to imitate by evaluating the adaptability of the student model to each reasoning step, and terminates the imitation at the appropriate position and guides the student model to explore itself, finally generates a reasoning path that meets the correctness constraint and fits the student model's ability, which is used for supervised learning and ability consolidation.

[0008] Specifically, a small language model data adaptation method based on expert guidance and imitation gap detection is suitable for small language models deployed on end-side devices, comprising the following steps:

[0009] (1) Expert path acquisition: calling a large language model as a teacher model to generate a structured expert solution path for the target problem, which includes multiple intermediate reasoning steps and the final answer;

[0010] (2) Step-by-step adaptability evaluation: for each reasoning step in the expert solution path, the adaptability of the student model to the step is evaluated by using answer simulation, and the success reasoning probability after introducing the current step is calculated as the adaptability score;

[0011] (3) Imitation gap detection and path truncation: time series analysis is performed on the adaptability score sequence to detect local peaks and identify the imitation gap point where the adaptability score decreases significantly, and the expert path is truncated at the imitation gap point;

[0012] (4) Student model autonomous exploration: after the truncated expert path, the student model generates subsequent reasoning steps autonomously, and completes the solution path under the condition of meeting the same final answer consistency with the expert model;

[0013] (5) Path splicing and training sample generation: the expert path prefix and the student model suffix are spliced into a complete reasoning trajectory as a supervised training sample;

[0014] (6) Sample screening and updating: if the student model cannot generate a correct answer within a set number of times, discard the problem sample and only keep the successfully generated trajectories for training;

[0015] (7) Supervised learning update: using the above filtered supervised training samples, the student model is supervised learning training to improve its reasoning ability and generalization performance in future similar tasks;

[0016] (8) Application: using the trained student model to solve problems.

[0017] The application can automatically generate training data with strong adaptability and consistent goals for small language models with different capabilities, greatly improving their reasoning performance and reducing data construction costs, and is especially suitable for deployment in mobile environments with limited computing resources.

[0018] In step (1), a large language model with strong reasoning ability is called as a teacher model to generate a complete expert solution track for a given reasoning task. The track is represented in a structured form and includes ordered intermediate reasoning steps and the final answer, which is used as a reference template for model learning paradigm. Each reasoning step provides explicit logical operations or intermediate conclusions, which helps to build a verifiable reasoning path and provides high-quality cognitive demonstration signals for the student model.

[0019] In step (2), the student model's adaptability to each expert step is evaluated using a step-by-step simulation method. The specific process of step (2) is as follows:

[0020] For each candidate step s in the expert solution path t , introduce the step based on the prefix s ＜t before it, and then let the student model simulate the remaining reasoning process from this point until the answer is obtained; repeat N times of independent reasoning, and count the frequency of successfully generating the correct answer as the adaptability score f t of the step.

[0021] The adaptability score f t can be regarded as a measure of the "imitation feasibility" of the step for the student model, and the calculation formula is as follows:

[0022]

[0023] In the formula, a (i) is the final answer obtained by the i-th simulation reasoning, and a * is the standard answer of the expert solution track.

[0024] In step (3), the local peak is detected and the imitation gap point is identified at the position where the adaptability score significantly decreases, specifically as follows:

[0025] Detect the local peak f peak of the adaptability score and its position t peak; then, continue to scan the step sequence backward, if the score of a step is lower than the local peak minus a threshold ∈, i.e.

[0026] f < f peak - ∈

[0027] then the step is identified as a mimicry gap point t gap . The system truncates the expert path before this point, only keeping the part that the student model can still imitate, for subsequent splicing and training. This mechanism can effectively avoid exposing the path with too heavy cognitive burden to the student model with limited ability.

[0028] In step (4), after truncating the expert path, stop providing subsequent expert path step prompts to the student model, and instead let the student model explore new problem-solving paths based on the existing path prefix to complete the remaining reasoning;

[0029] The student model attempts multiple path branches, generates a series of reasoning steps in each attempt, and continuously detects whether the problem-solving path can eventually lead to the same standard answer a * as the expert.

[0030] Implement the result consistency constraint for the explored problem-solving path, i.e.

[0031]

[0032] where, if the student model generates a problem-solving path τ adapt , the expert path generated by the teacher model is τ expert , O(τ adapt ) = O(τ expert ) means that the results of the two paths are consistent, at which time C(τ adapt ,τ expert ) is 1, otherwise it is 0; only when the final answer of the problem-solving path explored by the student model is consistent with the expert answer, the path is retained for training.

[0033] In step (5), after the student model successfully completes the exploration and meets the result consistency condition, the system splices the prefix part of the expert path that has been verified to be adaptive with the suffix of the path generated by the student, forming a complete and individually adapted reasoning track. The complete reasoning track after splicing retains the correct paradigm in the expert path, while also reflecting the reasoning ability and preference of the student model itself.

[0034] In step (6), after multiple rounds of exploration attempts, if the student model still cannot successfully build a reasoning path consistent with the expert answer, it is determined that the problem sample is not suitable for the current model and is discarded. This sample is not included in the training set, thereby ensuring the effectiveness and consistency of the training data, and avoiding the introduction of low-quality signals into the model parameter update.

[0035] In step (7), the student model is updated using a standard cross-entropy loss function. The formula is as follows:

[0036]

[0037] By maximizing the probability of correctly generating the next reasoning step at each step in the adaptive trajectory, the student model continuously strengthens its modeling ability for high-quality paths within its existing capabilities, thereby automatically building an efficient and feasible problem-solving path when encountering similar problems in the future, and improving reasoning generalization ability and stability.

[0038] Compared with the prior art, the present application has the following beneficial effects:

[0039] 1. Suitable for end-side model (mobile phone end) deployment. The method of the present application does not depend on a specific task field or model architecture, and can be applied to various tasks requiring multi-step reasoning such as mathematical reasoning, code reasoning, logical question answering, etc., and has good adaptability to small models of different sizes and different pre-training corpora. Especially in mobile device deployment scenarios, the present method can effectively align the training examples with the target model capabilities, endowing lightweight models running on devices such as mobile phones with reasoning capabilities close to large models, and improving the user experience of terminal intelligent applications.

[0040] 2. Improve reasoning performance and generalization ability: Since the student model is prevented from blindly imitating steps it cannot handle during training, the present application effectively reduces the distribution difference and error accumulation in the model reasoning process, significantly improving the accuracy and robustness of small models in reasoning tasks. Experiments show that the model trained using the present method outperforms small models directly fine-tuned from original expert data on various reasoning benchmarks, demonstrating stronger complex reasoning capabilities.

[0041] 3. High data efficiency: The present application generates diversified training trajectories that fit the student's ability through a small number of high-quality expert examples combined with student model autonomous exploration, achieving good results without the need to build large-scale customized data sets. Compared with solutions that rely on massive amounts of labeled data, the present method significantly reduces the training data requirements and is suitable for application in scenarios with limited data. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The flowchart of the method of the present application.

[0043] Figure 2 This is a general framework diagram corresponding to the method of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0045] like Figure 1 As shown, a small language model data adaptation method based on expert guidance and imitation gap detection includes the following steps:

[0046] S01, first obtain the reasoning demonstration step data from the expert model.

[0047] Suppose there is a question Q that requires multi-step reasoning, and its correct answer is a. First, a large teacher model (expert) is used to generate the complete reasoning process for solving the question: for example, the expert trajectory includes steps s1→s2→s3→s4→…, and finally reaches the answer a.

[0048] At step S02, the student model M has only a fraction of the parameters of the expert model, making it difficult to directly imitate the long chain of reasoning described above. Therefore, it is necessary to calculate the adaptability score of each expert model step. Starting from step s1, the expert trajectory is fed to the student model M for imitation. At each step, the student model is evaluated in real time to determine whether it is suitable to continue imitating the current expert step.

[0049] Specifically, when M adopts the expert's step s t Then, let M simulate subsequent reasoning from this step to check whether it can finally deduce the correct answer a. By randomly sampling M's reasoning multiple times, the frequency of M successfully deriving a is counted as the frequency of step s. t The adaptability score f t If f t If the value is higher (for example, higher than a threshold θ), it means that the student model has a greater probability of being competent for the reasoning of this step, then s t Incorporate students’ learning trajectories and allow M to continue to imitate the next steps t+1 If it drops significantly and falls below the threshold θ, it is considered that imitation difficulty occurs here.

[0050] S03. In the present invention, after completing the calculation of the adaptability score of the target model to each reasoning step in the expert path, the system further performs an imitation drop positioning operation to dynamically determine the truncation point of the adaptability path, thereby constructing an adaptive reasoning trajectory that matches the student model capabilities.

[0051] Specifically, the system first simulates each expert step according to the preset number of simulations to obtain a set of adaptive score sequences {f0,f1,…,fT Each score represents the estimated probability of the target model successfully generating the correct answer after introducing the current step. Subsequently, the system detects a local optimum f peak , whose corresponding position is denoted as t peak . This point can be regarded as the stage where the model has the most imitating ability in the expert path. After locating the local peak, the system continues to search backward along the time axis and, based on a set threshold parameter ∈, determines whether there is a step position that satisfies the following condition: f t < f peak - ∈. If there is a minimum t that satisfies the above condition, it is marked as the imitation drop point, denoted as t gap . This position marks the point where the student model's ability to imitate subsequent expert steps significantly decreases, and continuing to imitate may lead to a deviation in the reasoning path or mislead the training.

[0052] S04, Path Truncation: When an imitation drop is detected, the present invention stops providing subsequent step hints in the expert trajectory. Subsequently, the student model takes over the remaining reasoning process and explores a new problem-solving path autonomously from the current pause.

[0053] S05, Self-Exploration Logic of the Student Model: In the autonomous exploration phase, the student model uses its own knowledge and reasoning skills to fill in the gaps in the expert path. To ensure the correct exploration direction, the known correct answer a is used as a constraint: the student model can try different subsequent reasoning steps, but only accepts the path and ends the exploration if it eventually leads to the answer a. If the first self-generated step by the student model fails to lead to the correct answer, the present invention allows the model to backtrack and try other possible reasoning branches, similar to the trial-and-error process of humans when solving problems. When the student model successfully finds a path leading to the correct answer (which includes several self-generated steps), this mixed path of expert prefix and student autonomous exploration suffix is considered the adjusted reasoning trajectory.

[0054] For example, for a mathematical reasoning problem, if the expert example uses advanced techniques to skip multiple steps succinctly, and the student model cannot understand, the student model may choose to derive the answer step by step with more basic intermediate steps. Throughout the exploration process, the present invention verifies whether the student model can complete the remaining reasoning from each new step, similar to adaptive scoring evaluation, to avoid falling into the wrong direction. The student model can continuously generate the next candidate step s t ′ and check "if s t ′ is adopted, can the answer be solved eventually?". Once a s t ′The invention confirms this step and lets the student model continue to reason down, so as to make the subsequent reasoning have higher confidence to reach the correct answer. This iteration continues until the student model pushes out the answer a. During this process, the student model actually learns how to correct itself and perfect the reasoning chain, and thus has a deeper understanding of tasks within its own ability range.

[0055] S06, synthesis of new path and fine-tuning sample update: when the student model successfully derives the correct answer with the adjusted path, it obtains a problem-solving trajectory tailored for the student model.

[0056] S07, discard of unsuitable samples: not all expert paths can be converted into high-quality training signals. Therefore, the invention proposes a sample discarding strategy to ensure the quality of the supervision signal: if the model M cannot search for the answer after multiple autonomous explorations during the splicing path generation process, the sample is discarded, indicating that the sample is not suitable for stimulating the reasoning ability of the model M.

[0057] S08, finally, the invention adds the available adaptive splicing trajectory to the training set to further fine-tune and consolidate the student model. This step is equivalent to incorporating the new ideas obtained by the student model through self-exploration into its knowledge, so that the model can use its own logical path with a higher probability when encountering similar problems next time. Since the invention always ensures that the reasoning output of the student model aligns with the standard answer during training, the model not only learns the correct problem-solving steps, but also avoids learning invalid or deviating content from the correct answer, thereby ensuring the quality of training.

[0058] In Figure 2 , we visualize the general framework process, which is divided into three processes. The first is the small model solution simulation, corresponding to S01 to S04. The second part is the adaptive path exploration, corresponding to S05 to S07. The last is the trajectory learning based on result alignment, corresponding to S08.

[0059] The above-described embodiments detail the technical solutions and beneficial effects of the invention. It should be understood that the above-described embodiments are only specific embodiments of the invention and are not intended to limit the invention. Any modifications, supplements, and equivalent replacements made within the scope of the principles of the invention should be included in the protection scope of the invention.

Claims

1. A small language model data adaptation method based on expert guidance and imitation gap detection, suitable for small language models deployed on end-side devices, characterized by: The following steps are involved: (1) Using a large language model as a teacher model to generate a structured expert solution path for the target problem, which includes multiple intermediate reasoning steps and the final answer; (2) For each reasoning step in the expert's problem-solving path, the adaptability of the student model to that step is evaluated using a solution simulation method, and the probability of successful reasoning after the introduction of the current step is calculated as the adaptability score; (3) Performing temporal analysis on the adaptability score sequence of the steps, detecting local peaks and identifying the imitation drop point at the location where the adaptability score drops significantly, and truncating the expert path at the imitation drop point; (4) After the truncated expert path, the student model autonomously generates subsequent reasoning steps and completes the problem-solving path while satisfying the same final answer consistency condition as the expert model; (5) Concatenate the expert path prefix and the student model suffix into a complete reasoning trajectory as a supervised training sample; (6) If the student model cannot generate the correct answer within the set number of times, the question sample is discarded and only the successfully generated trajectory is retained for training; (7) Using the above-screened supervised training samples, the student model is trained through supervised learning to improve its reasoning ability and generalization performance in similar tasks in the future; (8) Use the trained student model to solve the problem.

2. The small language model data adaptation method based on expert guidance and imitation gap detection according to claim 1 is characterized in that: The specific process of step (2) is: For each candidate step s in the expert's solution path t , preceded by the prefix s ＜t This step is introduced based on the previous step, and then the student model is asked to simulate the remaining reasoning process from this point until the answer is obtained; repeat N independent reasonings, and count the frequency of successful generation of correct answers as the adaptability score f of this step t .

3. The small language model data adaptation method based on expert guidance and imitation gap detection according to claim 2 is characterized in that: Adaptability score f t The calculation formula is: Where a (i) is the final answer obtained by the ith simulation reasoning, a * The standard answer for the expert's problem-solving trajectory.

4. The small language model data adaptation method based on expert guidance and imitation gap detection according to claim 1 is characterized in that: In step (3), local peaks are detected and imitation drop points are identified at locations where the adaptability score drops significantly, specifically: Detect the local peak f of the fitness score peak and its location peak ; Then, continue to scan the step sequence backward. If the score of a step is found to be lower than the local peak minus the threshold ∈, that is: f<f peak -∈ Then this step is identified as imitating the drop point t gap .

5. The small language model data adaptation method based on expert guidance and imitation gap detection according to claim 1 is characterized in that: In step (4), after the truncated expert path, the student model is stopped from being prompted with the subsequent expert path steps. Instead, the student model is allowed to independently explore new problem-solving paths based on the existing path prefix and complete the remaining reasoning. The student model tries multiple path branches, generates a series of reasoning steps in each attempt, and continuously tests whether the solution path can eventually produce the same standard answer as the expert. * .

6. The small language model data adaptation method based on expert guidance and imitation gap detection according to claim 5 is characterized in that: For the explored problem-solving paths, result consistency constraints are implemented, namely: Among them, if the problem-solving path generated by the student model is τ adapt , the expert path generated by the teacher model is τ expert ,O(τ adapt )=O(τ expert ) indicates that the results of the two paths are consistent, then C(τ adapt ,τ expert ) is 1, otherwise it is 0; only when the final answer of the problem-solving path explored by the student model is consistent with the expert answer, the path will be retained for training.

7. The small language model data adaptation method based on expert guidance and imitation gap detection according to claim 1 is characterized in that: In step (5), the complete reasoning trajectory after splicing retains the correct paradigm in the expert path, and also reflects the reasoning ability and preference of the student model itself.

8. The small language model data adaptation method based on expert guidance and imitation gap detection according to claim 1 is characterized in that: In step (7), the standard cross entropy loss function is used to update the parameters of the student model.

Images