Small language model reasoning optimization method, device, equipment and medium
By generating training datasets and preference data, and optimizing small language models using the DPO algorithm, the problem of efficient training for logic and mathematical reasoning tasks in resource-constrained environments is solved, achieving high accuracy and low-cost deployment of small models on specific tasks.
Patent Information
- Application Number
- CN202511720618.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-13
AI Technical Summary
Existing technologies struggle to efficiently train small language models in resource-constrained environments, enabling them to exhibit high accuracy and reliability in logic and mathematical reasoning tasks at a low cost.
By defining the target inference task and callable functions, a training dataset containing candidate inference chains is generated. Comparative data is automatically generated using a large language model. Preference data is obtained by combining manual evaluation and automated verification. The Direct Preference Optimization (DPO) algorithm is used to train a small language model through reinforcement learning, thereby optimizing its ability to generate inference chains that conform to preferences.
It significantly reduces the computational resource consumption during model training and inference, achieves a logically rigorous and complete inference process, improves the accuracy and reliability of small language models on specific tasks, and achieves task performance comparable to or even better than large models.
Smart Images
Figure CN121525875A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, in particular to a small language model reasoning optimization method, device, equipment and medium. BACKGROUND
[0002] In recent years, large language models have made significant progress in natural language processing, demonstrating strong general task processing capabilities. These models, through pre-training on massive data, can understand and generate complex text, and to some extent, complete logical reasoning and mathematical calculation tasks. However, when such large models are directly applied to specific scenarios that require precise logical deduction and function calls (such as mathematical problem solving or information system operation and maintenance), their performance is still not satisfactory. The model may produce factually incorrect, logically confused hallucination outputs, or be unable to reliably perform structured, multi-step reasoning processes, limiting its practical application value in critical tasks.
[0003] On the other hand, the huge parameter quantity of large language models results in extremely high training and deployment costs, making it difficult to be widely deployed in edge environments with limited computing resources or online services that require high concurrent response. Although techniques such as knowledge distillation attempt to transfer the capabilities of large models to small models, such methods usually focus on general language understanding capabilities, and for complex reasoning tasks that require precise calling of external tools and APIs and follow strict logical chains, their optimization effect is limited. Small models still have a significant gap in performance compared to large models in such tasks.
[0004] In the prior art, reinforcement learning optimization methods based on human feedback are used to align model behavior with human preferences, but they usually rely on training a complex reward model as a proxy, which is cumbersome and has poor stability. Therefore, there is an urgent need in the art for a technical solution that can efficiently and cost-effectively train small language models, allowing them to be specifically optimized for logical and mathematical reasoning tasks, while maintaining resource efficiency and achieving task execution accuracy and reliability comparable to or even better than large models. SUMMARY
[0005] The technical problem to be solved by the present application is: how to use a resource-efficient training method to enable small language models to have high accuracy and reliable function calling capability in specific logical and mathematical reasoning tasks.
[0006] To solve the above technical problems, the technical solution adopted by the present application is: a small language model reasoning optimization method, comprising the following steps: determining at least one target reasoning task, and designing at least one callable function corresponding to the target reasoning task; Based on the target inference task and the callable function, a training dataset containing multiple candidate inference chains is generated, wherein each candidate inference chain consists of alternating model inference steps and function call steps; Obtain preference data for different candidate inference chains in the training dataset. The preference data is used to identify the superiority or inferiority relationship between different candidate inference chains under the same task input. Using the training dataset and the preference data as training samples, the Direct Preference Optimization (DPO) algorithm is used to train a pre-selected small language model through reinforcement learning, so as to optimize the ability of the small language model to generate inference chains that conform to the preferences. The target reasoning task is processed using the trained small language model.
[0007] Furthermore, the step of generating a training dataset containing multiple candidate inference chains based on the target inference task and the callable function specifically includes: Using a large language model, based on the target reasoning task and the callable function, comparative data containing correct and incorrect reasoning chains is automatically generated.
[0008] Furthermore, the erroneous reasoning chain includes at least one of the following: Incomplete reasoning chains or reasoning chains with incorrect function call parameters or results.
[0009] Furthermore, obtaining preference data for different candidate inference chains in the training dataset specifically includes: By using manual evaluation or automated verification, different candidate inference chains for the same task input are evaluated for their merits and demerits, and preference data pairs identifying winning and losing inference chains are generated.
[0010] Furthermore, the automated verification method specifically includes: In an isolated sandbox environment, execute the function call steps in the candidate inference chain; Collect system status indicators after execution; The system status indicators are evaluated based on predefined verification rules, and preference data that identifies the success or failure of the inference chain is automatically generated.
[0011] Furthermore, executing the candidate inference chain in an isolated sandbox environment specifically includes: Create at least one clone instance that matches the state of the target server; Execute the function call operation in the candidate inference chain on the clone instance; The cloned instance is destroyed after verification is complete.
[0012] Furthermore, the step of using the training dataset and the preference data as training samples, and employing the Direct Preference Optimization (DPO) algorithm to train a pre-selected small language model through reinforcement learning specifically includes: The preference data is constructed into data pairs, which are in the form of: hints, winning reasoning chains, and losing reasoning chains. Using the data pairs, the strategy of the small language model is directly optimized through the DPO loss function, making its output probability distribution more inclined towards the winning inference chain rather than the failing inference chain.
[0013] The present invention also provides a small language model inference optimization device, comprising: The task and function definition module is used to determine at least one target reasoning task and design at least one callable function corresponding to the target reasoning task. The dataset generation module is used to generate a training dataset containing multiple candidate inference chains based on the target inference task and the callable function, wherein each candidate inference chain consists of alternating model inference steps and function call steps; The preference data acquisition module is used to acquire preference data for different candidate inference chains in the training dataset. The preference data is used to identify the superiority or inferiority relationship between different candidate inference chains under the same task input. The model training module is used to train a pre-selected small language model using the training dataset and the preference data as training samples and employing the Direct Preference Optimization (DPO) algorithm to optimize the ability of the small language model to generate inference chains that conform to the preferences. A task processing module is used to process the target reasoning task using the trained small language model.
[0014] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the small language model inference optimization method as described above.
[0015] The present invention also provides a storage medium storing a computer program that, when executed by a processor, can implement the small language model inference optimization method described above.
[0016] The beneficial effects of this invention are as follows: by focusing on small models and utilizing the efficient DPO training paradigm, the computational resource consumption during model training and inference is significantly reduced, enabling feasible deployment in resource-constrained environments; by directly transforming the preference judgment of the inference chain into the optimization objective, the model is effectively guided to generate a logically rigorous and complete inference process, greatly improving its accuracy and reliability in specific logical and mathematical inference tasks; and the small language model possesses both efficient function call capabilities and complex multi-step inference capabilities, achieving task execution performance comparable to or even better than that of large models while maintaining low resource costs. Attached Figure Description
[0017] The specific structure of the present invention will now be described in detail with reference to the accompanying drawings.
[0018] Figure 1 This is a flowchart of a small language model inference optimization method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the automated verification process according to an embodiment of the present invention. Figure 3 This is a flowchart of function calls in the candidate inference chain of this invention. Figure 4 This is a flowchart illustrating the model training process according to an embodiment of the present invention. Figure 5 This is a block diagram of a small language model inference optimization device according to an embodiment of the present invention; Figure 6 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0023] like Figure 1 As shown, an embodiment of the present invention is: a method for optimizing inference in a small language model, including steps S1-S5, as detailed below: S1. Determine at least one target reasoning task, and design at least one callable function corresponding to the target reasoning task.
[0024] In this embodiment, the target reasoning task can be flexibly set according to actual application needs, covering various scenarios from knowledge-based question answering to mathematical calculations and even system maintenance. For example, the task could be verifying a film knowledge question, such as whether Tom Hanks starred in the movie Cast Away; it could be a simple arithmetic problem, such as calculating 12 plus 7; or it could be an IT maintenance task, such as diagnosing and repairing a cloud server with excessive CPU usage. For these different tasks, corresponding callable functions need to be designed to assist in reasoning. For film knowledge-based question answering, functions such as Actor, Movie, and ActsIn can be designed to query and verify information; for arithmetic problems, the Add function can be designed to perform calculations; and for maintenance tasks, system APIs such as ListProcesses, GetServiceByProcess, and RestartService can be designed to obtain status and perform operations. This method of tailoring functions to specific tasks provides small language models with the structured ability to interact with external knowledge or systems, forming the foundation for building an effective reasoning chain.
[0025] S2. Based on the target inference task and the callable function, generate a training dataset containing multiple candidate inference chains, wherein each candidate inference chain consists of alternating model inference steps and function call steps.
[0026] In one specific embodiment, step S2 specifically includes: Using a large language model, based on the target reasoning task and the callable function, comparative data containing correct and incorrect reasoning chains is automatically generated.
[0027] The erroneous inference chain includes at least one of the following: Incomplete reasoning chains or reasoning chains with incorrect function call parameters or results.
[0028] In this embodiment, the powerful generative capabilities of a large-scale language model are leveraged to simulate reasoning processes at different levels, thereby constructing training data in batches. For example, in the task of verifying an actor's participation in a movie, a correct reasoning chain will logically and rigorously call the Actor, Movie, and ActsIn functions sequentially, correctly interpreting the results of each step. An incorrect reasoning chain might manifest as incomplete reasoning logic, such as skipping the step of verifying the actor's or movie's identity and directly calling the ActsIn function; or it might manifest as incorrect function calls, such as calling Add(12, 5) in an addition task and obtaining the incorrect result 17. By carefully designing these positive and negative examples, a challenging comparative dataset can be constructed, forcing subsequent smaller models to learn to distinguish between rigorous and hasty reasoning, thus providing crucial learning material for deep optimization.
[0029] S3. Obtain preference data for different candidate inference chains in the training dataset. The preference data is used to identify the superiority or inferiority relationship between different candidate inference chains under the same task input.
[0030] In one specific embodiment, step S3 specifically includes: By using manual evaluation or automated verification, different candidate inference chains for the same task input are evaluated for their merits and demerits, and preference data pairs identifying winning and losing inference chains are generated.
[0031] In this embodiment, two complementary methods are employed to acquire high-quality preference data. Human evaluation relies on the knowledge of domain experts who can determine the rigor and completeness of the inference chain's logic, which is crucial for knowledge-based question-answering tasks. For tasks where results can be objectively measured, automated verification is used, particularly suitable for scenarios such as mathematical calculations or system maintenance. Automated verification generates a large amount of preference data efficiently and fairly by realistically executing function calls in the inference chain within an isolated sandbox environment and objectively evaluating the execution results based on predefined rules. This dual-track approach ensures the reliability and scale of the preference data, providing clear optimization signals for model training.
[0032] like Figure 2 As shown, in a specific embodiment, step S3, the automated verification method specifically includes: S31. In an isolated sandbox environment, execute the function call steps in the candidate inference chain; S32. Collect system status indicators after execution; S33. Evaluate the system status indicators based on predefined verification rules, and automatically generate preference data that identifies whether the inference chain is successful or failed.
[0033] In this embodiment, the core of automated verification lies in creating a secure execution environment. The system faithfully executes every function call in the inference chain, such as actually calling the `RestartService` function to restart a service or calling the `Add` function to perform a calculation. After execution, the system collects key result indicators, such as whether the service has returned to normal and whether the calculation result is correct. Then, based on pre-set rules, such as whether the CPU utilization rate drops below a threshold within a specified time or whether the addition result equals 19, the system automatically determines the success or failure of the inference chain. This process completely eliminates subjective interference, making the generated preference data highly objective and consistent, which is very suitable for driving the automated learning of models.
[0034] like Figure 3 As shown, in a specific embodiment, step S31 specifically includes: S311. Create at least one clone instance that matches the state of the target server; S312. Execute the function call operation in the candidate inference chain on the cloned instance; S313. Destroy the cloned instance after verification is completed.
[0035] In this embodiment, a sandbox environment is introduced to safely execute operations that may affect the system state during automated verification. Specifically, multiple instances completely identical to the target server are cloned in real time via a cloud platform interface. These cloned instances constitute a highly realistic testbed isolated from the production environment, ensuring that each candidate inference chain is tested under fair and consistent initial conditions. Once the verification process is complete, these temporarily created cloned instances are immediately destroyed. This design ensures both the accuracy and security of the evaluation while demonstrating refined management and cost control of cloud computing resources.
[0036] S4. Using the training dataset and the preference data as training samples, the Direct Preference Optimization (DPO) algorithm is used to perform reinforcement learning training on a pre-selected small language model to optimize the ability of the small language model to generate inference chains that conform to the preferences.
[0037] like Figure 4 As shown, in a specific embodiment, step S4 specifically includes: S41. Construct the preference data into data pairs, the data pairs being in the form of: (hint, winning inference chain, failing inference chain). S42. Using the data pair, directly optimize the strategy of the small language model through the DPO loss function, so that its output probability distribution is more inclined to the winning inference chain than the failing inference chain.
[0038] In this embodiment, Direct Preference Optimization (DPO) is one of the core innovations of this solution. During training, preference data is organized into distinct data pairs. For example, a prompt might be a server CPU high-intensity alert; a winning chain represents the refined approach of diagnosing the process and then restarting the specific service; and a losing chain represents the coarse approach of directly restarting the entire server. The DPO algorithm uses a clever loss function to directly compare and learn between these winning and losing chains, guiding the internal parameters of the small language model to adjust so that when faced with the same prompt in the future, the probability of generating a winning chain is far greater than the probability of generating a losing chain. This method eliminates the cumbersome steps of traditional reinforcement learning, which requires training a complex and unstable reward model, making the training process simpler, more stable, and more efficient. It allows the small model to more directly understand the high-quality reasoning patterns expected by human experts or the system.
[0039] The small-scale language model described in this invention is in contrast to large-scale language models with hundreds of billions or even trillions of parameters. It refers to language models with a typical number of billions or less of parameters, designed to be deployed in resource-constrained environments (such as edge devices, personal terminals, or scenarios requiring high-concurrency services) with lower computational costs and faster inference speeds. Its core feature is resource efficiency, rather than pursuing ultimate general-purpose capabilities.
[0040] In this embodiment, the selected small language model can be an open-source model with 7B or 8B parameters, such as Qwen-7B and Llama-2-7B. Models of this size are chosen because they maintain a certain level of inference capability while having significantly lower training and deployment costs than large models, perfectly aligning with the resource-efficient objective of this invention. Of course, those skilled in the art will understand that with technological advancements, other models with similar parameter sizes and resource efficiency characteristics are also applicable to this invention.
[0041] S5. The target reasoning task is processed using the trained small language model.
[0042] In this embodiment, the trained small language model becomes a highly efficient inference engine dedicated to a specific inference task. When a user poses a new question, such as asking for information about an actor or reporting a system malfunction, the model can automatically run, generate a logically clear and complete inference chain, accurately call relevant functions to obtain information or perform operations, and ultimately provide a reliable answer or solution. Because the model itself is small, it consumes very few computational resources during inference and has a fast response speed, making it easy to deploy on mobile phones, edge devices, or cloud services requiring high concurrency. The ultimate goal of this invention is to enable these lightweight models to exhibit accuracy and reliability comparable to or even surpassing that of large models on specific tasks, while maintaining their inherent low cost and low energy consumption advantages, providing new possibilities for the widespread adoption and application of artificial intelligence.
[0043] like Figure 5 As shown, this embodiment of the invention also provides a small language model inference optimization device, comprising: The task and function definition module 10 is used to determine at least one target reasoning task and design at least one callable function corresponding to the target reasoning task.
[0044] The target reasoning task is a fault diagnosis and repair task in information technology operation and maintenance, and the callable function includes a system application programming interface (API) for obtaining system status information or performing repair operations.
[0045] The dataset generation module 20 is used to generate a training dataset containing multiple candidate inference chains based on the target inference task and the callable function, wherein each candidate inference chain consists of alternating model inference steps and function call steps.
[0046] The preference data acquisition module 30 is used to acquire preference data for different candidate inference chains in the training dataset. The preference data is used to identify the superiority or inferiority relationship between different candidate inference chains under the same task input.
[0047] The model training module 40 is used to train a pre-selected small language model using the training dataset and the preference data as training samples and employing the Direct Preference Optimization (DPO) algorithm to optimize the ability of the small language model to generate inference chains that conform to the preferences.
[0048] Task processing module 50 is used to process the target reasoning task using the trained small language model.
[0049] In one specific embodiment, the dataset generation module 20 is specifically used for: Using a large language model, based on the target reasoning task and the callable function, comparative data containing correct and incorrect reasoning chains is automatically generated.
[0050] In one specific embodiment, the preference data acquisition module 30 is specifically used for: By using manual evaluation or automated verification, different candidate inference chains for the same task input are evaluated for their merits and demerits, and preference data pairs identifying winning and losing inference chains are generated.
[0051] In one specific embodiment, the automated verification method in the preference data acquisition module 30 specifically includes: In an isolated sandbox environment, execute the function call steps in the candidate inference chain; Collect system status indicators after execution; The system status indicators are evaluated based on predefined verification rules, and preference data that identifies the success or failure of the inference chain is automatically generated.
[0052] In one specific embodiment, executing the candidate inference chain in an isolated sandbox environment specifically includes: Create at least one clone instance that matches the state of the target server; Execute the function call operations in the candidate inference chain on the clone instance; The cloned instance is destroyed after verification is complete.
[0053] In one specific embodiment, the model training module 40 is specifically used for: The preference data is constructed into data pairs, which are in the form of: hints, winning reasoning chains, and losing reasoning chains. Using the data pairs, the strategy of the small language model is directly optimized through the DPO loss function, making its output probability distribution more inclined towards the winning inference chain rather than the failing inference chain.
[0054] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned small language model inference optimization device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.
[0055] The aforementioned small language model inference optimization device can be implemented as a computer program, which can, for example, Figure 6 It runs on the computer device shown.
[0056] Please see Figure 6 , Figure 6This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.
[0057] See Figure 6 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.
[0058] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a small language model inference optimization method.
[0059] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.
[0060] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a small language model inference optimization method.
[0061] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0062] The processor 502 is used to run the computer program 5032 stored in the memory to implement the small language model inference optimization method described above.
[0063] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0064] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.
[0065] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the small language model inference optimization method described above.
[0066] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0067] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0068] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0069] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.
[0071] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A small language model inference optimization method, characterized in that, The method comprises the following steps: determining at least one target reasoning task and designing at least one callable function corresponding to the target reasoning task; generating a training data set containing multiple candidate reasoning chains based on the target reasoning task and the callable function, wherein each candidate reasoning chain is alternately composed of model reasoning steps and function call steps; obtaining preference data for different candidate reasoning chains in the training data set, the preference data being used to identify the superior-inferior relationship between different candidate reasoning chains under the same task input; using the training data set and the preference data as training samples, performing reinforcement learning training on a pre-selected small language model by using a direct preference optimization (DPO) algorithm to optimize the ability of the small language model to generate reasoning chains that meet the preferences; processing the target reasoning task using the trained small language model.
2. The method of claim 1, wherein, The generating of the training data set containing multiple candidate reasoning chains based on the target reasoning task and the callable function specifically comprises: using a large language model to automatically generate comparison data containing correct reasoning chains and incorrect reasoning chains based on the target reasoning task and the callable function.
3. The method of claim 1, wherein, The incorrect reasoning chains include at least one of the following: a reasoning chain with incomplete reasoning logic or a reasoning chain with incorrect function call parameters or results.
4. The method of claim 1, wherein, The obtaining of the preference data for different candidate reasoning chains in the training data set specifically comprises: judging the advantages and disadvantages of different candidate reasoning chains for the same task input by manual evaluation or automated verification, and generating a pair of preference data identifying the winning reasoning chain and the losing reasoning chain.
5. The method of claim 4, wherein, The automated verification specifically comprises: executing the function call steps in the candidate reasoning chain in an isolated sandbox environment; collecting system state indicators after execution; evaluating the system state indicators based on predefined verification rules to automatically generate preference data identifying the success or failure of the reasoning chain.
6. The method of claim 5, wherein, The execution of the candidate reasoning chain in the isolated sandbox environment specifically comprises: creating at least one cloned instance consistent with the state of the target server; executing the function call operations in the candidate reasoning chain on the cloned instance; destroying the cloned instance after verification is completed.
7. The method of claim 1, wherein, The reinforcement learning training of the pre-selected small language model by using the training data set and the preference data as training samples and using the DPO algorithm specifically comprises: constructing the preference data into data pairs in the form of: prompt, winning reasoning chain, losing reasoning chain; using the data pairs to directly optimize the strategy of the small language model by using a DPO loss function, so that the output probability distribution of the small language model is more inclined to the winning reasoning chain than the losing reasoning chain.
8. A small language model inference optimization apparatus, characterized by, The method comprises: a task and function definition module for determining at least one target reasoning task and designing at least one callable function corresponding to the target reasoning task; a data set generation module for generating a training data set containing multiple candidate reasoning chains based on the target reasoning task and the callable function, wherein each candidate reasoning chain is alternately composed of model reasoning steps and function call steps; The preference data obtaining module is configured to obtain preference data for different candidate reasoning chains in the training data set, the preference data being used to identify the superior-inferior relationship between different candidate reasoning chains under the same task input; The model training module is configured to use the training data set and the preference data as training samples, and use a direct preference optimization (DPO) algorithm to perform reinforcement learning training on a pre-selected small language model, so as to optimize the ability of the small language model to generate reasoning chains in line with the preferences. The task processing module is configured to use the trained small language model to process the target reasoning task.
9. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the small language model reasoning optimization method according to any one of claims 1 to 7 when executing the computer program.
10. A storage medium, characterized by The storage medium stores a computer program, and the computer program can implement the small language model reasoning optimization method according to any one of claims 1 to 7 when executed by a processor.