A numerical reasoning large model fine-tuning method, system, device and storage medium

Abstract

The application provides a numerical reasoning large model fine-tuning method, system, device and storage medium, the method comprises the following steps: constructing a numerical reasoning data set, the numerical reasoning data set comprises input text and reference answers; introducing a multi-dimensional reward mechanism in the GRPO framework, including: fusing numerical consistency reward, unit rationality reward, formula explicitness reward and illusion penalty term, and generating a total reward function by fusing the multi-dimensional reward mechanism; based on the total reward function, an improved GRPO loss function is constructed; the improved GRPO loss function is used for fine-tuning training of a large language model, and finally a fine-tuned large language model is output. The application is particularly suitable for enhancing the memory ability and numerical reasoning ability of the large language model in the numerical knowledge field, and can reduce the tendency of the large language model to produce numerical illusions.

Description

Technical Field

[0001] The present application relates to the technical field of large model fine-tuning, and specifically to a method, system, device and storage medium for fine-tuning a large numerical reasoning model. Background Art

[0002] Large language models (such as the GPT series) can generate fluent and coherent text, and have outstanding performance in multiple tasks such as text generation, question-answering systems, and machine translation, providing strong support for solving various language-related problems.

[0003] However, existing large language models still face many challenges and limitations when dealing with tasks involving numerical knowledge and numerical reasoning, as follows:

[0004] Numerical hallucination is a common problem: Models often generate erroneous, fictitious, or inconsistent numbers. For example, when processing financial data, scientific research data, and other scenarios that require precise numerical values, models may fabricate non-existent values ​​or incorrectly modify or extrapolate existing values. This can lead to output results that are seriously inconsistent with reality, significantly impacting the model's credibility and reliability.

[0005] Weak numerical reasoning: Models often struggle to accurately complete complex numerical reasoning tasks, such as basic addition, subtraction, multiplication, and division, quantity comparisons, and unit conversions. They may fail to correctly understand the relationships and logic between numerical values ​​and the corresponding operational rules, resulting in erroneous reasoning. This limits their application in specialized fields requiring high numerical reasoning, such as engineering calculations and financial analysis.

[0006] Poor numerical memory: The model also performs poorly in scenarios where accurate memorization and reproduction of factual numbers is required. It may forget or confuse previously mentioned numerical information, resulting in inconsistencies in the subsequent numerical descriptions when generating content, undermining the coherence and accuracy of the text.

[0007] To address these issues and improve the performance of large language models in numerical tasks, researchers have explored various approaches in recent years, including guided fine-tuning (SFT), reinforcement learning (RLHF / GRPO, Reinforcement Learning with Human Feedback, and Guided Reinforcement Preference Optimization), and other optimization techniques. While these approaches can enhance model performance to a certain extent, they lack effective, targeted reinforcement mechanisms for numerical scenarios. These approaches are unable to fundamentally address issues such as numerical illusions, weak numerical reasoning, and poor numerical memory, making it difficult to meet the stringent requirements for high model accuracy and reliability in numerically sensitive domains.

[0008] In view of this, the present invention proposes a method, system, device and storage medium for fine-tuning a large numerical reasoning model, which can effectively improve the memory and numerical reasoning capabilities of a large language model in the field of numerical knowledge, while reducing the tendency of numerical hallucinations. Summary of the Invention

[0009] In order to solve the problems of numerical hallucination, weak numerical reasoning and poor numerical memory in the prior art, the present invention provides a method, system, device and storage medium for fine-tuning a large numerical reasoning model to solve the above technical defects.

[0010] In a first aspect, the present invention proposes a method for fine-tuning a large numerical reasoning model, comprising the following steps:

[0011] S1. Construct a numerical reasoning dataset, which includes input text and reference answers;

[0012] S2. Introduce a multi-dimensional reward mechanism into the GRPO framework, specifically by performing the following operations:

[0013] S21. Calculate the numerical consistency reward based on the error between the predicted value of the large language model output and the true value of the reference answer;

[0014] S22. Calculate the unit rationality reward based on the consistency between the generated units of the output content of the large language model and the target units of the input text;

[0015] S23. Detect whether the output of the large language model contains an operation expression, and perform an explicit reward calculation for the expression;

[0016] S24. Detect whether the output of the large language model contains fictitious values ​​or logically contradictory behavior, and perform hallucination penalty calculation;

[0017] S3. Integrate the numerical consistency reward, unit rationality reward, formula explicit reward, and hallucination penalty to generate a total reward function; and construct an improved GRPO loss function based on the total reward function;

[0018] S4. Use the improved GRPO loss function to fine-tune the large language model and finally output the fine-tuned large language model.

[0019] Preferably, in step S21, a numerical consistency reward is calculated based on the error between the predicted value of the output content of the large language model and the true value of the reference answer. The calculation expression of the numerical consistency reward is:

[0020]

[0021] Where, Represents the numerical consistency reward; represents the reward coefficient; represents the smoothing parameter; represents the predicted value of the large language model, Indicates the true value of the reference answer.

[0022] Preferably, in step S22, a unit rationality reward calculation is performed based on the consistency between the generation unit of the output content of the large language model and the target unit of the input text, which specifically includes the following sub-steps:

[0023] S221, extracting information about the generation unit of the output content of the large language model and the target unit of the input text respectively through a regular function;

[0024] S222: Based on the extracted information of the generating unit and the target unit, determine the unit matching between the generating unit and the target unit, the explicitness of the unit reasoning chain, and whether there is a unit conflict, and obtain a determination result;

[0025] S223. Generate a unit rationality reward value based on the judgment result obtained in step S222. The calculation expression of the unit rationality reward value is:

[0026]

[0027] Where, Indicates unit rationality reward; represents the weight coefficient; Indicates the unit matching value. If the unit generated by the large language model is consistent or equivalent to the target unit, the unit matching value is 1, otherwise it is 0; Indicates the explicit value of the unit inference chain. If there are traces of unit conversion reasoning in the output content of the large language model, the explicit value of the unit inference chain is 1, otherwise it is 0; Indicates the unit conflict value. If the unit generated by the large language model conflicts with the target unit, the unit conflict value is 0, otherwise it is 1.

[0028] Preferably, in step S23, it is detected whether the output content of the large language model contains an operation expression, and the calculation of the explicit expression reward is performed, which specifically includes the following sub-steps:

[0029] S231: Detect whether the output content of the large language model contains an operation expression. If it does, execute the operation of step S232;

[0030] S232: determining the rationality of the grammatical structure of the operation expression and determining the correlation between the operation expression and the numerical value in the input text, and obtaining a determination result;

[0031] S233: Generate a formula explicit reward value based on the detection result of step S231 and the judgment result obtained in step S232. The calculation expression of the formula explicit reward value is:

[0032]

[0033] Where, Reward for explicitness of expression; represents the weight coefficient; Indicates that the operation expression has a value. If the output of the large language model contains an operation expression, the operation expression has a value of 1, otherwise it is 0. Indicates the reasonable value of the grammatical structure. If the grammatical structure of the operation expression is reasonable, the reasonable value of the grammatical structure is 1, otherwise it is 0; Indicates the numeric association value. If the numeric value in the operation expression is associated with the numeric value in the input text, the numeric association value is 1, otherwise it is 0.

[0034] Preferably, in step S24, whether the output content of the large language model contains fictitious values ​​or logically contradictory behavior is detected, and hallucination penalty term calculation is performed, which specifically includes the following sub-steps:

[0035] S241. Perform a numerical check on the output of the large language model to determine whether the following fictitious values ​​exist: the value does not appear in the input text or there is no intermediate reasoning process to provide a basis for deducing the source of the value;

[0036] S242. Perform a logical analysis on the numerical values ​​in the output of the large language model to determine whether any of the following logical contradictions exist: the numerical value exists in the input text but does not match the context information, including a unit error, a semantic error, or the numerical value is logically inconsistent with other numerical values ​​in the input text;

[0037] S243, determine the value of the hallucination penalty term according to the judgment results of step S241 and step S242, the calculation expression of the hallucination penalty term is:

[0038]

[0039] In the formula, represents the hallucination penalty term; represents the hallucination penalty intensity coefficient; represents the hallucination indication function value, if step S241 determines that there is a fictitious value or step S242 determines that there is a logical contradiction behavior, the hallucination indication function value is 1, otherwise 0.

[0040] Preferably, in step S3, the numerical consistency reward, the unit rationality reward, the expression explicitness reward and the hallucination penalty term are fused to generate a total reward function; and based on the total reward function, an improved GRPO loss function is constructed, specifically including the following substeps:

[0041] S31, according to the following expression, the numerical consistency reward, the unit rationality reward, the expression explicitness reward and the hallucination penalty term are fused to generate a total reward function:

[0042]

[0043] In the formula, represents the total reward function; represents the numerical consistency reward; represents the unit rationality reward; represents the expression explicitness reward; represents the hallucination penalty term; , , respectively represent the weight coefficient of the corresponding term;

[0044] S32, according to the total reward function, an improved GRPO loss function is constructed, and the calculation expression of the improved GRPO loss function is:

[0045]

[0046] In the formula, represents the improved GRPO loss function; represents the total reward function; represents the baseline value; represents the logarithm of the output probability of the large language model; represents the language modeling loss function; represents the balance parameter.

[0047] Preferably, in step S1, the reference answer includes: value, unit, expression and / or verifiable reasoning chain.

[0048] In a second aspect, the present invention proposes a numerical reasoning large model fine-tuning system, the system comprising:

[0049] A data set construction module is configured to construct a numerical reasoning data set, which includes input text and reference answers;

[0050] The multi-dimensional reward mechanism module is configured to introduce a multi-dimensional reward mechanism into the GRPO framework, specifically performing the following operations:

[0051] Calculate the numerical consistency reward based on the error between the predicted value of the large language model output and the true value of the reference answer;

[0052] Calculate unit rationality rewards based on the consistency between the generated units of the large language model output and the target units of the input text;

[0053] Detect whether the output of the large language model contains an arithmetic expression and perform explicit reward calculation;

[0054] Detect whether the output of the large language model contains fictitious values ​​or logically contradictory behavior, and perform hallucination penalty calculation;

[0055] The loss function construction module is configured to integrate the numerical consistency reward, unit rationality reward, formula explicit reward and hallucination penalty to generate the total reward function; and based on the total reward function, it constructs the improved GRPO loss function;

[0056] The fine-tuning optimization module is configured to fine-tune the large language model using the improved GRPO loss function, and finally output the fine-tuned large language model.

[0057] In a third aspect, the present invention proposes a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above-mentioned methods for fine-tuning a large numerical reasoning model are implemented.

[0058] In a fourth aspect, the present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned numerical reasoning large model fine-tuning methods.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] (1) Targeted improvement of numerical reasoning ability: By introducing a multi-dimensional reward mechanism into the GRPO framework, the large language model is optimized from multiple aspects such as numerical consistency, unit rationality, and formula explicitness, which significantly improves the model's performance in numerical reasoning tasks. It can accurately perform complex reasoning such as addition, subtraction, multiplication, and division, quantity comparison, and unit conversion, thus improving the ability to solve numerical problems.

[0061] (2) Effectively reduce the incidence of numerical hallucinations: Design a dedicated hallucination penalty term to strictly detect and punish model output content, reducing the occurrence of model-generated erroneous, fictitious, or inconsistent numbers. This improves the credibility and reliability of model output results, ensuring accurate information is provided in scenarios requiring precise numerical values, such as processing financial data and scientific research data.

[0062] (3) Enhance numerical memory: Through the learning of numerical reasoning datasets and the guidance of multi-dimensional reward mechanisms, the model can better remember and reproduce factual numbers, avoid inconsistent numerical descriptions when generating subsequent content, and improve the coherence and accuracy of the text.

[0063] (4) Broad Application Prospects: The method of the present invention is applicable to large language models based on the Transformer architecture (a deep learning architecture based on the self-attention mechanism), such as LLaMA (Large Language Model Meta AI), Qwen (Tongyi Qianwen), DeepSeek (Deep Seek), ChatGLM (Zhipu Qingyan), etc., and has good versatility and flexibility. It can meet the strict requirements of high precision and high reliability of models in numerically sensitive fields such as finance, scientific research, and education, providing strong support for the application of large language models in various numerically related fields.

[0064] (5) Improve model interpretability: By rewarding explicitness of formulas, the model is encouraged to write clear intermediate calculation processes when performing numerical reasoning, which enhances the interpretability and reasoning transparency of the model and makes it easier for users to understand and trust the decision-making process of the model.

[0065] (6) Maintaining language generation coherence and accuracy: While optimizing numerical reasoning capabilities, the human preference alignment advantage of GRPO is retained, ensuring that the text generated by the model remains fluent and coherent at the language level, in line with human language habits and logical thinking.

[0066] (7) Efficient fine-tuning training method: Construct an improved GRPO loss function and perform fine-tuning training based on the characteristics of the numerical field to achieve fine-tuning of numerical reasoning capabilities and improve the training efficiency and performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0068] Figure 1 is a flow chart of a method for fine-tuning a large numerical reasoning model according to the present invention;

[0069] Figure 2 is a schematic diagram of a numerical reasoning large model fine-tuning system according to the present invention;

[0070] Figure 3 It is a schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0072] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0073] This paper proposes a method for fine-tuning a large numerical reasoning model. Figure 1 The flowchart of the numerical reasoning large model fine-tuning method of the present invention is shown as follows: Figure 1 As shown, the method includes the following steps:

[0074] S1. Construct a numerical reasoning dataset. Specifically, construct a dataset containing high-quality numerical reasoning samples, covering a wide range of areas such as mathematical word problems, statistical descriptions, unit conversion, and dimensional analysis. Perform the following preprocessing steps on this numerical reasoning dataset to convert it into a fine-tuning dataset for training a large language model:

[0075] (a) Sample screening: only samples with clear numerical reasoning logic are retained;

[0076] (b) Structured processing: Split each numerical reasoning sample into input text and reference answer;

[0077] (c) Standardization of reference answers: Ensure that the reference answers contain at least one element of a numerical value, unit, expression, or verifiable chain of reasoning. A verifiable chain of reasoning refers to a text containing complete mathematical derivation steps.

[0078] In traditional GRPO (Guided Reinforcement Preference Optimization), the reward function is mostly based on the human preference scorer. In order to improve the numerical accuracy and reasoning ability, the reference Figure 2 The numerical reasoning large model fine-tuning method provided by the present invention further includes the following steps: S2, introducing a multi-dimensional reward mechanism into the GRPO framework, specifically performing the following operations:

[0079] S21. Calculate the numerical consistency reward based on the error between the predicted value of the large language model output and the true value of the reference answer. If the value generated by the large language model is consistent with the correct value inferred from the given conditions, an additional reward is given; otherwise, a penalty is imposed. The calculation expression is:

[0080]

[0081] Where, Represents the numerical consistency reward; Represents the reward coefficient, which controls the magnitude of the maximum reward value. It is a positive hyperparameter that determines the upper limit of the reward. represents the smoothing parameter, which controls the speed of error decay. The larger the value, the smaller the impact of the error on the reward (the stronger the "error tolerance"); Represents the predicted value of the large language model, that is, the numerical value in the text output by the large language model in the current inference task (such as 1.6 meters in "Xiao Ming is 1.6 meters tall"); Represents the true value of the reference answer, that is, the corresponding true value in the task or the correct answer marked by humans.

[0082] S22. Calculate the unit sanity reward based on the consistency between the generated units of the large language model output and the target units of the input text. This includes the following sub-steps:

[0083] S221, extracting information about the generation unit of the output content of the large language model and the target unit of the input text respectively through a regular function;

[0084] S222. Based on the extracted information of the generating unit and the target unit, determine the unit compatibility between the generating unit and the target unit, the explicitness of the unit reasoning chain, and whether there is a unit conflict (e.g., units with different dimensions are considered to be in conflict), and obtain a determination result.

[0085] S223. Generate a unit rationality reward value based on the judgment result obtained in step S222. The calculation expression of the unit rationality reward value is:

[0086]

[0087] Where, Indicates unit rationality reward; represents the weight coefficient; Indicates the unit matching value. If the unit generated by the large language model is consistent or equivalent to the target unit, the unit matching value is 1, otherwise it is 0; Indicates the explicit value of the unit inference chain. If there are traces of unit conversion inference in the output of the large language model (for example, the output contains a unit conversion formula), the explicit value of the unit inference chain is 1, otherwise it is 0. Indicates the unit conflict value. If the unit generated by the large language model conflicts with the target unit, the unit conflict value is 0, otherwise it is 1.

[0088] In a specific embodiment, when calculating the unit rationality reward, a regular function is first used to extract unit information from the input text and the model output. For example, if the input text is "Xiao Ming drove for 2 hours at a speed of 60 kilometers per hour" and the large language model output is "The driving distance is 120 kilometers", the extracted input units are "kilometers / hour" and "hours", and the output unit is "kilometers".

[0089] Secondly, determine the matching between the generated unit and the target unit. If the unit output by the large language model is consistent with the target unit or belongs to the equivalent unit class (such as "kilometer" and "kilometer"), the unit matching value is If the value is 1, it is 0. At the same time, check whether there are any traces of unit conversion reasoning in the output content. For example, if the output of the large language model contains the conversion of "1 hour = 3600 seconds", the unit reasoning chain explicit value If the output unit conflicts with the target unit (e.g. the speed unit should be "km / h" but the model output is "kg"), the unit conflict value is 0 if yes, otherwise it is 1.

[0090] Finally, based on the above judgment results, the unit rationality reward calculation formula is substituted to obtain the corresponding reward value. This method encourages large language models to generate numerical results that are consistent with the input context units and have a reasonable reasoning process, avoiding reasoning errors or misinterpretation caused by incorrect units, and improving the model's performance in professional fields such as physical quantity calculation, engineering analysis, and financial reporting.

[0091] S23: Check whether the output of the large language model contains an expression and perform expression explicit reward calculation, which specifically includes the following sub-steps:

[0092] S231: Check whether the output of the large language model contains an operation expression. If so, execute the operation of step S232; if not, set the expression explicitness reward value to 0, and neither reward nor penalty is given.

[0093] S232: determining the rationality of the grammatical structure of the operation expression and determining the correlation between the operation expression and the numerical value in the input text, and obtaining a determination result;

[0094] S233: Generate a formula explicit reward value based on the detection result of step S231 and the judgment result obtained in step S232. The calculation expression of the formula explicit reward value is:

[0095]

[0096] Where, Reward for explicitness of expression; represents the weight coefficient; Indicates that the operation expression has a value. If the output of the large language model contains an operation expression, the operation expression has a value of 1, otherwise it is 0. Indicates the reasonable value of the grammatical structure. If the grammatical structure of the operation expression is reasonable, the reasonable value of the grammatical structure is 1, otherwise it is 0; Indicates the numeric association value. If the numeric value in the operation expression is associated with the numeric value in the input text, the numeric association value is 1, otherwise it is 0.

[0097] In a specific embodiment, when calculating an explicit reward, the first step is to detect whether the output of the large language model contains an arithmetic expression. For example, if the large language model output is "The driving distance is 60 × 2 = 120 kilometers," the arithmetic expression "60 × 2 = 120" is detected.

[0098] Secondly, determine whether the syntax structure of the operation expression is reasonable. Check whether the operation expression complies with the syntax rules, such as whether there are correct operators (such as +, -, *, / , =) and bracket matching. If the expression syntax is correct, the syntax structure is reasonable. is 1 if the value is set, otherwise it is 0.

[0099] At the same time, the correlation between the operation expression and the numerical value in the input text is determined. For example, if the input text is "Xiao Ming drove at a speed of 60 kilometers per hour for 2 hours", the operation expression output by the large language model is "60×2=120", then the numerical values ​​60 and 2 in the expression are both from the input text, and the numerical correlation value is is 1 if the value is set, otherwise it is 0.

[0100] Finally, the test and judgment results are substituted into the explicit reward formula to determine the corresponding reward value. This approach encourages the model to explicitly describe the intermediate calculation process when performing numerical reasoning tasks, rather than simply providing conclusions or guesses. This enhances the model's interpretability and the reliability of its numerical reasoning, improving its performance in areas such as mathematical word problems, physical quantity calculations, and engineering analysis.

[0101] S24: Detect whether the output of the large language model contains fictitious values ​​or logically contradictory behavior, and perform hallucination penalty calculation, which specifically includes the following sub-steps:

[0102] S241. Perform a numerical check on the output of the large language model to determine whether the following fictitious values ​​exist: the value does not appear in the input text or there is no intermediate reasoning process to provide a basis for deducing the source of the value;

[0103] S242. Perform a logical analysis on the numerical values ​​in the output of the large language model to determine whether any of the following logical contradictions exist: the numerical value exists in the input text but does not match the context information, including a unit error, a semantic error, or the numerical value is logically inconsistent with other numerical values ​​in the input text;

[0104] S243. According to the judgment results of step S241 and step S242, the value of the hallucination penalty term is determined. The calculation expression of the hallucination penalty term is:

[0105]

[0106] Where, represents the hallucination penalty term; represents the intensity coefficient of illusion penalty; Represents the hallucination indicator function value. If step S241 determines that there is a fictitious value or step S242 determines that there is a logically contradictory behavior, the hallucination indicator function value is 1, otherwise it is 0.

[0107] In a specific embodiment, when calculating the hallucination penalty, the output of the large language model is first checked for a numerical value. For example, if the large language model outputs "driving distance is 150 kilometers," but the input text does not mention this value and there is no intermediate reasoning process to support its presence, then the value is determined to be a fictitious value.

[0108] Secondly, we perform logical analysis on the numerical values ​​in the large language model output. For example, if the large language model outputs "Xiao Ming drove at a speed of 60 kilometers per hour for 2 hours, covering a distance of 120 kilograms," the unit "kilogram" does not match the unit "kilometer" in the input text, indicating a unit error and a logical contradiction.

[0109] Finally, the above detection and judgment results are substituted into the calculation formula of the hallucination penalty term to obtain the corresponding penalty value. In this way, the model is penalized for fabricating numbers, incorrectly citing numerical values ​​in the question stem, or making blind guesses, thereby reducing the incidence of numerical hallucinations and improving the accuracy and reliability of the model in numerical reasoning tasks.

[0110] S3. Integrate numerical consistency rewards, unit rationality rewards, expression explicitness rewards, and hallucination penalties to generate a total reward function. Based on the total reward function, construct an improved GRPO loss function. Targeted at the characteristics of numerical reasoning tasks, an improved formula for the GRPO (Guided Reinforcement Preference Optimization) loss function is designed, which includes the integration of reward function guides (including numerical accuracy, unit rationality, expression explicitness, and hallucination penalties), as well as adjustments to the loss function structure itself. By improving the loss function, the model's ability to memorize numerical knowledge is enhanced, the model's ability to express numerical reasoning processes is improved, the rate of numerical hallucination is reduced, and the human preference alignment advantage of GRPO is retained. Specifically, the following sub-steps are included:

[0111] S31. Generate a total reward function by integrating the numerical consistency reward, unit rationality reward, formula explicitness reward, and hallucination penalty according to the following expression:

[0112]

[0113] Where, represents the total reward function; Represents the numerical consistency reward; Indicates unit rationality reward; Reward for explicitness of expression; represents the hallucination penalty term; 、 、 Respectively represent the weight coefficients of the corresponding items;

[0114] S32. Based on the total reward function, an improved GRPO loss function is constructed. The calculation expression of the improved GRPO loss function is:

[0115]

[0116] Where, It represents an improved GRPO loss function, which includes two parts: numerical reward and language model supervision; represents the total reward function; Represents the baseline value, which is used to reduce variance and improve training stability, usually the batch average reward; Represents the logarithm of the output probability of a large language model, the policy gradient part of reinforcement learning; represents the language modeling loss function; represents the equilibrium parameter.

[0117] The standard GRPO loss function consists of a preference reward guidance term and a language modeling term:

[0118]

[0119] in, represents the loss of preference based on reward guidance; represents the language modeling loss; Adjust the balance parameter between the reward-guided preference loss and the language modeling loss.

[0120] This paper improves the loss function structure by introducing an improvement term based on numerical reasoning rewards, so that the reward function explicitly encourages: numerical accuracy, unit consistency, equation explicitness, and hallucination penalty.

[0121] The following is the overall improved loss function:

[0122]

[0123] In a specific embodiment, when constructing the improved GRPO loss function, the numerical consistency reward, unit rationality reward, formula explicit reward and hallucination penalty are first integrated according to the total reward function. For example, the weight coefficient is set =0.4, =0.5, =0.1, and by substituting the corresponding reward values, the total reward function is calculated.

[0124] Then, based on the total reward function, we improve the traditional GRPO loss function. The traditional GRPO loss function consists of a preference reward guidance term and a language modeling term. In this improved loss function, we introduce a modification term based on numerical reasoning rewards, making the reward function explicitly encourage numerical accuracy, unit rationality, and explicit expression. Furthermore, we use a hallucination penalty term to reduce the numerical hallucination rate.

[0125] By integrating a reward function guide term and adjusting the loss function structure, the improved loss function enhances the model's ability to retain numerical knowledge and improves its expressiveness in numerical reasoning, while retaining GRPO's advantage of aligning with human preferences. This enables the model to more accurately process and generate numerical information in numerical reasoning tasks, reduces the occurrence of hallucinations, and improves overall performance.

[0126] Continue to refer Figure 1 The present invention provides a method for fine-tuning a large numerical reasoning model, which further includes the following steps:

[0127] S4. Use the improved GRPO loss function to fine-tune the large language model and finally output the fine-tuned large language model.

[0128] Specifically, during the fine-tuning training phase, a pre-trained large language model, such as the Transformer-based LLaMA or Qwen model, is first used as the base model. The model parameters are initialized, typically keeping most of the model parameters unchanged and fine-tuning only a few top-level parameters.

[0129] The preprocessed numerical reasoning dataset is fed into the model and trained using the modified GRPO loss function. During training, the model parameters are adjusted using a backpropagation algorithm to optimize the modified GRPO loss function. Preferably, a learning rate of 0.0001 and a batch size of 16 are set, and multiple training rounds are performed until the model's performance on the validation set no longer significantly improves.

[0130] Further references Figure 2 As an implementation of the above method, the present invention provides a second embodiment of a structural diagram of a numerical reasoning large model fine-tuning system 200. The system can be applied to various electronic devices. The numerical reasoning large model fine-tuning system 200 includes the following modules:

[0131] A data set construction module 210 is configured to construct a numerical reasoning data set, wherein the numerical reasoning data set includes input text and reference answers;

[0132] The multi-dimensional reward mechanism module 220 is configured to introduce a multi-dimensional reward mechanism into the GRPO framework, and specifically performs the following operations:

[0133] Calculate the numerical consistency reward based on the error between the predicted value of the large language model output and the true value of the reference answer;

[0134] Calculate unit rationality rewards based on the consistency between the generated units of the large language model output and the target units of the input text;

[0135] Detect whether the output of the large language model contains an arithmetic expression and perform explicit reward calculation;

[0136] Detect whether the output of the large language model contains fictitious values ​​or logically contradictory behavior, and perform hallucination penalty calculation;

[0137] The loss function construction module 230 is configured to fuse the numerical consistency reward, the unit rationality reward, the formula explicit reward and the hallucination penalty term to generate a total reward function; and construct an improved GRPO loss function based on the total reward function;

[0138] The fine-tuning optimization module 240 is configured to use the improved GRPO loss function to fine-tune the large language model and finally output the fine-tuned large language model.

[0139] In a third aspect, the present invention proposes a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above-mentioned methods for fine-tuning a large numerical reasoning model are implemented.

[0140] In a fourth aspect, the present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned numerical reasoning large model fine-tuning methods.

[0141] Reference below Figure 3 , which shows a structural diagram of a computer system 300 suitable for implementing a terminal device or server of an embodiment of the present application. Figure 3 The terminal device or server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0142] like Figure 3 As shown, computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of computer system 300 are also stored in RAM 303. CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0143] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, mouse, and the like; an output section 307 including a liquid crystal display (LCD), speakers, and the like; a storage section 308 including a hard disk and the like; and a communication section 309 including a network interface card such as a LAN card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is installed in the drive 310 as needed, so that computer programs read from the media can be installed in the storage section 308 as needed.

[0144] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method illustrated in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 309 and / or installed from removable media 311. When the computer program is executed by the central processing unit (CPU) 301, the functions defined in the methods of this application are performed. It should be noted that the computer-readable medium described herein can be a computer-readable signal medium or a computer-readable medium, or any combination thereof. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable media include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium that contains or stores a program for use by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, embodying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, or any suitable combination thereof.

[0145] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0147] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to technical solutions formed by a specific combination of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for fine-tuning a large numerical reasoning model, characterized in that: The following steps are involved: S1. Construct a numerical reasoning dataset, wherein the numerical reasoning dataset includes input text and reference answers; S2. Introduce a multi-dimensional reward mechanism into the GRPO framework, specifically by performing the following operations: S21. Calculate a numerical consistency reward based on the error between the predicted value of the output content of the large language model and the true value of the reference answer; S22: Calculate the unit rationality reward based on the consistency between the generated units of the output content of the large language model and the target units of the input text, specifically including the following sub-steps: S221. Extracting information about the generation unit of the output content of the large language model and the target unit of the input text respectively through a regular function; S222: Based on the extracted information of the generating unit and the target unit, determine the unit matching between the generating unit and the target unit, the explicitness of the unit reasoning chain, and whether there is a unit conflict, and obtain a determination result; S223. Generate a unit rationality reward value according to the judgment result obtained in step S222. The calculation expression of the unit rationality reward value is: Where, Indicates unit rationality reward; represents the weight coefficient; Indicates the unit matching value. If the unit generated by the large language model is consistent or equivalent to the target unit, the unit matching value is 1, otherwise it is 0; Indicates the explicit value of the unit inference chain. If there are traces of unit conversion reasoning in the output content of the large language model, the explicit value of the unit inference chain is 1, otherwise it is 0; Indicates the unit conflict value. If the unit generated by the large language model conflicts with the target unit, the unit conflict value is 0, otherwise it is 1; S23. Detect whether the output of the large language model contains an operation expression, and perform an explicit reward calculation for the expression; S24. Detect whether the output of the large language model contains fictitious values ​​or logically contradictory behavior, and perform hallucination penalty calculation; S3. Integrate the numerical consistency reward, unit rationality reward, formula explicitness reward, and hallucination penalty term to generate a total reward function; And based on the total reward function, construct an improved GRPO loss function; S4. Fine-tune the large language model using the improved GRPO loss function, and finally output the fine-tuned large language model.

2. The method for fine-tuning a large numerical reasoning model according to claim 1, characterized in that: In step S21, a numerical consistency reward is calculated based on the error between the predicted value of the output content of the large language model and the true value of the reference answer. The calculation expression of the numerical consistency reward is: Where, Represents the numerical consistency reward; represents the reward coefficient; represents the smoothing parameter; represents the predicted value of the large language model, Represents the true value of the reference answer.

3. The method for fine-tuning a large numerical reasoning model according to claim 1, characterized in that: In step S23, it is detected whether the output of the large language model contains an operation expression, and the explicit expression reward calculation is performed, which specifically includes the following sub-steps: S231: Detect whether the output content of the large language model contains an operation expression. If it contains the operation expression, execute the operation of step S232; S232: determining the rationality of the grammatical structure of the operation expression and determining the correlation between the operation expression and the numerical value in the input text to obtain a determination result; S233: Generate a formula explicit reward value based on the detection result of step S231 and the judgment result obtained in step S232. The calculation expression of the formula explicit reward value is: Where, Reward for explicitness of expression; represents the weight coefficient; Indicates that the operation expression has a value. If the output of the large language model contains an operation expression, the operation expression has a value of 1, otherwise it is 0. Indicates the reasonable value of the grammatical structure. If the grammatical structure of the operation expression is reasonable, the reasonable value of the grammatical structure is 1, otherwise it is 0; Indicates the numeric association value. If the numeric value in the operation expression is associated with the numeric value in the input text, the numeric association value is 1, otherwise it is 0.

4. The method for fine-tuning a large numerical reasoning model according to claim 1, wherein: In step S24, the output of the large language model is detected to see whether it contains fictitious values ​​or logically contradictory behavior, and hallucination penalty calculation is performed, which specifically includes the following sub-steps: S241: Perform a numerical check on the output of the large language model to determine whether there are any fictitious values: the values ​​do not appear in the input text or there is no intermediate reasoning process to provide a basis for deducing the source of the values; S242: Performing a logical analysis on the numerical values ​​in the output of the large language model to determine whether the following logical contradictions exist: the numerical value exists in the input text but does not match the context information, including a unit error or a semantic error, or the numerical value is logically inconsistent with other numerical values ​​in the input text; S243. Determine the value of the hallucination penalty term according to the judgment results of step S241 and step S242. The calculation expression of the hallucination penalty term is: Where, represents the hallucination penalty term; represents the intensity coefficient of illusion penalty; Represents the hallucination indicator function value. If step S241 determines that there is a fictitious value or step S242 determines that there is a logically contradictory behavior, the hallucination indicator function value is 1, otherwise it is 0.

5. The method for fine-tuning a large numerical reasoning model according to claim 1, characterized in that: In step S3, the numerical consistency reward, unit rationality reward, formula explicitness reward and hallucination penalty are integrated to generate a total reward function; and based on the total reward function, an improved GRPO loss function is constructed, which specifically includes the following sub-steps: S31. Generate a total reward function by integrating the numerical consistency reward, unit rationality reward, formula explicitness reward, and hallucination penalty term according to the following expression: Where, represents the total reward function; Represents the numerical consistency reward; Indicates unit rationality reward; Reward for explicitness of expression; represents the hallucination penalty term; 、 、 Respectively represent the weight coefficients of the corresponding items; S32. Based on the total reward function, construct an improved GRPO loss function. The calculation expression of the improved GRPO loss function is: Where, Represents the improved GRPO loss function; represents the baseline value; Represents the logarithm of the output probability of the large language model; represents the language modeling loss function; represents the equilibrium parameter.

6. The method for fine-tuning a large numerical reasoning model according to claim 1, characterized in that: In step S1 , the reference answer includes: a numerical value, a unit, an expression and / or a verifiable reasoning chain.

7. A numerical reasoning large model fine-tuning system for executing the numerical reasoning large model fine-tuning method according to any one of claims 1 to 6, characterized in that: The system comprises: A data set construction module is configured to construct a numerical reasoning data set, wherein the numerical reasoning data set includes an input text and a reference answer; The multi-dimensional reward mechanism module is configured to introduce a multi-dimensional reward mechanism into the GRPO framework, specifically performing the following operations: Calculate a numerical consistency reward based on the error between the predicted value of the large language model output and the true value of the reference answer; Based on the consistency between the generated units of the output content of the large language model and the target units of the input text, a unit rationality reward calculation is performed, which specifically includes the following sub-steps: S221. Extracting information about the generation unit of the output content of the large language model and the target unit of the input text respectively through a regular function; S222: Based on the extracted information of the generating unit and the target unit, determine the unit matching between the generating unit and the target unit, the explicitness of the unit reasoning chain, and whether there is a unit conflict, and obtain a determination result; S223. Generate a unit rationality reward value according to the judgment result obtained in step S222. The calculation expression of the unit rationality reward value is: Where, Indicates unit rationality reward; represents the weight coefficient; Indicates the unit matching value. If the unit generated by the large language model is consistent or equivalent to the target unit, the unit matching value is 1, otherwise it is 0; Indicates the explicit value of the unit inference chain. If there are traces of unit conversion reasoning in the output content of the large language model, the explicit value of the unit inference chain is 1, otherwise it is 0; Indicates the unit conflict value. If the unit generated by the large language model conflicts with the target unit, the unit conflict value is 0, otherwise it is 1; Detect whether the output of the large language model contains an arithmetic expression and perform explicit reward calculation; Detect whether the output of the large language model contains fictitious values ​​or logically contradictory behavior, and perform hallucination penalty calculation; A loss function construction module is configured to generate a total reward function by integrating the numerical consistency reward, unit rationality reward, formula explicit reward and hallucination penalty term; and to construct an improved GRPO loss function based on the total reward function; The fine-tuning optimization module is configured to use the improved GRPO loss function to fine-tune the large language model and finally output the fine-tuned large language model.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the numerical reasoning large model fine-tuning method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for fine-tuning a large numerical reasoning model are implemented as claimed in any one of claims 1 to 6.

Images