Numerical reasoning large model fine tuning method, system and equipment and storage medium

Abstract

The invention provides a numerical reasoning large model fine tuning method, system and device and a storage medium, and the method comprises the following steps: constructing a numerical reasoning data set which comprises an input text and a reference answer; a multi-dimensional reward mechanism is introduced into the GRPO framework, the multi-dimensional reward mechanism comprises the steps that numerical value consistency reward, unit rationality reward, equation explicit reward and illusion penalty terms are fused, and the multi-dimensional reward mechanism is fused to generate a total reward function; based on the total reward function, constructing an improved GRPO loss function; and performing fine tuning training on the large language model by adopting the improved GRPO loss function, and finally outputting the fine-tuned large language model. The method is particularly suitable for enhancing the memory ability and the numerical reasoning ability of the large language model in the field of numerical knowledge, and the tendency of generating numerical illusion can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of large model fine-tuning, and specifically to a method, system, device and storage medium for fine-tuning a large numerical reasoning model. Background Art

[0002] Large language models (such as the GPT series) can generate fluent and coherent text, and have outstanding performance in multiple tasks such as text generation, question-answering systems, and machine translation, providing strong support for solving various language-related problems.

[0003] However, existing large language models still face many challenges and limitations when dealing with tasks involving numerical knowledge and numerical reasoning, as follows: Numerical hallucination is a common problem: Models often generate erroneous, fictitious, or inconsistent numbers. For example, when processing financial data, scientific research data, and other scenarios that require precise numerical values, models may fabricate non-existent values or incorrectly modify or extrapolate existing values. This can lead to output results that are seriously inconsistent with reality, significantly impacting the model's credibility and reliability.

[0004] Weak numerical reasoning: Models often struggle to accurately complete complex numerical reasoning tasks, such as basic addition, subtraction, multiplication, and division, quantity comparisons, and unit conversions. They may fail to correctly understand the relationships and logic between numerical values and the corresponding operational rules, resulting in erroneous reasoning. This limits their application in specialized fields requiring high numerical reasoning, such as engineering calculations and financial analysis.

[0005] Poor numerical memory: The model also performs poorly in scenarios where accurate memorization and reproduction of factual numbers is required. It may forget or confuse previously mentioned numerical information, resulting in inconsistencies in the subsequent numerical descriptions when generating content, undermining the coherence and accuracy of the text.

[0006] To address the above problems and improve the performance of large language models in numerical tasks, researchers have tried various methods in recent years, including optimization techniques such as instruction fine-tuning (SFT), reinforcement learning (RLHF / GRPO, Reinforcement Learning with Human Feedback, reinforcement learning with human feedback; Guided Reinforcement Preference Optimization, guided reinforcement preference optimization). These methods can enhance the model's performance to a certain extent, but in numerical scenarios, no targeted and effective reinforcement mechanism has been formed, and problems such as numerical hallucinations, weak numerical reasoning, and poor numerical memory cannot be fundamentally solved, making it difficult to meet the strict requirements for high precision and high reliability of the model in numerically sensitive fields.

[0007] In view of this, the present invention proposes a fine-tuning method, system, device, and storage medium for a large numerical reasoning model, which can effectively improve the memory ability and numerical reasoning ability of large language models in the field of numerical knowledge, while reducing the tendency of numerical hallucinations. Summary of the Invention

[0008] To solve the problems of numerical hallucinations, weak numerical reasoning, and poor numerical memory in the prior art, the present invention provides a fine-tuning method, system, device, and storage medium for a large numerical reasoning model to solve the above technical defect problems.

[0009] In a first aspect, the present invention proposes a fine-tuning method for a large numerical reasoning model, including the following steps: S1. Construct a numerical reasoning dataset, which includes input text and reference answers; S2. Introduce a multi-dimensional reward mechanism into the GRPO framework, and specifically perform the following operations: S21. Calculate the numerical consistency reward according to the error between the predicted value of the output content of the large language model and the true value of the reference answer; S22. Perform unit rationality reward calculation according to the consistency between the generation unit of the output content of the large language model and the target unit of the input text; S23. Detect whether the output content of the large language model contains arithmetic expressions, and perform explicit arithmetic expression reward calculation; S24. Detect whether the output content of the large language model contains fictional numerical values or has logically contradictory behaviors, and perform hallucination penalty term calculation; S3. Integrate the numerical consistency reward, unit rationality reward, explicit arithmetic expression reward, and hallucination penalty term to generate a total reward function; and construct an improved GRPO loss function based on the total reward function; S4. Fine-tune and train the large language model using the improved GRPO loss function, and finally output the fine-tuned large language model.

[0010] Preferably, in step S21, according to the error between the predicted value of the content output by the large language model and the true value of the reference answer, calculate the numerical consistency reward. The calculation expression of the numerical consistency reward is:

[0011] In the formula, represents the numerical consistency reward; represents the reward coefficient; represents the smoothing parameter; represents the predicted value of the large language model, represents the true value of the reference answer.

[0012] Preferably, in step S22, according to the consistency between the generation unit of the content output by the large language model and the target unit of the input text, perform the calculation of the unit rationality reward, which specifically includes the following sub-steps: S221. Respectively extract the information of the generation unit of the content output by the large language model and the target unit of the input text through a regular function; S222. Based on the extracted information of the generation unit and the target unit, judge the unit matching, the explicitness of the unit inference chain, and whether there is a unit conflict, and obtain the judgment result; S223. Generate the unit rationality reward value according to the judgment result obtained in step S222. The calculation expression of the unit rationality reward value is:

[0013] In the formula, represents the unit rationality reward; represents the weight coefficient; represents the unit matching value. If the generation unit of the large language model is the same as or equivalent to the target unit, the unit matching value is 1, otherwise it is 0; represents the explicitness value of the unit inference chain. If there is a unit conversion inference trace in the content output by the large language model, the explicitness value of the unit inference chain is 1, otherwise it is 0; represents the unit conflict value. If the generation unit of the large language model conflicts with the target unit, the unit conflict value is 0, otherwise it is 1.

[0014] Preferably, in step S23, detect whether the content output by the large language model contains an arithmetic expression, and perform the calculation of the arithmetic explicitness reward, which specifically includes the following sub-steps: S231. Detect whether the content output by the large language model contains an arithmetic expression. If it contains an arithmetic expression, perform the operation in step S232; S232. Determine the rationality of the syntax structure of the arithmetic expression and judge the numerical correlation between the arithmetic expression and the numerical values in the input text to obtain a judgment result; S233. Generate an explicit formula reward value according to the detection result of step S231 and the judgment result obtained in step S232. The calculation expression of the explicit formula reward value is:

[0015] In the formula, represents the explicit formula reward; represents the weight coefficient; represents the existence value of the arithmetic expression. If the content output by the large language model contains an arithmetic expression, the existence value of the arithmetic expression is 1, otherwise it is 0; represents the rational syntax structure value. If the syntax structure of the arithmetic expression is reasonable, the rational syntax structure value is 1, otherwise it is 0; represents the numerical correlation value. If the numerical value in the arithmetic expression is correlated with the numerical values in the input text, the numerical correlation value is 1, otherwise it is 0.

[0016] Preferably, in step S24, it is detected whether the content output by the large language model contains fictional numerical values or logical contradiction behaviors, and the hallucination penalty term calculation is performed, specifically including the following sub-steps: S241. Perform numerical detection on the content output by the large language model to judge whether there are the following fictional numerical values: the numerical value does not appear in the input text or there is no intermediate reasoning process to provide a derivation basis for the numerical value source; S242. Perform logical analysis on the numerical values in the content output by the large language model to judge whether there are the following logical contradiction behaviors: the numerical value exists in the input text but does not match the context information, including unit errors, semantic errors, or the numerical value is logically inconsistent with other numerical values in the input text; S243. Determine the value of the hallucination penalty term according to the judgment results of step S241 and step S242. The calculation expression of the hallucination penalty term is:

[0017] In the formula, represents the hallucination penalty term; represents the hallucination penalty intensity coefficient; represents the hallucination indicator function value. If step S241 determines the existence of fictional numerical values or step S242 determines the existence of logical contradiction behaviors, the hallucination indicator function value is 1, otherwise it is 0.

[0018] Preferably, in step S3, a numerical consistency reward, a unit rationality reward, an arithmetic explicit reward, and an hallucination penalty term are fused to generate a total reward function; and an improved GRPO loss function is constructed based on the total reward function, specifically including the following sub-steps: S31. Fuse the numerical consistency reward, the unit rationality reward, the arithmetic explicit reward, and the hallucination penalty term according to the following expression to generate a total reward function:

[0019] In the formula, represents the total reward function; represents the numerical consistency reward; represents the unit rationality reward; represents the arithmetic explicit reward; represents the hallucination penalty term; , , respectively represent the weight coefficients of the corresponding terms; S32. Construct an improved GRPO loss function according to the total reward function. The calculation expression of the improved GRPO loss function is:

[0020] In the formula, represents the improved GRPO loss function; represents the total reward function; represents the baseline value; represents the logarithm of the output probability of the large language model; represents the language modeling loss function; represents the balance parameter.

[0021] Preferably, in step S1, the reference answer includes: numerical value, unit, expression, and / or verifiable reasoning chain.

[0022] In a second aspect, the present invention proposes a numerical reasoning large model fine-tuning system, which includes: A dataset construction module configured to construct a numerical reasoning dataset, and the numerical reasoning dataset includes input text and a reference answer; A multi-dimensional reward mechanism module configured to introduce a multi-dimensional reward mechanism in the GRPO framework and specifically perform the following operations: Calculate the numerical consistency reward according to the error between the predicted value of the output content of the large language model and the true value of the reference answer; Perform unit rationality reward calculation according to the consistency between the generated unit of the output content of the large language model and the target unit of the input text; Detect whether the output content of the large language model contains an arithmetic expression and perform arithmetic explicit reward calculation; Detect whether the output content of the large language model contains fictional numerical values or has logical contradiction behaviors, and perform the calculation of the hallucination penalty term; A loss function construction module, configured to fuse the numerical consistency reward, unit rationality reward, formula explicitness reward, and hallucination penalty term to generate a total reward function; and based on the total reward function, construct an improved GRPO loss function; A fine-tuning optimization module, configured to perform fine-tuning training on the large language model using the improved GRPO loss function, and finally output the fine-tuned large language model.

[0023] In a third aspect, the present invention proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the numerical reasoning large model fine-tuning method as described in any one of the above are implemented.

[0024] In a fourth aspect, the present invention proposes a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the numerical reasoning large model fine-tuning method as described in any one of the above are implemented.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Targeted improvement of numerical reasoning ability: By introducing a multi-dimensional reward mechanism into the GRPO framework, the large language model is optimized from multiple aspects such as numerical consistency, unit rationality, and formula explicitness, so that the performance of the model in numerical reasoning tasks is significantly improved. It can accurately perform complex inferences such as addition, subtraction, multiplication, division, quantity comparison, and unit conversion, and improve the ability to solve numerical problems.

[0026] (2) Effectively reduce the incidence of numerical hallucinations: Design a special hallucination penalty term to strictly detect and punish the output content of the model, and reduce the situation where the model generates incorrect, fictional, or inconsistent numbers. Improve the credibility and reliability of the model output results, and ensure that accurate information is provided in scenarios that require precise numerical values such as financial data and scientific research data.

[0027] (3) Enhance numerical memory: Through the learning of the numerical reasoning data set and the guidance of the multi-dimensional reward mechanism, the model can better remember and reproduce factual numbers, avoid the situation of inconsistent numerical descriptions when generating subsequent content, and improve the coherence and accuracy of the text.

[0028] (4)Wide application prospects: The method of the present invention is applicable to large language models based on the Transformer architecture (a deep learning architecture based on self-attention mechanism), such as LLaMA (Large Language Model Meta AI), Qwen (Thousand Questions), DeepSeek (Deep Quest), ChatGLM (Zhipu Qingyan), etc., with good generality and flexibility. It can meet the strict requirements of high precision and high reliability of models in numerically sensitive fields such as finance, scientific research, and education, providing strong support for the application of large language models in various numerical-related fields.

[0029] (5)Improve the interpretability of the model: Through explicit arithmetic rewards, the model is encouraged to write clear intermediate calculation processes during numerical reasoning, enhancing the interpretability and reasoning transparency of the model, and facilitating users to understand and trust the decision-making process of the model.

[0030] (6)Maintain the coherence and accuracy of language generation: While optimizing the numerical reasoning ability, the advantage of human preference alignment of GRPO is retained, ensuring that the text generated by the model remains fluent and coherent at the language level, conforming to human language habits and logical thinking.

[0031] (7)Efficient fine-tuning training method: Construct an improved GRPO loss function and perform fine-tuning training in combination with the characteristics of the numerical field to achieve fine-tuning of numerical reasoning ability, improving the training efficiency and performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects, and advantages of the present application will become more apparent: Figure 1 is a flowchart of the fine-tuning method for the large numerical reasoning model according to the present invention; Figure 2 is a schematic diagram of the fine-tuning system for the large numerical reasoning model according to the present invention; Figure 3 is a schematic diagram of the structure of a computer system of an electronic device suitable for implementing the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the related invention, rather than limiting the invention. Additionally, it should be noted that for the sake of description, only parts related to the relevant invention are shown in the drawings.

[0034] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0035] The present invention proposes a fine-tuning method for a numerical reasoning large model. Figure 1 The flowchart of the fine-tuning method for the numerical reasoning large model of the present invention is shown, as Figure 1 shown, the method includes the following steps: S1. Construct a numerical reasoning data set. Specifically, construct a data set containing high-quality numerical reasoning samples, whose fields widely cover math application problems, statistical descriptions, unit conversions, dimensional analyses, etc. The following preprocessing steps are sequentially performed on this numerical reasoning data set to be transformed into a fine-tuning data set for training a large language model: (a) Sample screening: Only retain samples containing clear numerical reasoning logic; (b) Structured processing: Split each numerical reasoning sample into an input text and a reference answer; (c) Standardization of the reference answer: Ensure that the reference answer contains at least one of the elements of numerical value, unit, expression, or verifiable reasoning chain. Among them, the verifiable reasoning chain refers to the text containing complete mathematical derivation steps.

[0036] In traditional GRPO (Guided Reinforcement Preference Optimization), the reward function is mostly based on a human preference scorer. To improve numerical accuracy and reasoning ability, in combination with reference Figure 2 , the fine-tuning method for the numerical reasoning large model provided by the present invention further includes the following steps: S2. Introduce a multi-dimensional reward mechanism in the GRPO framework, and specifically perform the following operations: S21. Calculate the Numerical Consistency Reward according to the error between the predicted value of the output content of the large language model and the true value of the reference answer. If the numerical value in the generated result of the large language model is consistent with the correct numerical value deduced from the problem setting conditions, an additional reward is given; otherwise, a penalty is imposed. The calculation expression is:

[0037] In the formula, represents the Numerical Consistency Reward; represents the reward coefficient, which controls the amplitude of the maximum reward value, is a positive hyperparameter, and determines the upper limit of the reward; represents the smoothing parameter, which controls the speed of error attenuation. The larger the value, the smaller the impact of the error on the reward (the stronger the "fault tolerance"); represents the predicted value of the large language model, that is, the numerical value in the text output by the large language model in the current reasoning task (such as 1.6 meters in "Xiaoming's height is 1.6 meters"); Represents the true value of the reference answer, that is, the corresponding true numerical value in the task or the correct answer marked by humans.

[0038] S22. According to the consistency between the generation unit of the output content of the large language model and the target unit of the input text, perform the calculation of the Unit Sanity Reward, which specifically includes the following sub-steps: S221. Extract the information of the generation unit of the output content of the large language model and the target unit of the input text respectively through a regular function; S222. Based on the extracted information of the generation unit and the target unit, judge the unit matching, the explicitness of the unit inference chain, and whether there is a unit conflict (for example, units with different dimensions are regarded as conflicts) between the generation unit and the target unit, and obtain the judgment result; S223. Generate the unit sanity reward value according to the judgment result obtained in step S222. The calculation expression of the unit sanity reward value is:

[0039] In the formula, represents the unit sanity reward; represents the weight coefficient; represents the unit matching value. If the generation unit of the large language model is the same as or equivalent to the target unit, the unit matching value is 1, otherwise it is 0; represents the unit inference chain explicitness value. If there is a unit conversion inference trace in the output content of the large language model (such as the output content contains a unit conversion formula), the unit inference chain explicitness value is 1, otherwise it is 0; represents the unit conflict value. If the generation unit of the large language model conflicts with the target unit, the unit conflict value is 0, otherwise it is 1.

[0040] In a specific embodiment, when performing the calculation of the unit sanity reward, first extract the unit information from the input text and the model output through a regular function. For example, if the input text is "Xiaoming traveled at a speed of 60 kilometers per hour for 2 hours", and the output content of the large language model is "The travel distance is 120 kilometers", then the extracted input units are "kilometers / hour" and "hour", and the output unit is "kilometers".

[0041] Secondly, judge the matching between the generation unit and the target unit. If the output unit of the large language model is the same as or belongs to an equivalent unit class (such as "kilometer" and "kilometer") as the target unit, the unit matching value is 1, otherwise it is 0. At the same time, check whether there is a unit conversion inference trace in the output content. For example, if the large language model output contains the conversion of "1 hour = 3600 seconds", the unit inference chain explicitness value It is 1, otherwise it is 0. If the output unit conflicts with the target unit (for example, the speed unit should be "km / h" while the model outputs "kg"), then the unit conflict value is 0, otherwise it is 1.

[0042] Finally, according to the above judgment results, substitute them into the calculation formula of the unit rationality reward to obtain the corresponding reward value. In this way, the large language model is encouraged to generate numerical results that are consistent with the units in the input context and have a reasonable reasoning process, avoiding reasoning mistakes or semantic errors caused by unit errors, and improving the performance of the model in professional fields such as physical quantity calculation, engineering analysis, and financial reports.

[0043] S23. Detect whether the output content of the large language model contains an arithmetic expression and perform the calculation of the Expression Explicitess Reward, which specifically includes the following sub-steps: S231. Detect whether the output content of the large language model contains an arithmetic expression. If it contains an arithmetic expression, perform the operation in step S232; if it does not contain an arithmetic expression, set the expression explicitess reward value to 0, and neither give a reward nor a penalty; S232. Judge the rationality of the syntax structure of the arithmetic expression and judge the numerical relevance between the arithmetic expression and the numerical values in the input text to obtain the judgment result; S233. Generate the expression explicitess reward value according to the detection result in step S231 and the judgment result obtained in step S232. The calculation expression of the expression explicitess reward value is:

[0044] In the formula, represents the expression explicitess reward; represents the weight coefficient; represents the existence value of the arithmetic expression. If the output content of the large language model contains an arithmetic expression, the existence value of the arithmetic expression is 1, otherwise it is 0; represents the syntax structure rationality value. If the syntax structure of the arithmetic expression is reasonable, the syntax structure rationality value is 1, otherwise it is 0; represents the numerical relevance value. If the numerical values in the arithmetic expression are relevant to the numerical values in the input text, the numerical relevance value is 1, otherwise it is 0.

[0045] In a specific embodiment, when calculating the expression explicitess reward, first detect whether the output content of the large language model contains an arithmetic expression. For example, if the output content of the large language model is "The driving distance is 60×2 = 120 km", then it is detected that the arithmetic expression "60×2 = 120" is contained.

[0046] Secondly, judge the rationality of the syntactic structure of the arithmetic expression. Check whether the arithmetic expression conforms to the syntax rules, such as whether there are correct operators (such as +, -, *, / , =) and bracket matching, etc. If the expression syntax is correct, the syntactic structure rationality value is 1, otherwise it is 0.

[0047] Meanwhile, judge the numerical relevance between the arithmetic expression and the input text. For example, if the input text is "Xiaoming traveled at a speed of 60 kilometers per hour for 2 hours", and the arithmetic expression output by the large language model is "60×2 = 120", then the numerical values 60 and 2 in the expression both come from the input text, and the numerical relevance value is 1, otherwise it is 0.

[0048] Finally, according to the detection results and judgment results, substitute them into the calculation formula of the explicit reward for the arithmetic expression to obtain the corresponding reward value. In this way, the model is encouraged to write out the explicit intermediate calculation process when performing numerical reasoning tasks, rather than just giving conclusions or guesses, thereby enhancing the interpretability of the model and the reliability of numerical reasoning, and improving the performance of the model in fields such as math word problems, physical quantity calculations, and engineering analysis.

[0049] S24. Detect whether the content output by the large language model contains fictional numerical values or has logical contradiction behaviors, and perform the calculation of the Hallucination Penalty, which specifically includes the following sub-steps: S241. Perform numerical detection on the content output by the large language model to judge whether there are the following fictional numerical values: the numerical value does not appear in the input text or there is no intermediate reasoning process to provide a derivation basis for the source of the numerical value; S242. Perform logical analysis on the numerical values in the content output by the large language model to judge whether there are the following logical contradiction behaviors: the numerical value exists in the input text, but does not match the context information, including unit errors, semantic errors, or the numerical value is logically inconsistent with other numerical values in the input text; S243. According to the judgment results of step S241 and step S242, determine the value of the hallucination penalty term. The calculation expression of the hallucination penalty term is:

[0050] In the formula, represents the hallucination penalty term; represents the hallucination penalty intensity coefficient; represents the hallucination indicator function value. If step S241 determines that there are fictional numerical values or step S242 determines that there are logical contradiction behaviors, then the hallucination indicator function value is 1, otherwise it is 0.

[0051] In a specific embodiment, when calculating the hallucination penalty term, numerical detection is first performed on the output content of the large language model. For example, if the output content of the large language model is "The driving distance is 150 kilometers", and this value is not mentioned in the input text and there is no intermediate reasoning process to support its appearance, it is determined that there is a fabricated value.

[0052] Secondly, logical analysis is performed on the values in the output content of the large language model. For example, if the output content of the large language model is "Xiaoming drove at a speed of 60 kilometers per hour for 2 hours, and the driving distance is 120 kilograms", where the unit "kilogram" does not match the unit "kilometer" in the input text, there is a unit error, and it is determined that there is a logical contradiction behavior.

[0053] Finally, according to the above detection and judgment results, substitute them into the calculation formula of the hallucination penalty term to obtain the corresponding penalty value. In this way, the behavior of the model generating fabricated numbers, misquoting the values in the question stem, or guessing blindly is punished, reducing the occurrence rate of numerical hallucinations and improving the accuracy and reliability of the model in numerical reasoning tasks.

[0054] S3. Integrate the numerical consistency reward, unit rationality reward, arithmetic explicitness reward, and hallucination penalty term to generate a total reward function; and based on the total reward function, construct an improved GRPO loss function. For the characteristics of numerical reasoning tasks, the improved formula design of the GRPO (Guided Reinforcement Preference Optimization) loss function is carried out, which includes the integration of the guiding terms of the reward function (including numerical accuracy, unit rationality, expression explicitness, hallucination penalty), and the adjustment of the structure of the loss function itself. By improving the loss function, enhance the model's memory ability for numerical knowledge, improve the model's expression ability in the numerical reasoning process, reduce the numerical hallucination rate, and retain the advantage of GRPO's alignment with human preferences. Specifically, it includes the following sub-steps: S31. Integrate the numerical consistency reward, unit rationality reward, arithmetic explicitness reward, and hallucination penalty term according to the following expression to generate a total reward function:

[0055] In the formula, represents the total reward function; represents the numerical consistency reward; represents the unit rationality reward; represents the arithmetic explicitness reward; represents the hallucination penalty term; , , respectively represent the weight coefficients of the corresponding terms; S32. Construct an improved GRPO loss function according to the total reward function. The calculation expression of the improved GRPO loss function is:

[0056] In the formula, represents the improved GRPO loss function, which includes two parts: numerical reward and language model supervision; represents the total reward function; represents the baseline value, which is used to reduce variance and improve training stability, and is usually the batch average reward; represents the logarithm of the output probability of the large language model, which is the policy gradient part in reinforcement learning; represents the language modeling loss function; represents the balance parameter.

[0057] The standard GRPO loss function consists of a preference reward guidance term and a language modeling term:

[0058] Among them, represents the preference loss based on reward guidance; represents the language modeling loss; is the balance parameter that adjusts the balance between the preference loss guided by the reward and the language modeling loss.

[0059] The present invention improves the loss function structure. By introducing an improved term based on numerical reasoning reward, the reward function explicitly encourages: Numerical Accuracy, Unit Consistency, Equation Explicitness, and Hallucination Penalty.

[0060] The following is the overall improved loss function:

[0061] In a specific embodiment, when constructing the improved GRPO loss function, first fuse the numerical consistency reward, unit rationality reward, arithmetic expression explicitness reward, and hallucination penalty term according to the total reward function. For example, set the weight coefficients = 0.4, = 0.5, = 0.1, and calculate the total reward function by substituting the corresponding reward values.

[0062] Then, based on the total reward function, the traditional GRPO loss function is improved. The traditional GRPO loss function consists of a preference reward guidance term and a language modeling term. In the improved loss function, an improved term based on numerical reasoning reward is introduced, making the reward function explicitly encourage numerical precision, unit rationality, and expression explicitness, and reducing the numerical hallucination rate through a hallucination penalty term.

[0063] The improved loss function enhances the model's memory ability for numerical knowledge and its expression ability in the numerical reasoning process by integrating the reward function guidance term and adjusting the loss function structure, while retaining the human preference alignment advantage of GRPO. This enables the model to more accurately process and generate numerical information in numerical reasoning tasks, reduce the occurrence of hallucination phenomena, and improve the overall performance.

[0064] Continue to refer to Figure 1 , a method for fine-tuning a large numerical reasoning model provided by the present invention further includes the following steps: S4. Use the improved GRPO loss function to fine-tune and train the large language model, and finally output the fine-tuned large language model.

[0065] Specifically, in the fine-tuning training stage, first use a pre-trained large language model as the base model, such as models based on the Transformer architecture like LLaMA, Qwen, etc. Initialize the model parameters, usually keeping most of the model parameters unchanged and only fine-tuning some top-layer parameters.

[0066] Input the pre-processed numerical reasoning data set into the model and use the improved GRPO loss function for training. During the training process, adjust the model parameters through the backpropagation algorithm to optimize the improved GRPO loss function. Preferably, set the learning rate to 0.0001 and the batch size to 16, and perform multiple training rounds until the performance of the model on the validation set no longer improves significantly.

[0067] Further refer to Figure 2 , as an implementation of the above method, in a second aspect, the present invention provides an embodiment of the structure diagram of a system 200 for fine-tuning a large numerical reasoning model, and this system can be specifically applied to various electronic devices. The system 200 for fine-tuning a large numerical reasoning model includes the following modules: A data set construction module 210 configured to construct a numerical reasoning data set, and the numerical reasoning data set includes input text and reference answers; A multi-dimensional reward mechanism module 220 configured to introduce a multi-dimensional reward mechanism in the GRPO framework and specifically perform the following operations: Calculate the numerical consistency reward according to the error between the predicted value of the output content of the large language model and the true value of the reference answer; Execute the calculation of the unit rationality reward according to the consistency between the generation unit of the output content of the large language model and the target unit of the input text; Detect whether the output content of the large language model contains arithmetic expressions, and execute the calculation of the explicit arithmetic expression reward; Detect whether the output content of the large language model contains fictional numerical values or has logically contradictory behaviors, and execute the calculation of the hallucination penalty term; The loss function construction module 230 is configured to integrate the numerical consistency reward, unit rationality reward, explicit arithmetic expression reward, and hallucination penalty term to generate a total reward function; and based on the total reward function, construct an improved GRPO loss function; The fine-tuning optimization module 240 is configured to fine-tune and train the large language model using the improved GRPO loss function, and finally output the fine-tuned large language model.

[0068] In a third aspect, the present invention proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above numerical reasoning large model fine-tuning methods are implemented.

[0069] In a fourth aspect, the present invention proposes a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of any of the above numerical reasoning large model fine-tuning methods are implemented.

[0070] Next, refer to Figure 3 , which shows a schematic structural diagram of a computer system 300 suitable for implementing the terminal device or server of the embodiments of the present application. Figure 3 The shown terminal device or server is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0071] As Figure 3 shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage section 308 into the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the computer system 300 are also stored. The CPU 301, ROM 302, and RAM 303 are connected to each other through a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0072] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD) and the like, and a speaker and the like; a storage section 308 including a hard disk and the like; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0073] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 309 and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-described functions defined in the method of the present application are performed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable medium or any combination of the two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0074] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0076] The above description is only a preferred embodiment of this application and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, technical solutions formed by mutually replacing the above features with (but not limited to) technical features having similar functions disclosed in this application.

Claims

1. A fine-tuning method for a large numerical reasoning model, characterized in that, It includes the following steps: S1. Construct a numerical reasoning dataset, where the numerical reasoning dataset includes input text and reference answers; S2. Introduce a multi-dimensional reward mechanism into the GRPO framework, and specifically perform the following operations: S21. Calculate the numerical consistency reward according to the error between the predicted value of the output content of the large language model and the true value of the reference answer; S22. Perform the calculation of the unit rationality reward according to the consistency between the generated unit of the output content of the large language model and the target unit of the input text; S23. Detect whether the output content of the large language model contains an arithmetic expression, and perform the calculation of the arithmetic expression explicitness reward; S24. Detect whether the output content of the large language model contains fictional numerical values or has logical contradiction behaviors, and perform the calculation of the hallucination penalty term; S3. Integrate the numerical consistency reward, unit rationality reward, arithmetic expression explicitness reward, and hallucination penalty term to generate a total reward function; And construct an improved GRPO loss function based on the total reward function; S4. Use the improved GRPO loss function to fine-tune and train the large language model, and finally output the fine-tuned large language model.

2. The numerical reasoning large model fine-tuning method according to claim 1, wherein In step S21, according to the error between the predicted value of the output content of the large language model and the true value of the reference answer, calculate the numerical consistency reward, and the calculation expression of the numerical consistency reward is: In the formula, represents the numerical consistency reward; represents the reward coefficient; represents the smoothing parameter; represents the predicted value of the large language model, represents the true value of the said reference answer.

3. The numerical reasoning large model fine-tuning method according to claim 1, characterized in that In step S22, according to the consistency between the generated unit of the output content of the large language model and the target unit of the input text, perform the calculation of the unit rationality reward, which specifically includes the following sub-steps: S221. Extract the information of the generated unit of the output content of the large language model and the target unit of the input text through regular functions respectively; S222. Based on the extracted information of the generated unit and the target unit, judge the unit matching, unit inference chain explicitness, and whether there is a unit conflict between the generated unit and the target unit, and obtain a judgment result; S223. Generate a unit rationality reward value according to the judgment result obtained in step S222, and the calculation expression of the unit rationality reward value is: Wherein, represents the unit rationality reward; represents the weight coefficient; represents the unit matching value. If the unit generated by the large language model is the same as or equivalent to the target unit, the unit matching value is 1; otherwise, it is 0; represents the explicit value of the unit inference chain. If there are traces of unit conversion inference in the content output by the large language model, the explicit value of the unit inference chain is 1; otherwise, it is 0; represents the unit conflict value. If the unit generated by the large language model conflicts with the target unit, the unit conflict value is 0; otherwise, it is 1.

4. The numerical reasoning large model fine-tuning method according to claim 1, characterized in that In step S23, detect whether the output content of the large language model contains an arithmetic expression, and perform the calculation of the arithmetic expression explicitness reward, which specifically includes the following sub-steps: S231. Detect whether the output content of the large language model contains an arithmetic expression. If it contains the arithmetic expression, perform the operation in step S232; S232. Judge the syntactic structure rationality of the arithmetic expression and judge the numerical relevance between the arithmetic expression and the numerical values in the input text, and obtain a judgment result; S233. Generate an arithmetic expression explicitness reward value according to the detection result in step S231 and the judgment result obtained in step S232, and the calculation expression of the arithmetic expression explicitness reward value is: In the formula, represents the explicit reward of the arithmetic expression; represents the weight coefficient; represents the existence value of the arithmetic expression. If the content output by the large language model contains an arithmetic expression, the existence value of the arithmetic expression is 1, otherwise it is 0; represents the reasonable value of the grammatical structure. If the grammatical structure of the arithmetic expression is reasonable, the reasonable value of the grammatical structure is 1, otherwise it is 0; represents the numerical correlation value. If the numerical value in the arithmetic expression is correlated with the numerical value in the input text, the numerical correlation value is 1, otherwise it is 0.

5. The method for fine-tuning a large numerical reasoning model according to claim 1, characterized in that: In step S24, detect whether the output content of the large language model contains fictional numerical values or has logical contradiction behaviors, and perform the calculation of the hallucination penalty term, which specifically includes the following sub-steps: S241. Numerically detect the output content of the large language model to determine whether there are the following fictional numerical values: the numerical value does not appear in the input text or there is no intermediate reasoning process to provide a derivation basis for the source of the numerical value; S242. Logically analyze the numerical values in the output content of the large language model to determine whether there are the following logical contradiction behaviors: the numerical value exists in the input text but does not match the context information, including unit errors, semantic errors, or the numerical value is logically inconsistent with other numerical values in the input text; S243. Determine the value of the hallucination penalty term according to the judgment results of steps S241 and S242. The calculation expression of the hallucination penalty term is: Where, represents the hallucination penalty term; represents the intensity coefficient of illusion penalty; Represents the hallucination indicator function value. If step S241 determines that there is a fictitious value or step S242 determines that there is a logically contradictory behavior, the hallucination indicator function value is 1, otherwise it is 0.

6. The numerical reasoning large model fine-tuning method according to claim 1, wherein In step S3, fuse the numerical consistency reward, unit rationality reward, arithmetic expression explicitness reward, and hallucination penalty term to generate a total reward function; and based on the total reward function, construct an improved GRPO loss function, specifically including the following sub-steps: S31. Fuse the numerical consistency reward, unit rationality reward, arithmetic expression explicitness reward, and hallucination penalty term according to the following expression to generate a total reward function: In the formula, represents the total reward function; represents the numerical consistency reward; represents the unit rationality reward; represents the arithmetic explicit reward; represents the hallucination penalty term; and and respectively represent the weight coefficients of the corresponding terms; S32. Construct an improved GRPO loss function according to the total reward function. The calculation expression of the improved GRPO loss function is: In the formula, represents the improved GRPO loss function; represents the total reward function; represents the baseline value; represents the logarithm of the output probability of the large language model; represents the language modeling loss function; represents the balance parameter.

7. The numerical reasoning large model fine-tuning method according to claim 1, characterized in that In step S1, the reference answer includes: numerical value, unit, expression, and / or verifiable reasoning chain.

8. A fine-tuning system for a large numerical reasoning model, characterized in that, The system includes: A dataset construction module configured to construct a numerical reasoning dataset, where the numerical reasoning dataset includes input text and a reference answer; A multi-dimensional reward mechanism module configured to introduce a multi-dimensional reward mechanism in the GRPO framework and specifically perform the following operations: Calculate the numerical consistency reward according to the error between the predicted value of the output content of the large language model and the true value of the reference answer; Perform unit rationality reward calculation according to the consistency between the generated unit of the output content of the large language model and the target unit of the input text; Detect whether the output content of the large language model contains an arithmetic expression and perform arithmetic expression explicitness reward calculation; Detect whether the output content of the large language model contains fictional numerical values or has logical contradiction behaviors and perform hallucination penalty term calculation; A loss function construction module configured to fuse the numerical consistency reward, unit rationality reward, arithmetic expression explicitness reward, and hallucination penalty term to generate a total reward function; and based on the total reward function, construct an improved GRPO loss function; A fine-tuning optimization module configured to fine-tune and train the large language model using the improved GRPO loss function and finally output a fine-tuned large language model.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the numerical reasoning large model fine-tuning method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the numerical reasoning large model fine-tuning method according to any one of claims 1 to 7.

Images