Large language model training method and device, storage medium and program product

Abstract

This application discloses a method, apparatus, storage medium, and program product for training a large language model, relating to the field of large model training technology. The method includes acquiring a training dataset; the training dataset includes a text-based dataset consisting of multiple sets of text data pairs containing input instruction text and corresponding output text; the output text contains a reference thought chain and a reference answer; the input instruction text is input into the large language model to be trained, obtaining the inference thought chain and inference answer generated by the large language model; based on the inference thought chain and the reference thought chain, the semantic similarity loss between them is calculated; based on the inference answer and the reference answer, the cross-entropy loss of literal matching between them is calculated; the large language model is trained according to the semantic similarity loss and the cross-entropy loss to determine the trained target large language model. The solution of this application solves the problem in existing large language model training methods that cannot simultaneously achieve both the conciseness of the thought chain and the accuracy of the answer.

Description

Technical Field

[0001] This application relates to the field of large model training technology, specifically to a large language model training method, apparatus, storage medium, and program product. Background Technology

[0002] Current large language model training generally relies on supervised fine-tuning (SFT) and cross-entropy loss, which forces the model output to be literally identical to the reference answer, resulting in lengthy and unvarnished reasoning processes. Although some solutions have proposed moving earth distance optimization to introduce semantic similarity metrics, this method still treats the semantically flexible reasoning process and the literal accuracy of the final answer as a unified whole, failing to distinguish between their different needs. Furthermore, the moving earth distance optimization scheme relies on Wasserstein distance as the core loss, which has high computational complexity and requires the construction of feasible upper bounds, further limiting its practical application. Therefore, existing training paradigms struggle to simultaneously optimize the semantic consistency of the reasoning process and the literal accuracy of the answer, especially in tasks such as mathematics that require precise output, where the model is prone to generating semantically reasonable but literally incorrect solution steps. Summary of the Invention

[0003] At least one embodiment of this application provides a large language model training method, apparatus, storage medium, and program product to solve the problem that existing large language model training methods cannot simultaneously achieve both conciseness of thought chain and accuracy of answer.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a method for training a large language model, including:

[0006] Obtain a training dataset; the training dataset includes a text-based dataset consisting of multiple sets of text data pairs; the text data pairs include input instruction text and output text corresponding to the input instruction text, and the output text contains a reference thought chain and a reference answer for guiding the reasoning process of the large language model;

[0007] The input instruction text is input into the large language model to be trained, and the reasoning thought chain and reasoning answer generated by the large language model to be trained are obtained.

[0008] Based on the reasoning thought chain and the reference thought chain, calculate the semantic similarity loss between the reasoning thought chain and the reference thought chain;

[0009] Based on the inferred answer and the reference answer, calculate the cross-entropy loss of literal matching between the inferred answer and the reference answer;

[0010] The large language model to be trained is trained based on the semantic similarity loss and the cross-entropy loss to determine the trained target large language model.

[0011] Optionally, the input instruction text is input into the large language model to be trained to obtain the reasoning thought chain and reasoning answer generated by the large language model to be trained, including:

[0012] Each pair of text data in the training dataset is used as a sample; the output text contains the reference thought chain and the reference answer, which are explicitly distinguished by a predefined separator; the predefined separator is a marker used to distinguish the thought chain and the answer;

[0013] The input instruction text from the sample is input into the large language model to be trained, generating model output text containing a complete response sequence;

[0014] Based on the predefined delimiter, the model output text is segmented into the reasoning thought chain and reasoning answer generated by the large language model to be trained.

[0015] Optionally, based on the reasoning thought chain and the reference thought chain, the semantic similarity loss between the reasoning thought chain and the reference thought chain is calculated, including:

[0016] Through the embedding layer of the large language model to be trained, the reasoning thought chain and the reference thought chain are respectively converted into the corresponding first semantic vector and second semantic vector.

[0017] Calculate the cosine similarity between the first semantic vector and the second semantic vector;

[0018] Based on a preset conversion rule, the cosine similarity is converted into a non-negative semantic similarity loss value, and the semantic similarity loss value is determined to be the semantic similarity loss between the reasoning thought chain and the reference thought chain.

[0019] Optionally, based on the inferred answer and the reference answer, the cross-entropy loss of literal matching between the inferred answer and the reference answer is calculated, including:

[0020] For each word in the reasoning answer, based on the input instruction text, the reasoning thought chain, and the complete context of the reasoning answer, the conditional probability of the large language model to be trained predicting the word is calculated.

[0021] The predicted probability of each of the aforementioned words is converted into a negative log probability;

[0022] The cross-entropy loss of the literal matching is obtained by summing the negative log probabilities of all lexical units.

[0023] Optionally, the large language model to be trained is trained based on the semantic similarity loss and the cross-entropy loss to determine the trained target large language model, including:

[0024] The semantic similarity loss and the literal matching cross-entropy loss are weighted and summed according to preset hyperparameters to determine the target loss function of the thought chain-answer.

[0025] Based on the target loss function, backpropagation and model parameter updates are performed on the large language model to be trained until a preset training batch is reached, and the trained target large language model is determined.

[0026] Optionally, based on the target loss function, backpropagation and model parameter updates are performed on the large language model to be trained until a preset training batch is reached, thereby determining the trained target large language model, including:

[0027] Based on the target loss function, gradient calculation is performed on the large language model to be trained to determine the gradients of all trainable parameters;

[0028] Based on the gradient of the unfrozen trainable parameters, a preset optimization algorithm is used to perform gradient update on the unfrozen trainable parameters;

[0029] Before reaching the preset training batch, repeat the steps of determining the gradients and updating the gradients of all trainable parameters until the target large language model is determined.

[0030] Optionally, the preset hyperparameters are determined by calculation using a preset mapping function, and the value range of the preset hyperparameters is [0, 1].

[0031] Wherein, when the preset hyperparameter value is less than 0.5, it is used to indicate that the diversity of thought chains is emphasized during the training process; when the preset hyperparameter value is greater than 0.5, it is used to indicate that the accuracy of the answer is emphasized during the training process; the diversity of thought chains is used to indicate the degree of semantic similarity between the reasoning thought chain and the reference thought chain, and the lower the similarity, the higher the diversity of the reasoning thought chain.

[0032] Secondly, embodiments of this application provide a large language model training device, comprising:

[0033] The acquisition module is used to acquire the training dataset; the training dataset includes a text-based dataset consisting of multiple sets of text data pairs; the text data pairs include input instruction text and output text corresponding to the input instruction text, and the output text contains reference thought chains and reference answers for guiding the reasoning process of the large language model;

[0034] The first processing module is used to input the input instruction text into the large language model to be trained, and obtain the reasoning thought chain and reasoning answer generated by the large language model to be trained.

[0035] The second processing module is used to calculate the semantic similarity loss between the inference thinking chain and the reference thinking chain based on the inference thinking chain and the reference thinking chain.

[0036] The third processing module is used to calculate the cross-entropy loss of literal matching between the inferred answer and the reference answer based on the inferred answer and the reference answer;

[0037] The fourth processing module is used to train the large language model to be trained based on the semantic similarity loss and the cross-entropy loss, and determine the trained target large language model.

[0038] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method as described in any one of the first aspects.

[0039] Fourthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the method as described in any one of the first aspects.

[0040] Compared with existing technologies, the large language model training method, apparatus, storage medium, and program product provided in this application obtain a training dataset, which includes a text-based dataset consisting of multiple sets of text data pairs containing input instruction text and corresponding output text; the output text contains a reference thought chain and a reference answer; the input instruction text is input into the large language model to be trained to obtain the reasoning thought chain and reasoning answer generated by the large language model to be trained; the model's reasoning logic is dynamically adjusted based on the semantic alignment between the reference thought chain and the reasoning thought chain; and the accuracy of the answer is directly optimized using cross-entropy loss to ensure that the reasoning result matches the literal meaning of the reference answer. The large language model to be trained is trained based on semantic similarity loss and cross-entropy loss, enabling the model to learn both a reasonable reasoning process and the correct answer during training. This dual-loss joint optimization solves the problem in existing large language model training methods that cannot simultaneously balance the simplicity of the thought chain and the accuracy of the answer. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 One of the flowcharts for the large language model training method provided in the embodiments of this application;

[0043] Figure 2 A second schematic flowchart of the large language model training method provided in the embodiments of this application;

[0044] Figure 3 This is a schematic diagram of the structure of the large language model training device provided in the embodiments of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0047] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc.; an indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0048] To enable those skilled in the art to better understand the embodiments of this application, the following description is provided first:

[0049] The related earthmoving distance optimization technology improves upon the problem that traditional cross-entropy loss only focuses on text literal matching. It uses Wasserstein distance as the core indicator to measure the difference between the model's output distribution and the target distribution. During implementation, a transmission cost function based on word embedding and perceptible semantic information is introduced. By optimizing this function, the model can learn content that is semantically equivalent to the reference answer but has a different linguistic expression during training, thus achieving flexible semantic expression of the reasoning process.

[0050] The method for optimizing the moving earthwork distance still treats the entire output text of the model as a unified whole and uses a single loss function for training constraints. Although this method introduces the dimension of semantic similarity consideration, it does not specifically differentiate and optimize the semantic flexibility required in the process and the literal accuracy required in the final answer. At the same time, the process of directly optimizing the moving earthwork distance has high computational complexity, and a feasible upper bound needs to be constructed for it to complete the training.

[0051] As described in the background section, in existing large language model training, supervised fine-tuning suppresses reasoning diversity due to single cross-entropy loss, and schemes such as moving earth distance optimization fail to distinguish the differentiated needs of thought chains and answers, and have high computational complexity. They struggle to balance semantic consistency in the reasoning process with literal accuracy of the answer, severely impacting the model's performance and practical application in high-precision reasoning tasks. To address at least one of the above problems, embodiments of this application provide a large language model training method, apparatus, storage medium, and program product that ensures literal accuracy of the answer while simplifying computational logic, reducing complexity, and effectively improving the model's overall performance in complex reasoning tasks.

[0052] This application provides a method, apparatus, storage medium, and program product for training a large language model. The method and apparatus are based on the same concept, and since they solve problems based on similar principles, their implementations can be mutually referenced; repeated details will not be repeated.

[0053] Please refer to Figure 1 This application provides a method for training a large language model, comprising:

[0054] Step 11, obtain the training dataset; the training dataset includes a text-based dataset consisting of multiple sets of text data pairs; the text data pairs include input instruction text and output text corresponding to the input instruction text, and the output text contains reference thought chains and reference answers for guiding the reasoning process of the large language model.

[0055] Here, step 11 is used to construct a standardized text training data foundation, providing a reference for subsequent differential loss calculation and model training, and clarifying the compositional attributes of the training data. The training dataset is a text dataset, specifically consisting of multiple sets of text data pairs. Composition, in which, It represents the input command text, which can cover the input needs of various task scenarios such as mathematical calculation, logical deduction, and instruction following; This indicates the text of the input command. The output text corresponds one-to-one, and this step is for the output text. A targeted design was implemented, comprising two parts: a reference thought chain and a reference answer. The reference thought chain defines the semantic boundaries and logical paths of the model's reasoning process, providing a reference for subsequent evaluation of the model's reasoning effectiveness. The reference answer clarifies the literal precision of the final task output. These two components work together to provide a clear basis for calculating the differentiated loss of the thought chain and the answer, addressing the problem at the data level that existing technologies fail to differentiate between thought chains and answers during training and cannot adapt to their varying needs.

[0056] It should be noted that the "thought chain" refers to the intermediate reasoning process text generated by the large language model in response to the input instruction text in order to derive the final conclusion. It serves as the logical carrier for the model to analyze the problem and make step-by-step deductions. In this application, the thought chains are divided into two categories: one is the reference thought chain (the thought chain in the sample annotation), which is generated by data distillation and is used to define the semantic boundaries of a reasonable reasoning path; the other is the reasoning thought chain (the thought chain generated by the model), which is autonomously derived by the model to be trained and must maintain semantic consistency with the reference thought chain while preserving expressive diversity.

[0057] It should also be noted that the answer refers to the final conclusive text output by the large language model after reasoning based on the thought chain, which is a direct reflection of the task objective. In this application, the answers are divided into two categories: one is the reference answer (the answer in the sample annotation), which is the standard conclusion of the task and serves as the reference benchmark for literal matching; the other is the reasoned answer (the answer generated by the model), which must be completely consistent with the reference answer in literal terms, especially suitable for high-precision task scenarios such as mathematical calculations and structured output.

[0058] Step 12: Input the input instruction text into the large language model to be trained to obtain the reasoning thought chain and reasoning answer generated by the large language model to be trained.

[0059] Step 12 serves as the stage for establishing a comparison system between the model output and the reference standard, achieving the initial output and corresponding matching of the reasoning ability of the large language model to be trained. Specifically, it involves comparing the input instruction texts from each group in the training dataset obtained in step 11. Each input is fed into the large language model to be trained. Based on its current parameter configuration, the model parses and reasons about the input command text, generating a response. This response can be naturally divided into two parts: the reasoning chain and the reasoning answer. The reasoning chain corresponds to the reference chain in step 11, representing the logical process derived autonomously by the model. The reasoning answer corresponds to the reference answer in step 11, representing the conclusion reached by the model based on the reasoning process. This step ensures a one-to-one correspondence between the model output and the reference content, establishing a necessary comparative basis for the subsequent two steps of targeted loss calculation.

[0060] Step 13: Based on the reasoning thought chain and the reference thought chain, calculate the semantic similarity loss between the reasoning thought chain and the reference thought chain.

[0061] To address the need for semantic consistency and flexible literal expression in the thought chain, a loss function is designed to overcome the limitations of the single loss constraint in existing technologies and achieve precise optimization of the reasoning process. Since the thought chain lies in the semantic fit between the reasoning logic and the reference thought chain, rather than the complete unity of literal expression, step 13 uses a semantic similarity measurement method to quantify the degree of fit between the reasoning thought chain and the reference thought chain.

[0062] Step 14: Based on the inferred answer and the reference answer, calculate the cross-entropy loss of the literal matching between the inferred answer and the reference answer.

[0063] To prioritize the literal accuracy of the reference answer, a loss function is designed to ensure the accuracy of the model's final output. For tasks such as mathematical calculations and objective conclusion output, the reference answer must have strict literal consistency. Therefore, step 14 uses traditional cross-entropy loss to quantify the literal matching error between the inference answer and the reference answer.

[0064] Step 15: Train the large language model to be trained based on the semantic similarity loss and the cross-entropy loss to determine the trained target large language model.

[0065] In this embodiment, step 15 is the execution stage of model optimization. Through the synergistic effect of two types of losses, the model's reasoning ability and output accuracy are simultaneously improved. Specifically, firstly, preset hyperparameters are set according to task requirements. The semantic similarity loss obtained in step 13 and the cross-entropy loss obtained in step 14 are weighted and summed to construct a target loss function. Based on this target loss function, training is performed, and the trained target large language model is determined. Compared with the Wasserstein distance optimization scheme of the existing earthmoving distance optimization scheme, this method significantly simplifies the loss calculation logic, reduces the computational complexity of training, and improves the overall performance of the model in complex reasoning tasks.

[0066] This application, through the coordinated execution of steps 11 to 15, achieves differentiated training optimization of the thought chain and the answer. This solves the problem in existing large language model training methods that cannot simultaneously balance the conciseness of the thought chain and the accuracy of the answer, while also ensuring the semantic reliability of the model's reasoning process and the literal accuracy of the final answer. This enhances the practical application value of large language models in various complex and high-precision reasoning tasks. The large language model training method provided in this application, by applying differentiated loss constraints to the thought chain and the final answer, balances the semantic consistency of the reasoning process and the literal accuracy of the answer, while simplifying the training computation logic and overcoming the shortcomings of existing technologies.

[0067] Optionally, step 12 above includes:

[0068] Each pair of text data in the training dataset is used as a sample; the output text contains the reference thought chain and the reference answer, which are explicitly distinguished by a predefined separator; the predefined separator is a marker used to distinguish the thought chain and the answer;

[0069] The input instruction text from the sample is input into the large language model to be trained, generating model output text containing a complete response sequence;

[0070] Based on the predefined delimiter, the model output text is segmented into the reasoning thought chain and reasoning answer generated by the large language model to be trained.

[0071] It should be noted that step 11 above involves acquiring and preprocessing the training data. The training dataset is loaded before model training begins or at the start of each training batch. .in, It is an input command. yes The corresponding output. Wherein, the output... It consists of two parts: reference mind chain and reference answers These two parts are explicitly separated by predefined special delimiters (such as answer labels). During MindChain training, the model is intentionally trained to output complete text sequences of "reasoning process and answer," rather than just the answer. (See MindChain for reference.) It is an explicit inference text output.

[0072] In this embodiment of the application, each group of text data in the training dataset is paired } as independent training samples, where the input instruction text Input the sample, output the text This serves as a reference output for sample annotation; to achieve differentiated training of subsequent thought processes and answers, the output text is... An explicit structured design is employed, comprising a reference thought chain and a reference answer, precisely distinguished by a predefined separator (e.g., an answer label) to define the boundary between the thought chain and the answer. This clarifies the independent attributes of the thought chain and the answer at the data level, providing a unified rule for the subsequent splitting of the model output. Input instructions from the sample are fed one by one into the large language model to be trained. Based on its current parameter configuration, the model parses and infers from the input instructions using an autoregressive approach, generating a complete response sequence (i.e., the model output text) containing the reasoning process and the final conclusion. This ensures that the model output maintains the same structural dimension as the sample reference output, providing a matching basis for subsequent splitting and comparison. Following the rule of complete consistency with the sample output text, the complete response sequence generated by the model is precisely segmented by searching for the predefined separator, breaking it down into reasoning thought chains. and reasoning answers The system consists of two independent parts: the reasoning thought chain corresponds to the reference thought chain of the sample, and the reasoning answer corresponds to the reference answer of the sample. This achieves a one-to-one accurate match between the model output and the sample annotation, providing a clear comparison object for subsequent calculations of the semantic similarity loss of the thought chain and the literal matching cross-entropy loss of the answer. This is a key preliminary step for achieving differentiated loss supervision.

[0073] Optionally, step 13 above includes:

[0074] Through the embedding layer of the large language model to be trained, the reasoning thought chain and the reference thought chain are respectively converted into the corresponding first semantic vector and second semantic vector.

[0075] Calculate the cosine similarity between the first semantic vector and the second semantic vector;

[0076] Based on a preset conversion rule, the cosine similarity is converted into a non-negative semantic similarity loss value, and the semantic similarity loss value is determined to be the semantic similarity loss between the reasoning thought chain and the reference thought chain.

[0077] In this embodiment of the application, the reasoning thought chain generated by the model is quantified and measured. Reference thought chain with the sample The semantic consistency between the two, rather than the literal matching degree, is used to obtain a semantic similarity loss specifically for constraining the reasoning logic of the thought chain. This is achieved by converting text into semantic vectors, calculating vector similarity, and mapping it to loss values. Specifically, this application utilizes the embedding layer of the large language model to be trained to construct the embedding function. The reasoning chain With reference mind chain Enter the embedding function respectively By leveraging the feature extraction capabilities of the model's embedding layer, both text segments are converted into high-dimensional semantic representation vectors, corresponding to the first semantic vector of the reasoning thought chain. and the second semantic vector of the reference thought chain This achieves the transformation of text from the character level to the semantic feature level, providing a foundation for subsequent similarity calculation; among them, the embedding function The specific implementation includes two optional schemes: one is to take the hidden state of the last token in the token sequence of the model output text (i.e., the last column of the embedding layer matrix), and the other is to perform average pooling on the hidden states of all tokens in the token sequence. This application does not limit the above two implementation methods.

[0078] Based on the similarity measurement rules of vector space, the first semantic vector of the reasoning thought chain is calculated. and the second semantic vector of the reference thought chain The cosine similarity between two semantic vectors is calculated using the following formula: ; express and The cosine similarity value of two semantic vectors ranges from [-1, 1]. The closer the value is to 1, the more consistent the directions of the two vectors are, and the higher the semantic similarity of the thought chain. The closer the value is to -1, the opposite the directions of the vectors are, and the lower the semantic similarity. A value of 0 indicates that the vectors are orthogonal and have no semantic relevance. The dot product (inner product) operation represents the multiplication of two vectors. It is calculated by multiplying the corresponding elements of the two vectors one by one and then summing the results; the denominator... This involves normalizing the dot product of the numerators to ensure the final result is constrained within the range of [-1, 1]. This formula quantifies the degree of semantic overlap between two thought chains, with the calculated result ranging from [-1, 1]. A higher value indicates a stronger semantic and logical fit between the reasoning thought chain and the reference thought chain.

[0079] The cosine similarity results are mapped to non-negative loss values ​​to achieve the optimization goal of lower loss values ​​for higher semantic similarity. Specifically, a preset formula is used to convert similarity into semantic similarity loss for the thought chain. The calculation formula is: This formula converts the similarity score within the range [-1, 1] into a loss value within the range [0, 1], ultimately yielding... This is the loss value used to measure the semantic matching degree of the thought chain, which serves as the basis for optimizing the reasoning logic of the thought chain during subsequent model training.

[0080] It should be noted that in this step... The design is based on information theory, and this application reveals through rigorous mathematical proof that minimizing this semantic similarity loss function and maximizing conditional mutual information are synergistic. This theoretical relationship can be constrained and embodied by the following inequality: ;in, This represents the conditional mutual information between the reasoning chain T and the reference chain R, given the answer A. This is the core information quantity for maximizing the expectation. It is about semantic differences (i.e., what this application optimizes). The core part of the expected value, that is, the actual loss target to be minimized. and It is a normal constant related to the specific data distribution ("specific data distribution" refers to the statistical distribution characteristics of all data involved in the training process (input, thought chain, answer, etc.), which directly affect the values ​​of constants in the theoretical analysis and the convergence performance of the algorithm). The profound meaning of the formula lies in the fact that it reveals mutual information. With loss items There is a positive coupling relationship between them. During model training, when optimization algorithms (such as gradient descent) are committed to reducing the loss term... At this point, this optimization behavior will drive the model parameters into a new state, in which mutual information... The value of also increases accordingly. Therefore, the design of this step theoretically ensures that the model, during the learning and reasoning process, will optimize in the direction of "preserving the same core logical information as the reference thought chain to the greatest extent." This provides solid theoretical support for the use of semantic constraints in this method, proving its effectiveness in preserving the essence of reasoning.

[0081] Optionally, step 14 above includes:

[0082] For each word in the reasoning answer, based on the input instruction text, the reasoning thought chain, and the complete context of the reasoning answer, the conditional probability of the large language model to be trained predicting the word is calculated.

[0083] The predicted probability of each of the aforementioned words is converted into a negative log probability;

[0084] The cross-entropy loss of the literal matching is obtained by summing the negative log probabilities of all lexical units.

[0085] In this embodiment of the application, this step, to address the requirement of strict literal consistency in the answer and ensuring accurate output, employs word-by-word cross-entropy loss. Calculation Method. By quantifying the predicted probability of each word in the reference answer, the literal matching loss between the inferred answer and the reference answer is calculated. This forces the model to optimize the literal accuracy of the final answer, avoiding the problem of semantically reasonable reasoning chains but incorrect answers. Specifically, for each word in the reference answer, combined with complete contextual information (including the input instruction text, the model-generated reasoning chain, and all word sequences preceding that word in the inferred answer), the large language model predicts the conditional probability corresponding to the word at that position, given all the aforementioned contextual information. Here, "word" refers to the Chinese translation of "token." The answer text is divided and decoded using the BPE algorithm to ensure that the granularity of probability calculation is consistent with the granularity of the model's text processing. The conditional probability of each word obtained above is converted into a negative logarithmic probability. The negative logarithmic transformation maps the probability value to a non-negative value, achieving a conversion where a higher predicted probability corresponds to a smaller negative logarithmic probability, thereby quantifying the model's prediction error for a single word.

[0086] This application ensures the accuracy of the reasoning answer. Compared with the reference answer Strictly consistent in wording. Calculated using the standard token-by-token cross-entropy loss function: ;in, This represents the word-by-word cross-entropy loss value for the answer portion. The smaller the loss value, the higher the literal matching degree between the model's generated answer and the reference answer; the larger the loss value, the greater the deviation between the model's predicted words and the reference answer. The "-" sign represents a negative sign and is the core symbol of cross-entropy loss. It converts the negative log probability into a non-negative value (since the log probability itself is negative), ensuring that the loss value is non-negative, which conforms to the optimization logic that the smaller the loss, the better. This is the reference answer. The j-th word in the text is the smallest unit of text processing, which is obtained by segmenting the reference answer text using the Byte-Pair Encoding (BPE) algorithm. The model is given all the preceding context (including the input instruction text). Reasoning chain And the answer to the reasoning Given the complete context, predict the word element as The probability of . Where, less than j represents the sequence of all tokens before the j-th token, j represents the j-th word, and the comma before the less than sign is used to separate i, gen and j.

[0087] This application applies cross-entropy loss only to the constraints of the answer part, which is different from the existing technology that applies word-by-word metaliteral matching to the entire sequence of "thought chain and answer". This ensures the literal accuracy of the answer output and avoids the suppression of the diversity of thought chain expression by cross-entropy loss, thus realizing the supervision of the difference loss between thought chain and answer.

[0088] Optionally, the large language model to be trained is trained based on the semantic similarity loss and the cross-entropy loss to determine the trained target large language model, including:

[0089] The semantic similarity loss and the literal matching cross-entropy loss are weighted and summed according to preset hyperparameters to determine the target loss function of the thought chain-answer.

[0090] Based on the target loss function, backpropagation and model parameter updates are performed on the large language model to be trained until a preset training batch is reached, and the trained target large language model is determined.

[0091] In this embodiment, a target loss function for the thought chain-answer model is constructed through weighted fusion, and then backpropagation and parameter updates are performed using this target to finally determine the target model. The semantic similarity loss of the thought chain obtained above is then used... Cross-entropy loss that matches the literal meaning of the answer According to the preset hyperparameters We perform weighted summation to construct the target loss function of the thought chain-answer used for final model optimization. The formula is: in, Pre-set hyperparameters for training are strictly limited to the range [0,1] (exemplarily set to 0.5). Their function is to balance the importance of the two losses in model training and increase... It can strengthen the constraint on the semantic consistency of the thought chain and reduce... This strengthens the constraint on the literal accuracy of the answer. The objective loss function... It is not simply a matter of loss summation; its superiority is rigorously supported by generalization theory. This is proven through derivation using a pre-defined Probabilistically Approximately Correct Bayesian (PAC-Bayes) framework. The upper bound of the generalization error of the trained model is defined by the newly defined distributional dissimilarity metric in this application. The decision, from a mathematical perspective, ensures the model's ability to balance reasoning flexibility with answer accuracy.

[0092] This application uses the target loss function The backpropagation algorithm is executed on the large language model to be trained, and the trainable parameters of the model are updated based on the gradient results obtained from the backpropagation. The loss calculation, backpropagation, and parameter update processes are iteratively executed until all training data has been traversed or a preset training batch has been reached. Training is then stopped, and the trained target large language model is determined. It should be noted that all trainable parameters θ include, but are not limited to: word embedding layer parameters, positional encoding parameters, multi-head self-attention mechanism parameters, feedforward neural network parameters, layer normalization layer parameters, and output layer parameters used to generate answers. These parameters are all updated using gradient descent, and the frozen pre-trained weights do not participate in this update process. Here, the core of backpropagation is to find the weights that make the target large language model... The model parameter adjustment direction is minimized, and the model is optimized iteratively to simultaneously meet the training objectives of "semantic alignment of the thought chain with the reference logic" and "strict matching of the literal meaning of the answer with the reference content". The setting of dual termination conditions takes into account both the sufficiency and efficiency of training, and avoids undertraining or overfitting of the model.

[0093] Furthermore, based on the target loss function, backpropagation and model parameter updates are performed on the large language model to be trained until a preset training batch is reached, thereby determining the trained target large language model, including:

[0094] Based on the target loss function, gradient calculation is performed on the large language model to be trained to determine the gradients of all trainable parameters;

[0095] Based on the gradient of the unfrozen trainable parameters, a preset optimization algorithm is used to perform gradient update on the unfrozen trainable parameters;

[0096] Before reaching the preset training batch, repeat the steps of determining the gradients and updating the gradients of all trainable parameters until the target large language model is determined.

[0097] In this embodiment of the application, the target loss function is the thought chain-answer method. Based on the baseline, calculate The gradient is relative to the gradient of all unfrozen trainable parameters θ in the large language model to be trained; where all unfrozen trainable parameters θ refer to all trainable parameters in the model except those manually frozen, not a single parameter. The gradient result reflects the rate of change of the loss function with respect to each parameter, clarifying the direction and magnitude of parameter adjustment. Based on the full gradient results obtained above, a preset optimization algorithm (the AdamW algorithm is used as an example in this application, but other mainstream deep learning optimization algorithms can also be adapted) is used to perform gradient updates on all unfrozen trainable parameters θ of the model; the optimization algorithm dynamically adjusts the parameter update step size according to the magnitude and direction of the gradient to ensure the convergence efficiency and stability of model training.

[0098] Iterative execution and satisfying the two termination conditions: Steps 11 to 15 are treated as a complete training iteration unit, and the unit is repeatedly executed until either of the following termination conditions is met: (1) a complete traversal of all training data in the training dataset is completed; (2) the preset training batch is reached.

[0099] In this application, the model learns the feature patterns in the training data through full-process iteration, and the dual termination conditions can be flexibly selected according to actual training needs, taking into account both training effect and computational resource consumption.

[0100] Furthermore, the preset hyperparameters are determined by calculation using a preset mapping function, and the value range of the preset hyperparameters is [0, 1].

[0101] Wherein, when the preset hyperparameter value is less than 0.5, it is used to indicate that the diversity of thought chains is emphasized during the training process; when the preset hyperparameter value is greater than 0.5, it is used to indicate that the accuracy of the answer is emphasized during the training process; the diversity of thought chains is used to indicate the degree of semantic similarity between the reasoning thought chain and the reference thought chain, and the lower the similarity, the higher the diversity of the reasoning thought chain.

[0102] In this application, hyperparameters are clearly defined. The dynamic setting rules, unlike the traditional fixed-value setting method, allow the loss weights to adapt to the real-time state of the training process. Here, hyperparameters... These are not pre-set fixed values, but are jointly determined by two core real-time metrics for the current training batch: the Mind Chain Diversity Index (quantifies the expressive diversity of the mind chains generated by the model; a higher value indicates that the reasoning process is not rigidly literal and has greater semantic flexibility); and the Answer Accuracy Index (quantifies the literal matching rate between the model-generated answer and the reference answer; a higher value indicates better answer accuracy). These two core real-time metrics are input into a pre-set mapping function (such as a linear function or a non-linear regression function), and the mapping function calculates and outputs hyperparameters adapted to the current training state. .

[0103] It should be noted that if the diversity index of thought chains is low in the current training batch (meaning the model's reasoning expression is rigid), the mapping function will automatically increase. The value of , improve The loss weights encourage the model to learn richer reasoning expressions; if the accuracy of answers in the current training batch is low (poor literal matching of model answers), the mapping function will automatically decrease. The value of , improve The loss weights are used to strengthen the constraint on the literal accuracy of the answer. Hyperparameters The dynamic setting rules allow the loss weights to be adjusted in real time according to the diversity of the thought processes and the accuracy of the answers in the training batch, realizing dynamic collaborative optimization of the reasoning process and the answer output, and further enhancing the adaptability of the differentiated supervised training of this application.

[0104] The large language model training method in this application implements differentiated loss constraint training on the thought process and the final answer, balancing the semantic consistency of the reasoning process with the literal accuracy of the answer, while simplifying the training computation logic. (Refer to...) Figure 2 As shown, the specific steps of the large language model training method provided in this application embodiment are as follows:

[0105] Step S101: Obtain training data and perform preprocessing. Obtain training data consisting of multiple sets of text data pairs { The text training dataset consists of}, where, Enter the command text. To ensure that the output text corresponds to and explicitly includes the reference thought chain and reference answer, it provides a targeted reference for subsequent differential loss calculation, and distinguishes the different training needs of thought chain and answer from the data level.

[0106] Step S102: Perform model forward propagation, generate the response, and decompose it. Input the large language model to be trained Autoregression generates complete response sequences Then, it is decomposed into a chain of reasoning ideas according to the same delimiter rules. and reasoning answers Establish a comparative relationship with the reference content.

[0107] Step S103: Calculate the semantic similarity loss of the thought chain. This is done through the model embedding layer. and reference mind chain Convert to semantic vectors, calculate cosine similarity and map it to semantic similarity loss. This quantifies the semantic consistency between the two while preserving the diversity of reasoning expressions.

[0108] Step S104: Calculate the literal matching loss of the answer. The word-by-word cross-entropy loss method is used to calculate the inference answer. Compared with the reference answer Literal matching loss This forces the model to output a literal, exact answer.

[0109] Step S105: Weight and fuse the two losses to obtain the final thought chain - answer loss. and By hyperparameters Weighted summation yields the final target loss function for the thought chain-answer. The training needs to balance reasoning and answering.

[0110] Step S106: Perform backpropagation and update the model parameters. The objective function is to calculate the gradient of the unfrozen trainable parameters of the model, update the parameters through the optimization algorithm, and iteratively execute steps S101 to S106 until the preset training conditions are met to obtain the target large language model.

[0111] This application Figure 2 The method shown achieves differentiated training through six collaborative steps, which solves the problem that existing large language model training methods cannot balance the conciseness of the thought chain and the accuracy of the answer, and improves the generalization ability and application value of the model in complex and high-precision reasoning tasks.

[0112] The proposed large language model training method explicitly decomposes the output of the large language model into two independent parts: the thought chain and the answer. Semantic similarity loss and literal matching cross-entropy loss are applied selectively. The target loss function is constructed through the collaborative optimization and weighted fusion of these two methods. This fundamentally solves the shortcomings of existing technologies, such as the traditional MLE cross-entropy loss's constraint of uniformly processing the "thought chain and answer" sequences, which suppresses reasoning diversity, and the high computational complexity of the moving earth distance optimization loss, which cannot balance reasoning flexibility and answer accuracy. This method achieves targeted training of the reasoning process and answer output through standardized training dataset construction, precise decomposition of model response, differentiated loss calculation, and efficient model parameter optimization. It combines training efficiency with deployment practicality, and while ensuring the reproducibility of the technical solution, it brings multiple technical benefits, as detailed below:

[0113] (1) Achieving synergistic enhancement of reasoning quality and answer accuracy, breaking through performance bottlenecks. This application applies semantic similarity loss to the thinking chain part, only constraining the semantic consistency of the reasoning process, preserving the diversity and expressive flexibility of the model's reasoning logic, and effectively improving the overall score of the thinking chain; at the same time, it applies strict cross-entropy loss to the answer part, forcibly constraining the literal precision of the final conclusion, ensuring a stable improvement in answer accuracy, and completely solving the problem of the trade-off between reasoning flexibility and answer accuracy in existing moving earth distance optimization techniques, achieving a two-way improvement of both. Validated by tasks such as mathematical reasoning and text quality assessment, the model comprehensively surpasses MLE and moving earth distance optimization comparison methods in terms of overall thinking chain score and answer accuracy.

[0114] (2) Enhance the model's ability to follow instructions and adapt to the requirements of structured output tasks. The strong literal matching constraint on the answer part of this application forces the model to learn and accurately replicate the preset output format requirements. Compared with the earthmoving distance optimization method that only focuses on semantic consistency, it effectively avoids the problem of format mismatch, improves the model's answer format accuracy in structured output tasks such as JSON, greatly enhances the model's ability to follow complex output instructions and reliability, and broadens the application scope of the model in various practical scenarios that require standardized and structured output.

[0115] (3) Improve model inference efficiency and reduce actual deployment costs. This application imposes semantic similarity constraints on the thought chain, encouraging the model to learn from the core logic of the thought chain rather than redundant literal expressions, making the inference process generated by the model more concise and efficient, effectively shortening the length of the thought chain, improving model performance while accelerating inference speed, reducing the consumption of computing resources, reducing the deployment cost and operating overhead of large language models in practical applications, and improving the feasibility of the solution.

[0116] (4) Significantly reduce training computational complexity and improve model training efficiency. The semantic similarity loss in this application is achieved by combining the model's own embedding layer with cosine similarity. It has low computational overhead and simple logic. Compared with the moving earth distance optimization technique, which requires solving complex optimization problems, it does not require the construction of a feasible upper bound, which greatly simplifies the loss calculation logic. At the same time, by constructing the target loss function through simple weighted summation, and in conjunction with conventional backpropagation and gradient update strategies, a lighter and more efficient model training is achieved under the premise of ensuring solid mathematical theoretical support. This reduces the requirements for training hardware and improves the overall training efficiency.

[0117] In summary, the large language model training method presented in this application, through differentiated supervised training, forms a complete and efficient training system from data foundation, model output, loss calculation to parameter optimization. This not only addresses the core technical deficiencies of existing technologies but also achieves comprehensive improvements in inference quality, answer accuracy, instruction compliance, training efficiency, and inference efficiency. Actual experiments have verified that the model's performance indicators outperform existing MLE and moving earth distance optimization methods. This method effectively improves the overall performance of large language models in various complex and high-precision inference tasks, reduces the cost of model training and deployment, significantly enhances the practical application value of large language models, and provides a novel, efficient, and feasible solution for optimizing the inference capabilities of large language models.

[0118] The various methods of the embodiments of this application have been described above. Apparatus for implementing the above methods will now be provided.

[0119] Please refer to Figure 3This application also provides a large language model training device, comprising:

[0120] The acquisition module 31 is used to acquire a training dataset; the training dataset includes a text-based dataset consisting of multiple sets of text data pairs; the text data pairs include input instruction text and output text corresponding to the input instruction text, and the output text contains a reference thought chain and a reference answer for guiding the reasoning process of the large language model;

[0121] The first processing module 32 is used to input the input instruction text into the large language model to be trained, and obtain the reasoning thought chain and reasoning answer generated by the large language model to be trained.

[0122] The second processing module 33 is used to calculate the semantic similarity loss between the reasoning thought chain and the reference thought chain based on the reasoning thought chain and the reference thought chain.

[0123] The third processing module 34 is used to calculate the cross-entropy loss of literal matching between the inferred answer and the reference answer based on the inferred answer and the reference answer;

[0124] The fourth processing module 35 is used to train the large language model to be trained based on the semantic similarity loss and the cross-entropy loss, and to determine the trained target large language model.

[0125] Optionally, the first processing module 32 described above is specifically used for:

[0126] Each pair of text data in the training dataset is used as a sample; the output text contains the reference thought chain and the reference answer, which are explicitly distinguished by a predefined separator; the predefined separator is a marker used to distinguish the thought chain and the answer;

[0127] The input instruction text from the sample is input into the large language model to be trained, generating model output text containing a complete response sequence;

[0128] Based on the predefined delimiter, the model output text is segmented into the reasoning thought chain and reasoning answer generated by the large language model to be trained.

[0129] Optionally, the second processing module 33 described above is specifically used for:

[0130] Through the embedding layer of the large language model to be trained, the reasoning thought chain and the reference thought chain are respectively converted into the corresponding first semantic vector and second semantic vector.

[0131] Calculate the cosine similarity between the first semantic vector and the second semantic vector;

[0132] Based on a preset conversion rule, the cosine similarity is converted into a non-negative semantic similarity loss value, and the semantic similarity loss value is determined to be the semantic similarity loss between the reasoning thought chain and the reference thought chain.

[0133] Optionally, the third processing module 34 described above is specifically used for:

[0134] For each word in the reasoning answer, based on the input instruction text, the reasoning thought chain, and the complete context of the reasoning answer, the conditional probability of the large language model to be trained predicting the word is calculated.

[0135] The predicted probability of each of the aforementioned words is converted into a negative log probability;

[0136] The cross-entropy loss of the literal matching is obtained by summing the negative log probabilities of all lexical units.

[0137] Optionally, the fourth processing module 35 described above includes:

[0138] The first processing unit is used to perform a weighted summation of the semantic similarity loss and the literal matching cross-entropy loss according to preset hyperparameters to determine the target loss function of the thought chain-answer.

[0139] The second processing unit is used to perform backpropagation and model parameter update on the large language model to be trained according to the target loss function until a preset training batch is reached, and to determine the trained target large language model.

[0140] Optionally, the second processing unit described above is specifically used for:

[0141] Based on the target loss function, gradient calculation is performed on the large language model to be trained to determine the gradients of all trainable parameters;

[0142] Based on the gradient of the unfrozen trainable parameters, a preset optimization algorithm is used to perform gradient update on the unfrozen trainable parameters;

[0143] Before reaching the preset training batch, repeat the steps of determining the gradients and updating the gradients of all trainable parameters until the target large language model is determined.

[0144] Optionally, the preset hyperparameters are determined by calculation using a preset mapping function, and the value range of the preset hyperparameters is [0, 1].

[0145] Wherein, when the preset hyperparameter value is less than 0.5, it is used to indicate that the diversity of thought chains is emphasized during the training process; when the preset hyperparameter value is greater than 0.5, it is used to indicate that the accuracy of the answer is emphasized during the training process; the diversity of thought chains is used to indicate the degree of semantic similarity between the reasoning thought chain and the reference thought chain, and the lower the similarity, the higher the diversity of the reasoning thought chain.

[0146] It should be noted that the device in this embodiment corresponds to the method described above, and the implementation methods in each of the above embodiments are applicable to the embodiments of this device, achieving the same technical effect. The device provided in this application embodiment can implement all the method steps implemented in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0147] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described large language model training method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0148] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described large language model training method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0149] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0151] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for training a large language model, characterized in that, include: Obtain a training dataset; the training dataset includes a text-based dataset consisting of multiple sets of text data pairs; the text data pairs include input instruction text and output text corresponding to the input instruction text, and the output text contains a reference thought chain and a reference answer for guiding the reasoning process of the large language model; The input instruction text is input into the large language model to be trained, and the reasoning thought chain and reasoning answer generated by the large language model to be trained are obtained. Based on the reasoning thought chain and the reference thought chain, calculate the semantic similarity loss between the reasoning thought chain and the reference thought chain; Based on the inferred answer and the reference answer, calculate the cross-entropy loss of literal matching between the inferred answer and the reference answer; The large language model to be trained is trained based on the semantic similarity loss and the cross-entropy loss to determine the trained target large language model.

2. The method according to claim 1, characterized in that, The input instruction text is input into the large language model to be trained, and the reasoning thought chain and reasoning answer generated by the large language model to be trained are obtained, including: Each pair of text data in the training dataset is used as a sample; the output text contains the reference thought chain and the reference answer, which are explicitly distinguished by a predefined separator; the predefined separator is a marker used to distinguish the thought chain and the answer; The input instruction text from the sample is input into the large language model to be trained, generating model output text containing a complete response sequence; Based on the predefined delimiter, the model output text is segmented into the reasoning thought chain and reasoning answer generated by the large language model to be trained.

3. The method according to claim 1, characterized in that, Based on the reasoning thought chain and the reference thought chain, the semantic similarity loss between the reasoning thought chain and the reference thought chain is calculated, including: Through the embedding layer of the large language model to be trained, the reasoning thought chain and the reference thought chain are respectively converted into the corresponding first semantic vector and second semantic vector. Calculate the cosine similarity between the first semantic vector and the second semantic vector; Based on a preset conversion rule, the cosine similarity is converted into a non-negative semantic similarity loss value, and the semantic similarity loss value is determined to be the semantic similarity loss between the reasoning thought chain and the reference thought chain.

4. The method according to claim 1, characterized in that, Based on the inferred answer and the reference answer, calculate the cross-entropy loss of literal matching between the inferred answer and the reference answer, including: For each word in the reasoning answer, based on the input instruction text, the reasoning thought chain, and the complete context of the reasoning answer, the conditional probability of the large language model to be trained predicting the word is calculated. The predicted probability of each of the aforementioned words is converted into a negative log probability; The cross-entropy loss of the literal matching is obtained by summing the negative log probabilities of all lexical units.

5. The method according to claim 1, characterized in that, The large language model to be trained is trained based on the semantic similarity loss and the cross-entropy loss to determine the trained target large language model, including: The semantic similarity loss and the literal matching cross-entropy loss are weighted and summed according to preset hyperparameters to determine the target loss function of the thought chain-answer. Based on the target loss function, backpropagation and model parameter updates are performed on the large language model to be trained until a preset training batch is reached, and the trained target large language model is determined.

6. The method according to claim 5, characterized in that, Based on the target loss function, backpropagation and model parameter updates are performed on the large language model to be trained until a preset training batch is reached, thereby determining the trained target large language model, including: Based on the target loss function, gradient calculation is performed on the large language model to be trained to determine the gradients of all trainable parameters; Based on the gradient of the unfrozen trainable parameters, a preset optimization algorithm is used to perform gradient update on the unfrozen trainable parameters; Before reaching the preset training batch, repeat the steps of determining the gradients and updating the gradients of all trainable parameters until the target large language model is determined.

7. The method according to claim 5, characterized in that, The preset hyperparameters are determined by calculation using a preset mapping function, and the value range of the preset hyperparameters is [0, 1]. Wherein, when the preset hyperparameter value is less than 0.5, it is used to indicate that the diversity of thought chains is emphasized during the training process; when the preset hyperparameter value is greater than 0.5, it is used to indicate that the accuracy of the answer is emphasized during the training process; the diversity of thought chains is used to indicate the degree of semantic similarity between the reasoning thought chain and the reference thought chain, and the lower the similarity, the higher the diversity of the reasoning thought chain.

8. A large language model training device, characterized in that, include: The acquisition module is used to acquire the training dataset; the training dataset includes a text-based dataset consisting of multiple sets of text data pairs; the text data pairs include input instruction text and output text corresponding to the input instruction text, and the output text contains reference thought chains and reference answers for guiding the reasoning process of the large language model; The first processing module is used to input the input instruction text into the large language model to be trained, and obtain the reasoning thought chain and reasoning answer generated by the large language model to be trained. The second processing module is used to calculate the semantic similarity loss between the inference thinking chain and the reference thinking chain based on the inference thinking chain and the reference thinking chain. The third processing module is used to calculate the cross-entropy loss of literal matching between the inferred answer and the reference answer based on the inferred answer and the reference answer; The fourth processing module is used to train the large language model to be trained based on the semantic similarity loss and the cross-entropy loss, and determine the trained target large language model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images