Model training method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a model training method and device, electronic equipment and storage medium, which are related to the field of artificial intelligence, and can be used in the field of large models, the field of reinforcement learning, etc. The method comprises: obtaining a first training set, each training data comprising a first training sample and a corresponding first label; training an initial model based on the first training set to obtain a fine-tuned training model, and obtaining an error output of the initial model about each first training sample during the training process; obtaining a second training set, and training the fine-tuned model based on the second training set to obtain a logical reasoning model. By providing sufficient and various types of training data, the problem of single training mode, one-sided model capability development, weak model thinking ability and weak generalization ability is solved, so that the model obtained by training has strong generalization ability, can adapt to changes of different types of input data, and improves the applicability, generalization and effect of the model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a model training method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continued rise of generative large language models, they have attracted much attention due to their powerful natural language processing, understanding, and generation capabilities. Existing large language models often follow a "fast-thinking" approach when performing generation tasks (i.e., when solving a user's input problem, they quickly infer and directly generate the answer response without any thinking or reasoning process). This "fast-thinking" generation method relies on the large language model's learning and training on massive amounts of data, enabling it to quickly capture the semantic and syntactic structure in the context, thus making efficient and reasonable predictions when generating the next character.

[0003] However, for tasks that require large language models to perform thinking and logical reasoning, the "fast thinking" mode exhibits weaker reasoning ability and lower accuracy in generating results, especially in large language models with a small number of parameters. Particularly for generation tasks that require thinking and logical reasoning (such as multi-dimensional information extraction), the results generated by the "fast thinking" mechanism are often unsatisfactory. Therefore, a training method is needed to enhance the thinking ability of large language models and make their generated results more accurate. Summary of the Invention

[0004] The purpose of this application is to at least solve one of the aforementioned technical defects. The technical solution provided by the embodiments of this application is as follows: In a first aspect, embodiments of this application provide a model training method, including: Obtain the first training set, which includes multiple training data. Each training data includes a first training sample and a corresponding first label. Each first training sample is a piece of text content, and each first label includes the inference result obtained based on the corresponding text content. The initial model is trained based on the first training set to obtain the fine-tuned model, and the erroneous output of the initial model for each first training sample during the training process is obtained; wherein, the erroneous output is the output of the initial model for each first training sample that is different from the corresponding first label. Obtain a second training set, which includes multiple second training samples and corresponding positive and negative sample labels for each second training sample. A second training sample can be a first training sample, a first combination consisting of a first training sample and its corresponding first label, or a second combination consisting of a first training sample and its corresponding error output. When a second training sample is a first training sample, the positive sample label is the first label corresponding to that first training sample, and the negative sample label is the error output corresponding to that first training sample. When a second training sample is a first combination, the positive sample label indicates correct reasoning, and the negative sample label indicates incorrect reasoning. When a second training sample is a second combination, the positive sample label indicates incorrect reasoning, and the negative sample label indicates correct reasoning. The fine-tuned model is trained based on the second training set to obtain the logical reasoning model.

[0005] Secondly, embodiments of this application provide a model training apparatus, including: The first training set acquisition module is used to acquire the first training set, which includes multiple training data. Each training data includes a first training sample and a corresponding first label. Each first training sample is a piece of text content, and each first label includes the reasoning result obtained based on the corresponding text content. The first model training module is used to train the initial model based on the first training set to obtain the fine-tuned model, and to obtain the error output of the initial model for each first training sample during the training process; wherein, the error output is the output of the initial model for each first training sample that is different from the corresponding first label. The second training set acquisition module is used to acquire a second training set, which includes multiple second training samples and positive and negative sample labels corresponding to each second training sample. A second training sample can be a first training sample, a first combination consisting of a first training sample and its corresponding first label, or a second combination consisting of a first training sample and its corresponding error output. When the second training sample is a first training sample, the positive sample label is the first label corresponding to that first training sample, and the negative sample label is the error output corresponding to that first training sample. When the second training sample is a first combination, the positive sample label indicates correct reasoning, and the negative sample label indicates incorrect reasoning. When the second training sample is a second combination, the positive sample label indicates incorrect reasoning, and the negative sample label indicates correct reasoning. The second model training module is used to train the fine-tuned model based on the second training set to obtain the logical reasoning model.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory; The processor executes a computer program to implement the method provided in the first aspect embodiment or any alternative embodiment of the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the first aspect embodiment or any optional embodiment of the first aspect.

[0008] The beneficial effects of the technical solutions provided in this application are: First, the solution provided in this application provides sufficient training data for the model training process by obtaining a first training set containing text content and corresponding inference results, and a second training set containing multiple text contents and positive and negative sample labels about the inference results. Secondly, in this embodiment, the initial model is trained using a first training set, enabling the model to acquire preliminary reasoning ability. Then, the model is further enhanced through a second training set, thereby improving the model's reasoning ability and increasing the reasoning accuracy. This application implements a training method for a model, which solves the problems of single training mode, one-sided development of model capabilities, weak model thinking ability, and weak generalization ability by providing sufficient and diverse training data. This enables the trained model to have strong generalization ability, adapt to the changes of different types of input data in application scenarios, and improve the applicability, generalization and effect of the model. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0010] Figure 1 A schematic flowchart illustrating a model training method provided in an embodiment of this application; Figure 2 This is an example diagram of a prompt message in one embodiment of this application; Figure 3 This is an example diagram of a thought chain generation paradigm in one embodiment of this application; Figure 4 This is a schematic diagram of the overall process of a model training method in one example of an embodiment of this application; Figure 5 A structural block diagram of a model training device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0012] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0014] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0015] Figure 1 This application provides a flowchart illustrating a model training method, the execution subject of which can be a terminal (e.g., a computer, mobile phone, etc.). Figure 1 As shown, the method may include: Step S101: Obtain the first training set, which includes multiple training data. Each training data includes a first training sample and a corresponding first label. Each first training sample is a piece of text content, and each first label includes a reasoning result obtained based on the corresponding text content.

[0016] In an embodiment of this application, the first training sample can be a descriptive text, such as "On the 30th of this month, Li Si and I went on a business trip to Kunming for a visit and study, by plane, for four days, and then went to Guiyang for project exchange." The inference result for the corresponding first label can be the content inferred from this description, such as "Business trip date is the 30th, business trip destination is Kunming, business trip purpose is visit and study and project exchange, business trip duration is four days, business trip method is by plane, current time is May 27, 2024, business trip date is May 30, 2024, ..., If the business trip ends on June 2nd, then the corresponding first label is "{"Travel Purpose":"Visit, study and project exchange", "Travel Information":[{'Traveler':'I','Travel Mode:'Airplane','Travel Type:'One-way','Destination:'ArrivalCity':'Kunming','Travel Date:'2024-05-30','End Date:'2024-06-02'}]}". Optionally, the first label in this embodiment may also include the specific process of obtaining the reasoning result based on the first training sample.

[0017] Specifically, while some well-trained large language models with powerful and comprehensive logical reasoning capabilities already exist on the market, their deployment often consumes significant resources. In many application scenarios, sufficient resources cannot be provided for these large language models, and in most cases, users' daily needs are not particularly high, making the deployment of these large language models a waste of resources. Therefore, the purpose of this application is to train a model that consumes fewer resources but possesses strong logical reasoning capabilities. Because of its low resource consumption, this model can be deployed in more scenarios and can also meet the daily needs of general users. Before proceeding, a first training set for training the model needs to be obtained. The first training set contains multiple pre-collected training data. Each training data contains a first training sample and a corresponding first label. The first training sample can be a piece of language text organized by the trainer, or a document found on the Internet, etc. This application embodiment does not limit this. The first label can be the reasoning result inferred by the trainer based on the corresponding first training sample, or it can be the thinking logic and reasoning result inferred from the first training sample after inputting the first training sample into other pre-trained large language models, etc., which are used as the reasoning result of the first label. This application embodiment does not limit this.

[0018] Step S102: Train the initial model based on the first training set to obtain the fine-tuned model, and obtain the error output of the initial model for each first training sample during the training process; wherein, the error output is the output of the initial model for each first training sample that is different from the corresponding first label.

[0019] In the embodiments of this application, the initial model may be a model that has not been trained in a specific application scenario, and may lack any logical reasoning ability or possess only a weak logical reasoning ability. The fine-tuned model may be a model obtained by training the initial model on a first training set, and this model possesses a certain logical reasoning ability. The error output may be the erroneous reasoning results output by the model during the training of the initial model. These results can be used for further reinforcement training of the model to avoid the model outputting the same erroneous reasoning results again. Optionally, the error output may also be output by the fine-tuned model obtained after the initial model has been trained, or it may be user-defined. This embodiment of the application does not limit this.

[0020] Specifically, after obtaining the first training set, the initial model can be trained using the first training set. After training, a fine-tuned model with a certain logical reasoning ability in a specific application scenario can be obtained. At the same time, error outputs generated by the model during training that are different from the first label are recorded for use in the next reinforcement training. Optionally, all outputs with differences greater than the set threshold can be recorded as error outputs by setting a threshold. This embodiment of the application does not limit this.

[0021] Step S103: Obtain a second training set. The second training set includes multiple second training samples and positive and negative sample labels corresponding to each second training sample. A second training sample is a first training sample, a first combination consisting of a first training sample and its corresponding first label, or a second combination consisting of a first training sample and its corresponding error output. When the second training sample is a first training sample, the positive sample label is the first label corresponding to the first training sample, and the negative sample label is the error output corresponding to the first training sample. When the second training sample is a first combination, the positive sample label represents correct reasoning, and the negative sample label represents incorrect reasoning. When the second training sample is a second combination, the positive sample label represents incorrect reasoning, and the negative sample label represents correct reasoning.

[0022] Understandably, when the second training sample is a first training label (i.e., a piece of text content), the model's output for that second training sample should be the inference result for that first training label. Therefore, a positive sample label should be the correct inference result (i.e., the first label), and a negative sample label should be the incorrect inference result (i.e., the incorrect output). Optionally, the negative sample label could also be an incorrect output that the model might generate based on its own training experience. When the second training sample is the first combination of the first training sample and the corresponding first label, the model's output for the first combination should be the judgment result of "whether the first label is the correct inference result of the first training sample." Since the first label is the true inference result of the first training sample, a positive sample label should represent correct inference, and a negative sample label should represent incorrect inference. Conversely, when the second training sample is the first training sample and the corresponding incorrect output, similarly, a positive sample label should represent incorrect inference, and a negative sample label should represent correct inference.

[0023] Specifically, the fine-tuned model already possesses a certain level of logical reasoning ability. However, to further improve the model's logical reasoning ability and the accuracy of the generated results, making the generated reasoning results more in line with human preferences, this embodiment of the application requires further reinforcement learning training on the fine-tuned model. Before this, a second training set for reinforcement learning training needs to be obtained. The purpose of reinforcement training is mainly to enable the model to identify possible erroneous outputs during training, so as to avoid the model generating the same erroneous outputs again. Therefore, a positive and negative sample training method will be adopted, that is, each second training sample in the second training set corresponds to two completely opposite labels. These labels can be obtained directly from the first training set, or from the erroneous outputs of the initial model during the training of the initial model, etc. After obtaining the labels, each training sample and its corresponding label will generate a training data set, and these training data sets will then constitute the second training set.

[0024] This application proposes an innovative reinforcement learning training method to enhance the logical reasoning ability of large models. It proposes a strategy for constructing three types of reinforcement training learning data, which enables the model to focus more on optimizing the correctness of core reasoning results and final answers during the training process.

[0025] Step S104: Train the fine-tuned model based on the second training set to obtain the logical reasoning model.

[0026] In the embodiments of this application, the logical reasoning model is the model that is to be applied and deployed in a real-world scenario after training.

[0027] Specifically, after obtaining the second training set, further reinforcement training can be carried out on the fine-tuned model using the second training set. After completing the training, a logical reasoning model that can be applied to more scenarios can be obtained.

[0028] The solution provided in this application, firstly, provides sufficient training data for the model training process by obtaining a first training set containing text content and corresponding reasoning results, and a second training set containing multiple text contents and positive and negative sample labels about the reasoning results; Secondly, in this embodiment, the initial model is trained using a first training set, enabling the model to acquire preliminary reasoning ability. Then, the model is further enhanced through a second training set, thereby improving the model's reasoning ability and increasing the reasoning accuracy. This application implements a training method for a model, which solves the problems of single training mode, one-sided model capability development, weak model reasoning ability, and weak generalization ability by providing sufficient and diverse training data. This results in a trained model with strong generalization ability, capable of adapting to different types of input data in application scenarios, thus improving the model's applicability, generalization, and performance. When supervised labeled data is limited, it can quickly enhance the logical reasoning ability of a large language model in generating "fast thinking" and "slow thinking" responses. When supervised labeled data is relatively abundant, it can further enhance the model's reasoning depth and generation accuracy, improving the reliability and accuracy of fast and slow thinking generation results.

[0029] Based on the above embodiments, as an optional embodiment, the fine-tuned model is trained on a second training set to obtain a logical reasoning model, specifically including: Each second training sample in the second training set is sequentially input into the fine-tuned model for training until the training termination condition is met, thus obtaining the logical reasoning model. Each round of training includes: Obtain the second training sample for the current training round, input the second training sample and the corresponding positive sample label into the model corresponding to this round of training, so that the model generates a first bias of positive sample label, and input the second training sample and the corresponding negative sample label into the model corresponding to this round of training, so that the model generates a second bias of negative sample label; wherein, the model corresponding to the first round of training is the fine-tuned model, and the bias is used to characterize the degree of tendency of the model to generate the corresponding sample label; The first and second propensity values ​​are compared to determine the first loss value of the represented model in terms of propensity, and the second loss value of the represented model in terms of inference accuracy is determined by a preset loss function. The first loss value and the second loss value are weighted and summed according to preset weights to obtain the total loss value of the model. The model parameters are adjusted based on the total loss value, and the model with adjusted parameters is used as the model for the next round of training.

[0030] In the embodiments of this application, the propensity can be used to characterize the degree of tendency of the model to generate the corresponding label. The higher the propensity, the easier it is for the model to generate the corresponding label, and vice versa. The propensity calculation results of the two types of sample labels reflect the relative probability of the model generating the target keyword in the positive sample label and the negative sample label based on the input. If the propensity of the positive sample label is k, it means that the probability of the large language model generating the positive sample label is k times the probability of it not generating the sample label. The loss value can be used to characterize the accuracy of the model output after each round of training. The smaller the loss value, the more accurate the output generated by the model, and vice versa. The termination condition can be the total number of training rounds set in advance, or the loss value calculated after a certain round of training is less than a certain threshold, or the output reaches the highest accuracy in a certain dataset / validation set, etc., which are not limited in the embodiments of this application.

[0031] Specifically, during reinforcement training, one or more second training samples can be input into the model corresponding to each training round. After receiving the second training samples, before outputting the result, the model will calculate the first bias towards positive sample labels and the second bias towards negative sample labels, as shown in the following formula: ; ; In the above formula, That is, the first tendency. That is, the second tendency. It can be the probability value of generating positive sample labels (the first probability value). It can be the probability value of generating negative sample labels (second probability value).

[0032] Then, the first and second propensities can be compared to obtain the first loss value in terms of propensity. The specific formula can be found in the following formula: ; in, That is, the first loss value. It can be a Sigmoid (an activation function) function, which makes the calculation result fall within a preset range (in this embodiment, the range can be [0,1]).

[0033] Additionally, the model can determine a second loss value for inference accuracy by using a preset loss function. The specific calculation method will be introduced in the following text.

[0034] After obtaining the first and second loss values, a weighted sum can be calculated based on the actual needs to determine the total loss value of the model obtained in this round of training. The specific formula is as follows: ; in, That is, the total loss value. This indicates the expected calculation. That is, the weight is preset. In this embodiment of the application, only the first loss value is weighted. In actual application scenarios, the second loss value can also be further weighted according to actual needs. This embodiment of the application does not limit this.

[0035] After determining the total loss value, the model parameters of the model trained in this round are adjusted according to the total loss value. After the adjustment is completed, the model corresponding to the next round of training is obtained. If it is found that the termination condition of this training is reached after a certain training is completed, it means that the reinforcement of the model is close to saturation. At this time, the training can be terminated and the trained logical reasoning model is obtained.

[0036] It should be noted that, unlike typical training, the training in this embodiment does not wait for the model to generate complete text content before comparing its output with the corresponding labels to adjust the model parameters. Instead, the model continuously predicts the probability distribution of the next character based on known contextual information and continuously compares this prediction with the true labels to obtain a loss value. Finally, after the model completes the prediction of the last character, the total prediction loss for the entire sample is calculated by averaging or summing the loss values ​​at each character position. The model parameters are then adjusted based on this total prediction loss.

[0037] Based on the above embodiments, as an optional embodiment, the first tendency is calculated in the following manner: For each second training sample, whenever the model generates any character in the first output, the character position of any character in the first output is determined, and a first probability value is determined based on the second training sample and the corresponding positive sample label that any character is the same as the character at the corresponding character position in the positive sample label; wherein, the first output is the inference result output by the model; For each character position, if the characters at the corresponding character positions in the positive and negative sample labels are the same, then the weight of the character position is set to the first value; if they are different, then the weight of the character position is set to the second value. The positive and negative sample labels contain the same number of characters, and the first value is less than the second value. Based on the weights of each character position in the first output, the first probability values ​​of each character position are weighted and summed to obtain the second probability value that the first output is a positive sample label; The first propensity is determined based on the second probability value.

[0038] In the embodiments of this application, the probability value can characterize the probability that the generated character is the same as or matches the character at the corresponding position in the corresponding label before the model outputs each character.

[0039] Specifically, when each second training sample is input into the model, the model will generate the corresponding output based on the text content of the second training sample. Since the model trained in this embodiment is essentially a large language model, its output will be output one character at a time. In this process, before the model outputs a character, it will calculate the probability that the character is the same as the character in the corresponding label (the positive sample label is used as an example here, and the calculation method for the negative sample label is the same, so it will not be elaborated further). For example, if the model is about to output the third character, then the probability that the third character that the model is about to output is the same as the third character in the corresponding sample label is calculated here.

[0040] At the same time, it is understandable that the actual semantic impact of characters in different positions varies. (For example, in the sentences "I'm going to eat" and "I'm going to go out," although the words "go to eat" and "go out" are different, their semantic impact is small, as both express "about to do something." However, the words "eat" and "go out" are not only different in word form but also in meaning, expressing different "things about to be done.") Therefore, for each character in a different position, different weights need to be set according to their degree of semantic impact. The impact of different character positions on semantics can be determined by directly comparing positive and negative sample labels, or by using a word vector encoding model. For character positions with the same characters or similar vectors, it indicates that these character positions have little impact on semantics and can be assigned smaller weights (i.e., the first value). For character positions with different characters, it indicates that they have a greater impact on semantics and can be assigned larger weights (i.e., the second value).

[0041] After determining the weight of each character position and the probability that each character generated by the model is the same as the character at the corresponding position in the corresponding label, a second probability value for generating the corresponding sample can be obtained by weighted summing of these probability values. See the following formula for details: ; ; in, This represents the second probability value when the corresponding label is a positive sample label. This represents the second probability value when the corresponding label is a negative sample label. and These represent the weights of each character position in the positive and negative sample labels, respectively, where T represents the total number of characters in the sample label. This represents the first probability value that each character is the same as the character at the corresponding position in the positive sample label. This represents the first probability value that each character is the same as the character at the corresponding position in the negative sample label.

[0042] Then, the first tendency of the model to generate corresponding labels based on each second training sample can be calculated according to the formula described in the whole text.

[0043] It should be noted that in practical applications, the number of characters included in the actual positive sample label and the actual negative sample label are often not exactly the same. In this case, for the convenience of calculation, we can first compare the characters contained in the positive sample label and the negative sample label, and then add empty characters to the side with fewer characters at the corresponding character positions that do not affect the semantics, until the number of characters on the side with fewer characters is the same as the number of characters included on the other side after adding empty characters. It can be understood that the weight value of the character position where the empty character is located can be set to the first value to reduce the impact of the empty character.

[0044] Based on the above embodiments, as an optional embodiment, a first probability value is determined based on the second training samples and the corresponding positive sample labels, which is the same as the character at the corresponding character position in the positive sample labels. Specifically, this includes: Retrieve a preset vocabulary, which includes multiple preset characters; The third combination is obtained by combining the characters before the character position in the first output with the second training sample; The third combination is converted into the corresponding text vector, and the second probability value of each preset character in the preset vocabulary is calculated based on the neural network parameters of the model and the text vector. The second probability value of the preset character that is the same as the character at the corresponding character position in the positive sample label is used as the first probability value.

[0045] In an embodiment of the present application, the preset vocabulary can be a vocabulary including characters that can be output by multiple models, and each preset character is one of the characters that can be output by the model. The text vector can be obtained by methods such as word embedding, and the embodiments of the present application do not limit this here.

[0046] Specifically, when the model in the embodiment of the present application outputs a character, it often selects a character from its own preset vocabulary for output. In the process of selecting this character, the probability value of each character being output at this character position needs to be calculated first, and the probability value needs to be determined by combining the characters before this character position in the corresponding sample label and the second training sample. Specifically, when a character at a certain character position needs to be output, other characters before the corresponding character position can be combined with the characters in the second training sample to obtain a third combination. Then, the third combination is transformed into the form of a corresponding text vector, and then the text vector is calculated through the neural network parameters of the model to calculate the generation probability value (i.e., the second probability value) of each preset character in the preset vocabulary. Then, the actual character at this character position in the positive sample label can be obtained, and the probability value corresponding to this character in the preset vocabulary is found as the first probability value.

[0047] For example, if the model is now going to generate the 5th character, then the model will combine the first 4 characters in the first output with the second training sample to obtain multiple third combinations, and then transform the third combinations into the corresponding text vectors, and calculate the probability value of each preset character in the preset vocabulary being the 5th character based on the neural network parameters of the model and the text vectors of the third combinations.

[0048] After calculating the probability value of each preset character being the 5th character, since it is now generating the 5th character, then it will look for what the 5th character is in the positive sample label. For example, if the 5th character in the positive sample label is the character "我" now, then find the character "我" in the preset vocabulary and query the probability value of the preset character "我" being the 5th character as the first probability value.

[0049] After understanding the above concepts, the second loss value described above can be calculated through the following formula. The specific formula is as follows: ; where, That is, the second loss value, is the one-hot encoding of the true label, which is 1 when the generated character is the same as the character at the corresponding position in the corresponding label, and 0 otherwise. is the probability value of the model generating the same character, N is the number of characters included in the sample label, and M is the number of characters in the preset vocabulary.

[0050] It should be noted that, as described above, the calculation method for the first probability value of the negative sample label is the same as that for the positive sample label, and will not be repeated here in the embodiments of this application.

[0051] Based on the above embodiments, as an optional embodiment, determining the first tendency based on the second probability value specifically includes: A first tendency is determined based on the degree of deviation of the second probability value from the preset value. When the second probability value is greater than the preset value, the first tendency is directly proportional to the degree of deviation; when the second probability value is not greater than the preset value, the first tendency is inversely proportional to the degree of deviation.

[0052] In the embodiments of this application, the preset value can be 0.5, and the degree of deviation can characterize the difference between the second probability value and the preset value. Specifically, in the embodiments of this application, as described in the formula above, the first tendency is actually the ratio of the probability of generating the corresponding sample label to the probability of not generating the corresponding sample label. When the two probabilities are the same, the ratio is 1. When the second probability value is not greater than the preset value, the first tendency will be smaller as the degree of deviation increases. When the second probability value is greater than the preset value, the first tendency will be larger as the degree of deviation increases.

[0053] Based on the above embodiments, as an optional embodiment, the first training set is obtained in the following way: Obtain multiple first training samples and a second label corresponding to each first training sample; each first training sample is a piece of text content, and each second label includes the reasoning result obtained based on the text content. For each first training sample, semantic analysis is performed on the first training sample and the corresponding second label to obtain the reasoning logic based on the text content and reasoning results. Prompt information is generated based on the reasoning logic. The reasoning logic includes multiple reasoning steps, the reasoning target corresponding to each reasoning step, and the order between the reasoning steps. For each first training sample, the first training sample, the corresponding second label, and the prompt information are input into a preset large language model to obtain a second output; wherein, the second output includes the thinking logic obtained by the large language model from the first training sample based on the reasoning logic, and the reasoning result; the thinking logic includes the reasoning process of obtaining the corresponding reasoning target from each reasoning step; For each first training sample, the second output corresponding to the first training sample is used as the third label of the first training sample. First training data is generated based on the first training sample and the corresponding second label, and second training data is generated based on the first training sample and the corresponding third label. The first label includes the second label and the third label. The first training set is generated based on each first training data and each second training data.

[0054] In the embodiments of this application, the second label may be a reasoning result obtained by the trainer from the semantics of the first training sample, provided simultaneously with the first training sample. The third label may be the thought logic and reasoning result output after the first training sample is input into a preset large language model. It should be noted that the second label often does not include the thought logic part.

[0055] Reasoning logic can include the goal of each reasoning step and the order of the steps, but not the specific process of how the goal is reached. Thinking logic, on the other hand, includes the specific reasoning process used to arrive at the goal within each step. The reasoning result only includes the final conclusion and does not include any content related to the reasoning process.

[0056] Specifically, the training provider first provides a set of initial training samples and a second label for each of these initial training samples. This training data only includes the text content to be reasoned (i.e., the initial training samples) and the corresponding reasoning results (i.e., the second labels). Training the initial model using only this data only establishes a preliminary direct mapping relationship between input (question) and output (answer). However, this training method fails to explicitly guide the model to understand and generate intermediate reasoning steps, resulting in low generalization ability for the model's "fast thinking" in specific application scenarios. Furthermore, if the model overfits the training data during training, its "fast thinking" results in specific application scenarios often become incorrect. To address these shortcomings, this embodiment of the application "expands" this portion of the training data. Specifically, these initial training samples are input into a pre-defined large language model (such as Deepseak, a model with strong logical reasoning capabilities), allowing the pre-defined large language model to output its thinking logic and reasoning results regarding the initial training samples. In this process, to ensure that the content output by the pre-set large language model can be easily used as training data for subsequent training, the reasoning logic that the pre-set large language model needs to consider can also be input as prompt information when the first training sample is input into the pre-set large language model. An example of the prompt information could be as follows: Figure 2 As shown (the "Prompt" message in the image), reasoning logic can be a paradigm for generating thought chains. This paradigm includes multiple pre-defined reasoning steps required by a large language model, the reasoning objective for each step, and the order of these steps. A specific paradigm for generating thought chains can be shown below. Figure 3As shown in the diagram, Step 1, Step 2, and Step 3 refer to the reasoning steps. The key reasoning methods in the diagram include the reasoning objectives corresponding to each reasoning step, such as travelReason (reason for business trip), arrivalCity (arrival at destination), tr_startDate (departure time), etc.

[0057] After inputting the prompts and each first training sample into the preset large language model, the preset large language model can obtain a second output based on the above content. The second output includes the detailed reasoning logic of the first training sample and the reasoning result obtained from the reasoning. Since the output format of the preset large language model has been standardized through the thought chain generation paradigm, this second output can be directly used as the third label of the first training sample. At this time, the first training sample has both a second label and a third label. Since these two labels emphasize different content (the second label emphasizes the rapid output of reasoning results, while the third label emphasizes the output of more detailed reasoning logic and reasoning results), they need to be used as different training data. That is, each first training label and its corresponding second label can be used together as the first training data, and each first training label and its corresponding third label can be used together as the second training data. Then, the first training data and the second training data are integrated to obtain the first training set.

[0058] The first training set obtained through the above method includes training data for both rapid reasoning and outputting reasoning results, as well as training data for outputting reasoning logic and results after detailed consideration. This allows the trained model to possess both capabilities simultaneously, making it adaptable to a wider range of application scenarios. The model is trained not only to predict the final answer label (corresponding to "fast thinking" output) but also to learn to generate and understand the sequence of intermediate reasoning steps (corresponding to the "slow thinking" process). This innovative multi-task learning method allows the model to optimize both its ability to predict rapid answer output and its ability to generate complete reasoning chains within the same training loop. In this way, the "fast thinking" and "slow thinking" mechanisms within the model are trained collaboratively and reinforce each other. Ultimately, this enables the model to flexibly select either thinking mode for content generation based on the user's specific needs or context, thereby comprehensively improving its adaptability and performance in different reasoning scenarios.

[0059] Based on the above embodiments, as an optional embodiment, the initial model is trained on the first training set to obtain a fine-tuned model, specifically including: Each first training sample in the first training set is sequentially input into the initial model for training until the training termination condition is met, resulting in a fine-tuned model. Each round of training includes: Obtain the first training sample of the current training round, and input the first training sample into the model corresponding to this round, so that the model determines the third probability value that each character in the generated third output is the same as the character at the corresponding position in the first label of the first training sample; The model parameters of the model corresponding to this round of training are adjusted based on the third probability value, and the model with adjusted model parameters is used as the model for the next round of training.

[0060] Specifically, after obtaining the first training set, the training steps for the initial model can be started. Each training round will use one training data set, that is, the first training sample from a certain training data set can be input into the model corresponding to each training round. In each training round, the model will predict the probability value of each character in the third output being the same as the character at the position of the first label corresponding to the first training sample, as described above. Then, the model parameters are adjusted according to the probability value. When the training termination condition is met in a certain training round, the training operation can be terminated, and the fine-tuned model can be obtained.

[0061] The following is combined Figure 4 The overall process of the model training method provided in the embodiments of this application will be introduced. Figure 4 This is a schematic diagram of the overall process of a model training method provided in an embodiment of this application, as shown below. Figure 4 As shown, the model training method provided in this application can be divided into three main steps: S1, data augmentation driven by a large model (i.e., the model to be trained in this application); S2, supervised fine-tuning training / SFT fine-tuning training of the large model; and S3, reinforcement learning to enhance the logical reasoning ability of the large model. In step S1, firstly, in step S1.1, the trainer obtains supervised data, namely the first training sample and the second label corresponding to each first training sample. Then, in step S1.2, the trainer summarizes the reasoning method (i.e., reasoning logic) for inferring the second label from the first training sample based on each first training sample and the corresponding second label. Next, in step S1.3, the corresponding thought chain generation paradigm is constructed. Then, in step S1.4, prompt information about the preset large language model is generated and input into the preset large language model along with the first training sample. Finally, in step S1.5, the preset large language model outputs the corresponding thinking logic and reasoning results. This data corresponds to the "slow thinking data" containing the thought chain process in the diagram. The introduction of "slow-thinking data" aims to enable large language models to learn a logical thinking approach during training, deriving correct results through step-by-step reasoning. Therefore, "slow-thinking data," through its meticulously designed thought chain, provides large language models with a clear logical reasoning training path, compensating for their shortcomings in complex reasoning tasks.

[0062] Next, step S2 begins. In step S2, the supervised data from step S1.1 (i.e., the first training data) and the "slow-thinking data" obtained at the end of step S1 (i.e., the second training data) are first merged. Then, step S2.2 performs supervised fine-tuning (i.e., the training process) on the initial large model's parameters, resulting in the fine-tuned large model. During this process, the same user input is learned by the large language model through both "fast thinking" and "slow thinking" methods. Slow-thinking training helps the large language model more deeply understand how to extract key information from user input and generate results through step-by-step reasoning based on this information. This enhanced training of the thinking process not only improves the large language model's logical reasoning ability but also feeds back into fast-thinking training, enabling fast thinking to more efficiently connect key information in user input with the final answer. Under this dual training effect, fast thinking and slow thinking complement each other, ultimately allowing the trained large language model to quickly focus on the core information in user input to generate accurate results, and also to demonstrate a complete logical reasoning process when needed, thus simultaneously improving the large language model's fast and slow thinking generation capabilities.

[0063] Next, step S3 begins. In step S3, reinforcement learning data is first constructed in step S3.1. The data constructed in this process constitutes the second training set. The "reasoning result correctness judgment data" refers to the first or second combination mentioned earlier, the "fast thinking reinforcement learning data" is the first training data, and the "slow thinking reinforcement learning data" is the second training data. These three types of reinforcement learning data are constructed to allow the model to focus more on optimizing the correctness of core reasoning results and the final answer during training. After obtaining the second training set, step S3.2 begins to further reinforce the fine-tuned model. After reinforcement learning is completed, the trained logical reasoning model is obtained. By introducing the above data for reinforcement learning in stage S3, the large model can be provided with an overview of the input and correct or incorrect output samples, providing a large comparative perspective of positive and negative samples and deepening the model's understanding of the differences between them. In particular, through explicit "no" labels, the model can identify which reasoning results are unacceptable, thereby reducing the occurrence of similar errors in future predictions and significantly reducing the correlation between input and incorrect output results.

[0064] The purpose of this design is to enable large language models to recognize that simply generating superficially reasonable thought chains is insufficient when performing reinforcement learning. The model needs to understand that if the key reasoning logic and key reasoning results are incorrect, even if the thought chain appears logical, the final result will still be wrong, and the model will receive negative reinforcement feedback. Conversely, only by generating correct reasoning processes, reasoning logic, and correct final reasoning results can significant positive reinforcement learning feedback be obtained.

[0065] Meanwhile, the core objective of using error outputs as training data for reinforcement learning training is to allow the model to learn and improve from its own shortcomings, thereby enhancing its logical reasoning ability and the accuracy of its generated results. Although the model, after supervised fine-tuning, already possesses the ability to generate reasoning results according to prompts, it may still make reasoning errors in certain details, especially when faced with subtle differences in user input. Therefore, by systematically collecting the prediction biases of the supervised fine-tuned model at these key logical points, we construct fast-thinking reinforcement learning data, enabling the model to focus more on understanding and correcting these logical reasoning errors during reinforcement learning training.

[0066] Through innovative reinforcement learning data construction and training methods, the model focuses more on optimizing the correctness of core reasoning results and the final answer during training. The model prioritizes how to derive correct generated results from user input, thus producing more accurate and user-preferred outputs in both fast and slow thinking scenarios. Ultimately, this enables the trained model to maintain stable output under high response speed requirements (fast thinking) and generate accurate, logical, and user-preferred results in scenarios requiring in-depth analysis (slow thinking).

[0067] Figure 5 A structural block diagram of a model training device provided in an embodiment of this application is shown below. Figure 5 As shown, the model training device 500 may include: a first training set acquisition module 501, a first model training module 502, a second training set acquisition module 503, and a second model training module 504, wherein, The first training set acquisition module 501 is used to acquire a first training set, which includes multiple training data. Each training data includes a first training sample and a corresponding first label. Each first training sample is a piece of text content, and each first label includes a reasoning result obtained based on the corresponding text content. The first model training module 502 is used to train the initial model based on the first training set to obtain the fine-tuned model, and to obtain the error output of the initial model for each first training sample during the training process; wherein, the error output is the output of the initial model for each first training sample that is different from the corresponding first label. The second training set acquisition module 503 is used to acquire a second training set, which includes multiple second training samples and positive and negative sample labels corresponding to each second training sample. A second training sample can be a first training sample, a first combination consisting of a first training sample and its corresponding first label, or a second combination consisting of a first training sample and its corresponding error output. When the second training sample is a first training sample, the positive sample label is the first label corresponding to that first training sample, and the negative sample label is the error output corresponding to that first training sample. When the second training sample is a first combination, the positive sample label indicates correct reasoning, and the negative sample label indicates incorrect reasoning. When the second training sample is a second combination, the positive sample label indicates incorrect reasoning, and the negative sample label indicates correct reasoning. The second model training module 504 is used to train the fine-tuned model based on the second training set to obtain the logical reasoning model.

[0068] The solution provided in this application, firstly, provides sufficient training data for the model training process by obtaining a first training set containing text content and corresponding reasoning results, and a second training set containing multiple text contents and positive and negative sample labels about the reasoning results; Secondly, in this embodiment, the initial model is trained using a first training set, enabling the model to acquire preliminary reasoning ability. Then, the model is further enhanced through a second training set, thereby improving the model's reasoning ability and increasing the reasoning accuracy. This application implements a model training method that addresses the problems of single training modes, one-sided model capability development, weak model thinking ability, and weak generalization ability by providing sufficient and diverse training data. This results in a trained model with strong generalization ability, capable of adapting to different types of input data in application scenarios, thus improving the model's applicability, generalization, and performance. Compared to existing "supervised fine-tuning methods," this application innovatively incorporates a large-model-driven data augmentation module before model training, solving the challenges of obtaining high-quality training data and requiring large amounts of data for effective training.

[0069] Based on the above embodiments, as an optional embodiment, the second model training module is specifically used for: Each second training sample in the second training set is sequentially input into the fine-tuned model for training until the training termination condition is met, thus obtaining the logical reasoning model. Each round of training includes: Obtain the second training sample for the current training round, input the second training sample and the corresponding positive sample label into the model corresponding to this round of training, so that the model generates a first bias of positive sample label, and input the second training sample and the corresponding negative sample label into the model corresponding to this round of training, so that the model generates a second bias of negative sample label; wherein, the model corresponding to the first round of training is the fine-tuned model, and the bias is used to characterize the degree of tendency of the model to generate the corresponding sample label; The first and second propensity values ​​are compared to determine the first loss value of the represented model in terms of propensity, and the second loss value of the represented model in terms of inference accuracy is determined by a preset loss function. The first loss value and the second loss value are weighted and summed according to preset weights to obtain the total loss value of the model. The model parameters are adjusted based on the total loss value, and the model with adjusted parameters is used as the model for the next round of training.

[0070] Based on the above embodiments, as an optional embodiment, the second model training module is further used for: For each second training sample, whenever the model generates any character in the first output, the character position of any character in the first output is determined, and a first probability value is determined based on the second training sample and the corresponding positive sample label that any character is the same as the character at the corresponding character position in the positive sample label; wherein, the first output is the inference result output by the model; For each character position, if the characters at the corresponding character positions in the positive and negative sample labels are the same, then the weight of the character position is set to the first value; if they are different, then the weight of the character position is set to the second value. The positive and negative sample labels contain the same number of characters, and the first value is less than the second value. Based on the weights of each character position in the first output, the first probability values ​​of each character position are weighted and summed to obtain the second probability value that the first output is a positive sample label; The first propensity is determined based on the second probability value.

[0071] Based on the above embodiments, as an optional embodiment, the second model training module can also be used for: Obtain a preset vocabulary, which includes multiple first preset characters; Each first preset character in the preset vocabulary is combined with the other characters before the corresponding character position in the positive sample label to obtain each third combination; The third combination is obtained by combining the characters before the character position in the first output with the second training sample; The third combination is converted into the corresponding text vector, and the second probability value of each preset character in the preset vocabulary is calculated based on the neural network parameters of the model and the text vector. The second probability value of the preset character that is the same as the character at the corresponding character position in the positive sample label is used as the first probability value.

[0072] Based on the above embodiments, as an optional embodiment, the second model training module can also be used for: A first tendency is determined based on the degree of deviation of the second probability value from the preset value. When the second probability value is greater than the preset value, the first tendency is directly proportional to the degree of deviation; when the second probability value is not greater than the preset value, the first tendency is inversely proportional to the degree of deviation.

[0073] Based on the above embodiments, as an optional embodiment, the first training set acquisition module is specifically used for: Obtain multiple first training samples and a second label corresponding to each first training sample; each first training sample is a piece of text content, and each second label includes the reasoning result obtained based on the text content. For each first training sample, semantic analysis is performed on the first training sample and the corresponding second label to obtain the reasoning logic based on the text content and reasoning results. Prompt information is generated based on the reasoning logic. The reasoning logic includes multiple reasoning steps, the reasoning target corresponding to each reasoning step, and the order between the reasoning steps. For each first training sample, the first training sample, the corresponding second label, and the prompt information are input into a preset large language model to obtain a second output; wherein, the second output includes the thinking logic obtained by the large language model from the first training sample based on the reasoning logic, and the reasoning result; the thinking logic includes the reasoning process of obtaining the corresponding reasoning target from each reasoning step; For each first training sample, the second output corresponding to the first training sample is used as the third label of the first training sample. First training data is generated based on the first training sample and the corresponding second label, and second training data is generated based on the first training sample and the corresponding third label. The first label includes the second label and the third label. The first training set is generated based on each first training data and each second training data.

[0074] Based on the above embodiments, as an optional embodiment, the first model training module is specifically used for: Each first training sample in the first training set is sequentially input into the initial model for training until the training termination condition is met, resulting in a fine-tuned model. Each round of training includes: Obtain the first training sample of the current training round and input the first training sample into the model corresponding to this round so that the model determines the third probability value that each character in the generated third output is the same as the character at the corresponding position in the first label of the first training sample; wherein, the model corresponding to the first round of training is the initial model; The model parameters of the model corresponding to this round of training are adjusted based on the third probability value, and the model with adjusted model parameters is used as the model for the next round of training.

[0075] The following is for reference. Figure 6 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., performing...). Figure 1 The diagram shows the structure of the terminal device or server 600 of the method shown. The electronic devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (e.g., vehicle navigation terminals), wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0076] The electronic device includes a memory and a processor. The memory stores a program for executing the methods described in the various method embodiments above. The processor is configured to execute the program stored in the memory. The processor may be referred to as processing device 601 as described below. The memory may include at least one of read-only memory (ROM) 602, random access memory (RAM) 603, and storage device 608 as described below, as follows: like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0077] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0078] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 609, or installed from storage device 608, or installed from ROM 602. When the computer program is executed by processing device 601, it performs the functions defined in the methods of embodiments of this application.

[0079] It should be noted that the computer-readable storage medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0080] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0081] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0082] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: Obtain a first training set, which includes multiple training data sets. Each training data set includes a first training sample and a corresponding first label. Each first training sample is a piece of text content, and each first label includes a reasoning result obtained based on the corresponding text content. Train the initial model based on the first training set to obtain a fine-tuned model, and obtain the error outputs of the initial model for each first training sample during the training process. The error outputs are those outputs of the initial model for each first training sample that differ from the corresponding first label. Obtain a second training set, which includes multiple second training samples and a positive sample label and a negative sample label for each second training sample. The label, the second training sample is a first training sample, a first combination consisting of a first training sample and its corresponding first label, or a second combination consisting of a first training sample and its corresponding error output; when the second training sample is a first training sample, the positive sample label is the first label corresponding to the first training sample, and the negative sample label is the error output corresponding to the first training sample; when the second training sample is the first combination, the positive sample label represents correct reasoning, and the negative sample label represents incorrect reasoning; when the second training sample is the second combination, the positive sample label represents incorrect reasoning, and the negative sample label represents correct reasoning; the fine-tuned model is trained based on the second training set to obtain the logical reasoning model.

[0083] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The modules or units described in the embodiments of this application can be implemented in software or hardware. The names of modules or units do not necessarily limit the specific unit; for example, a first constraint acquisition module can also be described as a "module for acquiring the first constraint".

[0086] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0087] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0088] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0089] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A model training method, characterized in that, include: Obtain a first training set, which includes multiple training data, each training data including a first training sample and a corresponding first label; Each first training sample is a piece of text content, and each first label includes the reasoning result obtained based on the corresponding text content; The initial model is trained based on the first training set to obtain a fine-tuned model, and the erroneous output of the initial model for each first training sample during the training process is obtained; wherein, the erroneous output is the output of the initial model for each first training sample that is different from the corresponding first label. A second training set is obtained, comprising multiple second training samples and positive and negative sample labels corresponding to each second training sample. A second training sample can be a first training sample, a first combination consisting of a first training sample and its corresponding first label, or a second combination consisting of a first training sample and its corresponding error output. When a second training sample is a first training sample, the positive sample label is the first label corresponding to that first training sample, and the negative sample label is the error output corresponding to that first training sample. When a second training sample is a first combination, the positive sample label indicates correct reasoning, and the negative sample label indicates incorrect reasoning. When a second training sample is a second combination, the positive sample label indicates incorrect reasoning, and the negative sample label indicates correct reasoning. The fine-tuned model is trained based on the second training set to obtain a logical reasoning model.

2. The method according to claim 1, characterized in that, The step of training the fine-tuned model based on the second training set to obtain a logical reasoning model includes: Each second training sample in the second training set is sequentially input into the fine-tuned model for training until the training termination condition is met, thus obtaining the logical reasoning model. Each round of training includes: The second training sample for the current training round is obtained. The second training sample and the corresponding positive sample label are input into the model corresponding to this round of training so that the model generates a first bias of the positive sample label. The second training sample and the corresponding negative sample label are input into the model corresponding to this round of training so that the model generates a second bias of the negative sample label. The model corresponding to the first round of training is the fine-tuned model, and the bias is used to characterize the degree of tendency of the model to generate the corresponding sample label. The first propensity and the second propensity are compared to determine the first loss value representing the model in terms of propensity, and the second loss value representing the model in terms of inference accuracy is determined by a preset loss function. The first loss value and the second loss value are weighted and summed based on preset weights to obtain the total loss value of the model. The model parameters of the model are adjusted based on the total loss value, and the model with adjusted model parameters is used as the model for the next round of training.

3. The method according to claim 2, characterized in that, The first propensity is calculated as follows: For each second training sample, whenever the model generates any character in the first output, the character position of the character in the first output is determined, and a first probability value is determined based on the second training sample and the corresponding positive sample label that the character is the same as the character at the corresponding character position in the positive sample label; wherein, the first output is the inference result output by the model; For each character position, if the characters at the corresponding character positions in the positive sample label and the negative sample label are the same, then the weight of the character position is set to a first value; if they are different, then the weight of the character position is set to a second value; wherein the positive sample label and the negative sample label contain the same number of characters, and the first value is less than the second value; Based on the weights of each character position in the first output, the first probability values ​​of each character position are weighted and summed to obtain the second probability value of the first output being the positive sample label; The first tendency is determined based on the second probability value.

4. The method according to claim 3, characterized in that, The step of determining a first probability value that any character is the same as the character at the corresponding position in the positive sample label based on the second training sample and the corresponding positive sample label includes: Obtain a preset vocabulary, which includes multiple preset characters; The characters preceding the character position in the first output are combined with the second training sample to obtain a third combination; The third combination is converted into a corresponding text vector, and a second probability value is calculated based on the neural network parameters of the model and the text vector for each preset character in the preset vocabulary to be the character at the character position. The second probability value of the preset character that is the same as the character at the corresponding character position in the positive sample label is used as the first probability value.

5. The method according to claim 3, characterized in that, Determining the first propensity based on the second probability value includes: The first tendency is determined based on the degree of deviation of the second probability value from the preset value, wherein when the second probability value is greater than the preset value, the first tendency is directly proportional to the degree of deviation; when the second probability value is not greater than the preset value, the first tendency is inversely proportional to the degree of deviation.

6. The method according to claim 1, characterized in that, The first training set was obtained in the following way: Obtain multiple first training samples and a second label corresponding to each first training sample; each first training sample is a piece of text content, and each second label includes a reasoning result obtained based on the text content. For each first training sample, semantic analysis is performed on the first training sample and the corresponding second label to obtain the reasoning logic based on the text content and the reasoning result, and prompt information is generated based on the reasoning logic. The reasoning logic includes multiple reasoning steps, the reasoning objective corresponding to each reasoning step, and the order between the reasoning steps. For each first training sample, the first training sample, the corresponding second label, and the prompt information are input into a preset large language model to obtain a second output; wherein, the second output includes the thinking logic obtained by the large language model from the first training sample based on the reasoning logic, and the reasoning result; the thinking logic includes the reasoning process of obtaining the corresponding reasoning target from each reasoning step; For each first training sample, the second output corresponding to the first training sample is used as the third label of the first training sample. First training data is generated based on the first training sample and the corresponding second label, and second training data is generated based on the first training sample and the corresponding third label. The first label includes the second label and the third label. The first training set is generated based on each first training data and each second training data.

7. The method according to claim 1, characterized in that, The step of training the initial model based on the first training set to obtain the fine-tuned model includes: Each first training sample in the first training set is sequentially input into the initial model for training until the training termination condition is met, thus obtaining the fine-tuned model. Each round of training includes: Obtain the first training sample of the current training round, and input the first training sample into the model corresponding to this round, so that the model determines the third probability value that each character in the generated third output is the same as the character at the corresponding position in the first label corresponding to the first training sample; wherein, the model corresponding to the first round of training is the initial model; Based on the third probability value, the model parameters of the model corresponding to this round of training are adjusted, and the model with adjusted model parameters is used as the model for the next round of training.

8. A model training device, characterized in that, include: The first training set acquisition module is used to acquire a first training set, which includes multiple training data, and each training data includes a first training sample and a corresponding first label. Each first training sample is a piece of text content, and each first label includes the reasoning result obtained based on the corresponding text content; The first model training module is used to train the initial model based on the first training set to obtain the fine-tuned model, and to obtain the error output of the initial model for each first training sample during the training process; wherein, the error output is the output of the initial model for each first training sample that is different from the corresponding first label. The second training set acquisition module is used to acquire a second training set, which includes multiple second training samples and positive and negative sample labels corresponding to each second training sample. A second training sample can be a first training sample, a first combination consisting of a first training sample and its corresponding first label, or a second combination consisting of a first training sample and its corresponding error output. When the second training sample is a first training sample, the positive sample label is the first label corresponding to that first training sample, and the negative sample label is the error output corresponding to that first training sample. When the second training sample is a first combination, the positive sample label indicates correct reasoning, and the negative sample label indicates incorrect reasoning. When the second training sample is a second combination, the positive sample label indicates incorrect reasoning, and the negative sample label indicates correct reasoning. The second model training module is used to train the fine-tuned model based on the second training set to obtain a logical reasoning model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-7.

Images