A language model training method, device, storage medium and electronic device

Abstract

The present invention provides a language model training method, device, storage medium and electronic device, which belong to the field of natural language processing technology. The embodiment of the present invention first trains the initial language model based on a basic data set to obtain a first language model; then, according to the factual scores of the answer text pairs generated by the first language model under the same prompt word, the answer text with a higher factual score is marked as a preferred answer, and the answer text with a lower factual score is marked as a non-preferred answer, so as to construct a preference data set; finally, based on the preference data set and a preset factual reinforcement algorithm, the first language model is trained to obtain a target language model. The embodiment of the present invention uses the preference data set to perform further factual reinforcement training on the first language model, so that the trained target language model can give priority to texts with higher factuality when generating content, thereby significantly improving the factual accuracy of the content generated by the model.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a language model training method, device, storage medium and electronic device. Background Art

[0002] In the field of natural language processing, although deep learning-based language models can understand text semantics and generate coherent and relevant text fragments through pre-training on large-scale text data, they still face huge challenges in the factual accuracy of generated content. Language models often generate content that seems reasonable but is actually wrong.

[0003] Therefore, how to improve the factual accuracy of the content generated by the language model is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a language model training method, device, storage medium and electronic device to solve the problem that the factual accuracy of the generated content of the current language model is poor.

[0005] In order to solve the above problems, the present invention adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a language model training method, the method comprising:

[0007] Based on the basic data set, the initial language model is trained to obtain the first language model;

[0008] Acquire multiple answer text pairs output by the first language model according to multiple prompt words;

[0009] For the answer text pair corresponding to any prompt word, determine the factual score of each answer text in the answer text pair; mark the answer text with a higher factual score in the answer text pair as a preferred answer, and mark the answer text with a lower factual score in the answer text pair as a non-preferred answer, so as to construct a preference data set;

[0010] Based on the preference data set and a preset factual reinforcement algorithm, the first language model is trained to obtain a target language model.

[0011] In one embodiment of the present invention, the initial language model is trained based on the basic data set to obtain a first language model, including:

[0012] For any training text in the basic data set, obtaining a first predicted text output by the initial language model for the training text;

[0013] Determine a first plural representation of the first predicted text in a plural space, and a second plural representation of the text label of the training text in the plural space;

[0014] Determining an angle loss function value based on the first complex representation and the second complex representation;

[0015] Based at least on the angle loss function value, the network parameters of the initial language model are updated to obtain the first language model.

[0016] In one embodiment of the present invention, determining an angle loss function value based on the first complex number representation and the second complex number representation includes:

[0017] determining an angular difference between the first complex representation and the second complex representation;

[0018] Based on the angle difference and a preset angle loss function, the angle loss function value is determined.

[0019] In one embodiment of the present invention, the method further includes:

[0020] Determine a cosine value between the training text and the first predicted text, and determine a cosine loss function value based on the cosine value and a preset cosine loss function;

[0021] Determine the training text as a positive sample, and randomly sample the basic data set to obtain a plurality of negative samples; determine the value of the intra-batch negative sample loss function based on the positive sample, the plurality of negative samples and a preset intra-batch negative sample loss function;

[0022] At least based on the angle loss function value, updating the network parameters of the initial language model to obtain the first language model includes:

[0023] Determine a first loss function value based on the angle loss function value, the cosine loss function value, and the in-batch negative sample loss function value;

[0024] Based on the first loss function value, the network parameters of the initial language model are updated to obtain the first language model.

[0025] In one embodiment of the present invention, determining a first loss function value based on the angle loss function value, the cosine loss function value, and the in-batch negative sample loss function value includes:

[0026] Obtaining a first weight of the angle loss function value, a second weight of the cosine loss function value, and a third weight of the intra-batch negative sample loss function value;

[0027] The first loss function value is determined based on the angle loss function value, the first weight, the cosine loss function value, the second weight, the in-batch negative sample loss function value and the third weight.

[0028] In one embodiment of the present invention, the first language model is trained based on the preference data set and a preset factual reinforcement algorithm to obtain a target language model, including:

[0029] For any preference data in the preference data set, input the preference data into the first language model, and determine a first probability that the first language model generates the preferred answer and a second probability that the first language model generates the non-preferred answer;

[0030] Determine a reinforcement algorithm loss function value based on the first probability, the second probability, and the reinforcement algorithm loss function corresponding to the factual reinforcement algorithm;

[0031] Based at least on the loss function value of the reinforcement algorithm, the network parameters of the first language model are updated to obtain the target language model.

[0032] In one embodiment of the present invention, the method further includes:

[0033] For any preference data in the preference data set, input the prompt word of the preference data into the first language model, output a second predicted text, determine the cross entropy loss function value based on the generation probability of each word in the second predicted text; and determine the regularization term parameter based on the factual score of each word in the second predicted text;

[0034] At least based on the loss function value of the reinforcement algorithm, updating the network parameters of the first language model to obtain the target language model includes:

[0035] Determine a second loss function value based on the reinforcement algorithm loss function value, the cross entropy loss function value and the regularization term parameter;

[0036] Based on the second loss function value, the network parameters of the first language model are updated to obtain the target language model.

[0037] In a second aspect, based on the same inventive concept, an embodiment of the present invention provides a language model training device, the device comprising:

[0038] A first training module, used to train the initial language model based on the basic data set to obtain a first language model;

[0039] A text acquisition module, used to acquire a plurality of answer text pairs output by the first language model according to a plurality of prompt words;

[0040] A data construction module is used to determine the factuality score of each answer text in the answer text pair corresponding to any prompt word; mark the answer text with a higher factual score in the answer text pair as a preferred answer, and mark the answer text with a lower factual score in the answer text pair as a non-preferred answer, so as to construct a preference data set;

[0041] The second training module is used to train the first language model based on the preference data set and a preset factual reinforcement algorithm to obtain a target language model.

[0042] In one embodiment of the present invention, the first training module includes:

[0043] A first prediction submodule, configured to obtain, for any training text in the basic data set, a first predicted text output by the initial language model for the training text;

[0044] a plural mapping submodule, configured to determine a first plural representation of the first predicted text in a plural space, and a second plural representation of the text label of the training text in the plural space;

[0045] A first loss determination submodule, for a submodule, for determining an angle loss function value based on the first complex number representation and the second complex number representation;

[0046] The first updating submodule is used to update the network parameters of the initial language model based at least on the angle loss function value to obtain the first language model.

[0047] In one embodiment of the present invention, the complex number mapping submodule includes:

[0048] an angle difference determining unit, configured to determine an angle difference between the first complex number representation and the second complex number representation;

[0049] The first loss determination unit is used to determine the angle loss function value based on the angle difference and a preset angle loss function.

[0050] In one embodiment of the present invention, the language model training device includes:

[0051] A cosine loss function value determination module, used to determine a cosine value between the training text and the first predicted text, and determine a cosine loss function value based on the cosine value and a preset cosine loss function;

[0052] The module for determining the negative sample loss function value within the batch is used to determine the training text as a positive sample, and randomly sample the basic data set to obtain a plurality of negative samples; based on the positive sample, the plurality of negative samples and a preset negative sample loss function within the batch, determine the negative sample loss function value within the batch;

[0053] The first updating submodule comprises:

[0054] A first loss unit, configured to determine a first loss function value based on the angle loss function value, the cosine loss function value, and the in-batch negative sample loss function value;

[0055] A first updating unit is used to update the network parameters of the initial language model based on the first loss function value to obtain the first language model.

[0056] In one embodiment of the present invention, the first loss unit includes:

[0057] A weight acquisition subunit, used to acquire a first weight of the angle loss function value, a second weight of the cosine loss function value, and a third weight of the intra-batch negative sample loss function value;

[0058] The first loss determination subunit is used to determine the first loss function value based on the angle loss function value, the first weight, the cosine loss function value, the second weight, the in-batch negative sample loss function value and the third weight.

[0059] In one embodiment of the present invention, the second training module includes:

[0060] a probability determination submodule, for inputting any preference data in the preference data set into the first language model, and determining a first probability that the first language model generates the preferred answer and a second probability that the first language model generates the non-preferred answer;

[0061] A second loss determination submodule, configured to determine a reinforcement algorithm loss function value based on the first probability, the second probability, and the reinforcement algorithm loss function corresponding to the factual reinforcement algorithm;

[0062] The second updating submodule is used to update the network parameters of the first language model based on at least the loss function value of the reinforcement algorithm to obtain the target language model.

[0063] In one embodiment of the present invention, the language model training device further includes:

[0064] a loss determination module, for inputting a prompt word of the preference data into the first language model for any preference data in the preference data set, outputting a second predicted text, determining a cross entropy loss function value based on a generation probability of each word in the second predicted text; and determining a regularization term parameter based on a factual score of each word in the second predicted text;

[0065] The second updating submodule includes:

[0066] A third loss determination unit, configured to determine a second loss function value based on the reinforcement algorithm loss function value, the cross entropy loss function value, and the regularization term parameter;

[0067] The second updating unit is used to update the network parameters of the first language model based on the second loss function value to obtain the target language model.

[0068] In the third aspect, based on the same inventive concept, an embodiment of the present invention provides a computer-readable storage medium having an executable program stored thereon, which, when executed by a processor, implements the language model training method proposed in the first aspect of the present invention.

[0069] In a fourth aspect, based on the same inventive concept, an embodiment of the present invention provides an electronic device, including:

[0070] A memory for storing an executable program;

[0071] processor;

[0072] When the executable program is executed by the processor, the language model training method proposed in the first aspect of the present invention is implemented.

[0073] Compared with the prior art, the present invention has the following advantages:

[0074] A language model training method provided by an embodiment of the present invention first trains an initial language model based on a basic data set to obtain a first language model; then obtains multiple answer text pairs output by the first language model according to multiple prompt words, and for any answer text pair corresponding to a prompt word, determines the factuality score of each answer text in the answer text pair, and then marks the answer text with a higher factuality score in the answer text pair as a preferred answer, and marks the answer text with a lower factuality score in the answer text pair as a non-preferred answer, so as to construct a preference data set; finally, based on the preference data set and a preset factuality reinforcement algorithm, the first language model is trained to obtain a target language model. The embodiment of the present invention calculates the factuality scores of two answer texts generated by the first language model under the same prompt word, and obtains a preference data set according to the factuality score annotation, and can use the preference data set to further factually reinforce the first language model, so that the trained target language model can give priority to texts with higher factuality when generating content, thereby significantly improving the factual accuracy of the content generated by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 It is a flow chart of the steps of a language model training method in one embodiment of the present invention.

[0077] Figure 2 It is a functional module diagram of a language model training device in one embodiment of the present invention.

[0078] Figure 3 It is a schematic structural diagram of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] It should be noted that in the field of natural language processing, especially in text generation tasks, although there has been significant progress in recent years, the existing technology still has many shortcomings. Although deep learning-based language models can understand text semantics and generate coherent and relevant text fragments through pre-training on large-scale text data, they still face huge challenges in the factual accuracy of generated content. Generative models often produce content that seems reasonable but is actually wrong.

[0081] These problems mainly stem from the following limitations: First, existing language models are mainly trained with the goal of maximum likelihood, and what they learn is the statistical laws of language rather than the factual knowledge of the real world, so they are prone to generating false or inconsistent content. Second, for long text generation in open domains, existing methods have difficulty accurately evaluating the factuality of generated content, and there is a lack of effective manually annotated data. In addition, commonly used text generation sampling strategies such as greedy sampling or beam search, although they can generate fluent text, often lack diversity and cannot ensure factual accuracy.

[0082] In view of the problem that the factual accuracy of the content generated by the current language model is relatively poor, the present invention aims to provide a language model training method. First, based on a basic data set, an initial language model is trained to obtain a first language model. Then, the factuality scores of two answer texts generated by the first language model under the same prompt word are calculated, and a preference data set is obtained according to the factuality score annotation. The preference data set can be used to perform further factual reinforcement training on the first language model, so that the trained target language model can give priority to texts with higher factuality when generating content, thereby significantly improving the factual accuracy of the content generated by the model.

[0083] Reference Figure 1 , showing a language model training method of the present invention, the method may include the following steps:

[0084] S101: Based on the basic data set, an initial language model is trained to obtain a first language model.

[0085] In this embodiment, the basic data set refers to the original text data used to train the initial language model. The original text data may include text data of various text types, such as news articles, social media content, books, Wikipedia, etc., to ensure that the original text data can cover the diversity and complexity of the required language.

[0086] In a specific implementation, the basic data set can be cleaned and preprocessed, including removing noise, segmenting, processing punctuation and special characters, and converting the text into a unified format to ensure the quality and consistency of the data; after completing the data preprocessing, the basic data set divides the basic data set into a training set and a validation set, wherein the training set is used for the actual training of the initial language model, and the validation set is used to monitor the performance of the model to prevent overfitting. In the model training stage, the training text in the training set is input into the initial language model, so that the initial language model performs forward propagation according to the training text, and calculates the loss function value, and then uses the back propagation algorithm to update the network parameters of the initial language model, through multiple iterations, until the initial language model meets the first training cutoff condition, and obtains the first language model. Among them, the first training cutoff condition can be set to the loss function value being in a stable state, or the training round reaches the preset maximum round, etc. It should be noted that the first training cutoff condition can be set and adjusted by comprehensively considering the complexity of the model, the characteristics of the data, and the requirements of the task, and this implementation is not specifically limited.

[0087] It should be noted that the first language model trained with the basic data set can understand the text semantics of the input text and generate coherent and relevant text fragments, but it may not ensure the factual accuracy of the generated content. Therefore, in order to ensure the factual accuracy of the generated text, the first language model will be further trained to enhance the factuality so that it can output text with higher factuality when generating content.

[0088] S102: Acquire a plurality of answer text pairs output by the first language model according to a plurality of prompt words.

[0089] It should be noted that in the field of natural language processing and machine learning, prompts are a technical means to guide or stimulate models to generate specific outputs. It is usually used as part of the input to provide the model with the contextual information or instructions required to perform tasks. By designing precise prompts, researchers and developers can guide pre-trained language models to complete various tasks, such as text generation, sentiment analysis, question answering, etc., without the need for large-scale retraining or fine-tuning of the model. The effective use of prompts can significantly improve the performance and adaptability of the model on specific tasks, making it more flexible and efficient when handling more complex natural language processing tasks.

[0090] In this embodiment, in order to ensure the factual accuracy of the first language model when processing various language tasks, a sufficient number of different types of prompt words can be used to instruct the first language model to generate answer text pairs in various fields. The answer text includes two answer texts obtained by downsampling the same prompt word.

[0091] In a specific implementation, for any prompt word among the multiple prompt words, by sequentially inputting the prompt word into the first language model, two answer texts output by the first language model can be obtained, and then an answer text pair corresponding to the prompt word can be obtained.

[0092] S103: For the answer text pair corresponding to any prompt word, determine the factual score of each answer text in the answer text pair; mark the answer text with a higher factual score in the answer text pair as a preferred answer, and mark the answer text with a lower factual score in the answer text pair as a non-preferred answer, so as to construct a preference data set.

[0093] In this embodiment, by calculating the factuality score of each answer text in each answer text pair, the accuracy and reliability of each answer text can be effectively evaluated.

[0094] In this embodiment, a structured knowledge base, such as Wikipedia, DBpedia (a knowledge base), Wikidata, etc., can be used for comparison. The generated answer text is compared with the relevant facts in the knowledge base to evaluate the factuality score of the answer text.

[0095] In this embodiment, for any answer text, the probability of each word in the answer text can be generated based on the context according to the first language model to determine the factual score of each word, and then the factual score of the answer text can be determined according to the factual score of each word.

[0096] In a specific implementation, the factual score of each word in the answer text generated by the first language model based on the context can be determined according to the following formula:

[0097] S fact (ω)=α·P model (ω|context)+ (1-α)P fact (ω) (1);

[0098] Among them, ω represents the words in the answer text; S fact (ω) is the factual score of word ω in the answer text; P model (ω|context) is the probability of the first language model generating word ω based on the context; P fact (ω) is the factual verification probability of word ω; α is a weight parameter, a value between 0 and 1, which is used to control the weight between the prediction probability and the factual probability of the first language model. The factual verification probability of word ω can be determined based on the probability that word ω appears as a fact in a preset fact database.

[0099] In this embodiment, after calculating the factual scores of the two answer texts in the answer text pair, the answer text with a higher factual score in the answer text pair can be marked as a preferred answer, and the answer text with a lower factual score in the answer text pair can be marked as a non-preferred answer, thereby obtaining a preference data. The preference data includes a prompt word, an answer text marked as a preferred answer, and an answer text marked as a non-preferred answer. In other words, different preference data can be obtained according to different prompt words, and then based on the preference data corresponding to each of the multiple prompt words, a preference data set containing multiple preference data is constructed.

[0100] S104: Based on the preference data set and a preset factual reinforcement algorithm, the first language model is trained to obtain a target language model.

[0101] It should be noted that the factual reinforcement algorithm is an algorithm designed to improve the authenticity and accuracy of the text content generated by the language model. The first language model is trained based on the preference data set using the factual reinforcement algorithm, so that the first language model gradually learns a response generation mode that is biased towards real text while maintaining its original language generation ability.

[0102] In a specific implementation, the factual reinforcement algorithm can be used to fine-tune the first language model so that it tends to generate text content that is biased towards preferred answers and away from non-preferred answers. Specifically, the preference data in the preference data set can be first input into the first language model, so that the first language model performs forward propagation according to the input preference data, and then the loss value is calculated according to the reinforcement algorithm loss function corresponding to the factual reinforcement algorithm, and then the network parameters of the first language model are updated using the back propagation algorithm, and multiple iterations are performed until the first language model meets the second training cutoff condition to obtain the target language model. Among them, the second training cutoff condition can be set with reference to the first training cutoff condition.

[0103] It should be noted that, in the application stage of the target language model, the target language model is used to output a target answer text based on the target text input by the user. The target answer text is an answer text that is semantically coherent and meets factual accuracy.

[0104] In this embodiment, the initial language model is trained by using a basic data set, so that a first language model that can generate coherent and relevant answer texts can be first trained; then, based on the first language model, the factuality scores of two answer texts generated by the first language model under the same prompt word are calculated, and a preference data set is obtained according to the factuality scores. The preference data set can be used to perform further factual reinforcement training on the first language model, so that the trained target language model can give priority to texts with higher factuality when generating content, thereby significantly improving the factual accuracy of the content generated by the model.

[0105] In a feasible manner, in order to improve the training effect of the first language model and to improve the text quality of the content generated by the first language model, S101 may specifically include the following sub-steps:

[0106] S101 - 1 : For any training text in the basic data set, obtain a first predicted text output by the initial language model for the training text.

[0107] In this embodiment, in the preliminary preparation stage of text generation, any training text in the basic data set can be mapped to the complex space to decompose each element of the training text into a real part and an imaginary part, thereby obtaining the complex representation of the training text. Specifically, the complex representation of the text can be expressed by the following formula:

[0108] Z=X+Yi (2);

[0109] Among them, Z is the text representation in the mapped complex space; X and Y represent the mapping of the original text in the real space and imaginary space respectively; i is the imaginary unit.

[0110] It should be noted that traditional text representation methods (such as bag-of-words models, word embedding, etc.) usually use real vectors to represent text. Complex space text mapping embeds text into complex space so that each text element can be represented not only by the real part but also by the imaginary part. This makes the expression of each text richer because it can contain information of two dimensions at the same time, which means that the text representation in complex space can contain more information than in real space. This high-dimensional mapping enables the model to learn in a larger feature space, thereby extracting richer features and improving the model training effect.

[0111] For example, the real part can represent some subjective or deterministic features. For example, in sentiment analysis, the real part can be used to represent the basic emotional intensity or bias of the text, such as the quantitative value of positive or negative sentiment; it can also represent some significant features in the text content, such as word frequency, importance score, etc. The imaginary part can represent some complementary or complex information. In the example of sentiment analysis, the imaginary part can reflect the subtle differences in emotions, such as the diversity of emotions, the rotation or change of emotions, etc.; it can also represent other dimensions of information in the text content, such as the richness of the topic, the complexity of the context, etc. By separating the real part and the imaginary part, the complex representation can better capture the multidimensional information in the text data. For example, in some tasks of natural language processing, the model may need to process different levels of information (such as the intensity of the wording and the complexity of the emotion), and the complex representation allows this information to be encoded and processed independently. In this way, using the mathematical properties of complex operations (such as phase, amplitude, etc.) can help the initial language model to more flexibly capture these multidimensional information when processing text, thereby improving the performance and expression ability of the model.

[0112] S101 - 2 : Determine a first plural representation of the first predicted text in the plural space, and a second plural representation of the text label of the training text in the plural space.

[0113] In this embodiment, after the initial language model analyzes and identifies the plural representation of the training text, it can output a first plural representation of the first predicted text in the plural space. It should be noted that each training text also includes its corresponding text label, and by mapping the text label to the plural space, a second plural representation of the text label in the plural space can also be obtained.

[0114] In this embodiment, by mapping the text data into a complex space consisting of real and imaginary parts, the dimension and expression richness of the language model when processing text similarity tasks can be effectively enhanced, thereby improving the accuracy and diversity of the text. The use of complex space allows the language model to perform a deeper analysis of the text data, thereby achieving higher accuracy and naturalness in tasks such as text generation and machine translation.

[0115] S101 - 3 : Determine an angle loss function value based on the first complex number representation and the second complex number representation.

[0116] In a specific implementation, considering that the complex representation of text data in the complex space is composed of a real part and an imaginary part, the complex representation of each text data corresponds to an angle, and then based on the first complex representation and the second complex representation, the angle difference between the first complex representation and the second complex representation can be first determined, and then based on the angle difference and the preset angle loss function, the angle loss function value can be determined.

[0117] In this embodiment, let the first predicted text be X m , the text label is represented by X n , in complex space, X m With X n It can be expressed as:

[0118] X m = z = a + bi (3);

[0119] X n =w=c+di (4);

[0120] Among them, z is the first predicted text X m The first complex representation mapped to the complex space, a is the real part of the first complex representation, b is the imaginary part of the first complex representation; w is the text label X n The second complex number representation mapped to the complex number space, c is the real part of the second complex number representation, and d is the imaginary part of the second complex number representation.

[0121] In this implementation, the angle difference between the first complex representation and the second complex representation may be determined according to the following formula:

[0122]

[0123] Among them, Δθ zw represents the angular difference between the first complex representation and the second complex representation; θ z represents the angle represented by the first complex number; θ w represents the angle represented by the second complex number; r z represents the modulus of the first complex number representation; r w represents the modulus of the second complex representation.

[0124] In this embodiment, after determining the angle difference between the first complex representation and the second complex representation, the angle loss function value may be calculated according to the following angle loss function:

[0125]

[0126] Among them, L angle Represents the angle loss function value, sim(X m ,X n ) represents the first predicted text X of the current training text m and text label X n The similarity of sim(X i ,X j ) represents the first predicted text X of other training texts in the basic dataset i and text label X j Similarity of ; Δθ mnRepresents the first predicted text X of the current training text m and text label X n The angle difference between ij The first predicted text X represents other training texts i and text label X j The angle difference between them; τ represents the temperature parameter.

[0127] It should be noted that other training texts refer to texts in the basic data set other than the current training text. When calculating the angle loss function value corresponding to any current training text, it is necessary to ensure that the sim(X i ,X j ) is greater than the sim(X m ,X n ).

[0128] S101 - 4: At least based on the angle loss function value, update the network parameters of the initial language model to obtain a first language model.

[0129] In this embodiment, by minimizing the angle loss function during the training of the initial language model, the angle difference in the complex space can be effectively optimized, so that the generated text is more semantically related, especially when processing long texts, the overall semantic structure of the text can be captured more accurately. Compared with traditional text embedding methods, angle difference optimization can analyze text similarity more carefully and deeply.

[0130] In this embodiment, in order to further improve the training effect of the initial language model, the initial language model can also be trained based on the angle loss function value, combined with the cosine loss function value and the batch negative sample loss function value. In a specific implementation, the language model training method can also include the following steps:

[0131] S201: Determine a cosine value between a training text and a first predicted text, and determine a cosine loss function value based on the cosine value and a preset cosine loss function.

[0132] In the specific implementation, the cosine loss function value can be calculated based on the following cosine loss function:

[0133]

[0134] Among them, L cos Represents the cosine loss function value; sim(X m ,X n ) represents the first predicted text X of the current training text m and text label X n The similarity of sim(X i,X j ) represents the first predicted text X of other training texts in the basic dataset i and text label X j Similarity; cos(X m ,X n ) represents the first predicted text X m and text label X n The cosine value of cos(X i ,X j ) represents the first predicted text X of other training texts i and text label X j The cosine value of ; τ represents the temperature parameter.

[0135] In this embodiment, by learning the cosine loss function, the initial language model can sort the cosine similarities of different text pairs according to the size relationship of the label similarities, so that the cosine similarity is more consistent with the manually annotated similarity label, and then through comparative learning, a text representation space that is closer to the semantic similarity is learned.

[0136] S202: Determine the training text as a positive sample, and randomly sample the basic data set to obtain multiple negative samples; determine the intra-batch negative sample loss function value based on the positive sample, the multiple negative samples and a preset intra-batch negative sample loss function.

[0137] It should be noted that the in-batch negative objective function is a method used in the field of natural language processing to optimize the performance of text embedding models, and is particularly suitable for improving the model's sensitivity to text semantic differences. This method enhances the initial language model's ability to distinguish between positive and negative sample pairs by comparing the similarities between different texts within a batch. Specifically, it regards each text in the batch with other unpaired texts as a negative sample, and optimizes the parameters of the initial language model by calculating the similarity difference between positive and negative sample pairs. This method utilizes all possible negative sample pairs in the batch without the need for additional negative sampling, thereby effectively improving training efficiency and the generalization ability of the model. In tasks such as text similarity analysis, information retrieval, and language model pre-training, the in-batch negative loss function can significantly improve the quality of text embedding and the performance of the model.

[0138] In this embodiment, the basic data set can be divided into multiple batches, each batch contains a positive sample and multiple negative samples, and it is hoped that each sample is more similar to its positive sample and less similar to other negative samples. The intra-batch negative sample loss function does not require additional construction of negative samples, which can effectively reduce the training time and space complexity; at the same time, because each batch is randomly sampled, the intra-batch negative samples also have a certain ability to mine difficult samples, and can learn more discriminative feature representations.

[0139] In the specific implementation, the value of the negative sample loss function within the batch can be determined according to the following negative sample loss function within the batch:

[0140]

[0141] Among them, L ib represents the loss function value of negative samples within a batch; N is the batch size; b represents the bth batch; and represents the i-th sample and the j-th sample in batch b; and They are and positive samples; m is the number of positive data pairs in batch b. When the number of positive samples is 1, m is 1.

[0142] In this implementation, in combination with the cosine loss function value and the intra-batch negative sample loss function value, S101-4 may specifically include the following steps:

[0143] S101-4-1: Determine a first loss function value based on the angle loss function value, the cosine loss function value, and the in-batch negative sample loss function value.

[0144] In a specific implementation, a first weight of the angle loss function value, a second weight of the cosine loss function value, and a third weight of the in-batch negative sample loss function value are obtained; based on the angle loss function value, the first weight, the cosine loss function value, the second weight, the in-batch negative sample loss function value, and the third weight, the first loss function value is determined.

[0145] Specifically, the first loss function value can be calculated according to the following formula:

[0146] L1=w1L angle +w2L cos +w3L ib (9);

[0147] Among them, L1 represents the first loss function value; L angle Represents the angle loss function value; L cos Represents the cosine loss function value; L ibrepresents the negative sample loss function value within the batch; w1 represents the first weight of the angle loss function value; w2 represents the second weight of the cosine loss function value; w3 represents the third weight of the negative sample loss function value within the batch. The sum of w1, w2 and w3 is 1.

[0148] It should be noted that the cosine function has a saturation region near -1 and 1, and the gradient is close to 0, which may affect the optimization, and the angle difference has more obvious changes in these areas. Therefore, the angle loss function can alleviate the saturation region problem of the cosine loss function.

[0149] S101-4-2: Based on the first loss function value, update the network parameters of the initial language model to obtain a first language model.

[0150] In this embodiment, by combining the angle loss function, the intra-batch negative sample loss function and the cosine loss function, a more comprehensive first loss function can be obtained, which helps to learn a better text representation, thereby improving the flexibility and performance of the model in processing different types of text data and tasks.

[0151] In a feasible implementation, S104 may specifically include the following sub-steps:

[0152] S104-1: For any preference data in the preference data set, input the preference data into a first language model, and determine a first probability that the first language model generates a preferred answer and a second probability that the first language model generates a non-preferred answer.

[0153] In this embodiment, the factual reinforcement algorithm can adopt DPO (Direct Preference Optimization). It should be noted that DPO is a strategy used to optimize the prediction performance of models in the fields of machine learning and artificial intelligence. It improves the decision quality of the model by directly applying the optimization algorithm to the predicted preferences, rather than the traditional error minimization method. DPO can more accurately capture the true preferences of users or tasks by optimizing the direct comparison of preference scores, thereby improving the relevance of recommendations and user satisfaction. DPO emphasizes the ultimate goal of the optimization task, rather than simply reducing the prediction error, so that the model is closer to actual application needs.

[0154] In this embodiment, since the preference data includes preferred answers and non-preferred answers under the same prompt word, the first language model can be trained for factual reinforcement using factual reinforcement algorithms such as DPO, so that the generated content of the first language model can gradually tend to the preferred answers and away from the non-preferred answers.

[0155] In this embodiment, the enhancement algorithm loss function corresponding to the DPO can be constructed based on the first probability and the second probability, and then the enhancement algorithm loss function is calculated by the first probability and the second probability, which can make the first language model tend to generate highly factual answers.

[0156] S104-2: Determine a reinforcement algorithm loss function value based on the first probability, the second probability, and the reinforcement algorithm loss function corresponding to the factual reinforcement algorithm.

[0157] In a specific implementation, based on the first probability of generating a preferred answer and the second probability of generating a non-preferred answer by the first language model, the following reinforcement algorithm loss function can be constructed:

[0158] L DPO =log(P(y_w|x) / π(y_w|x))-log(P(y_r|x) / π(y_r|x)) (10);

[0159] in:

[0160] L DPO represents the loss function value of the reinforcement algorithm; x represents the prompt word input into the first language model;

[0161] P(y_w|x) represents the first probability of the first language model generating a preferred answer under the condition of the prompt word x, where y_w represents the preferred response text sequence, i.e., the preferred answer; π(y_w|x) represents the conditional probability of the baseline model (i.e., the first language model before factual reinforcement training) generating y_w under the same prompt word x; π represents the probability distribution strategy of the baseline model, and π(y_w|x) is used to prevent the first language model from deviating too much from the initial language ability during the training process;

[0162] P(y_r|x) represents the second probability of the first language model generating a non-preferred answer under the condition of the prompt word x, where y_r represents a response text sequence that is evaluated by humans as having lower quality, i.e., a non-preferred answer; π(y_r|x) represents the conditional probability of the baseline model (i.e., the first language model before factual reinforcement training) generating y_r under the same prompt word x; π(y_r|x) is used to evaluate the degree of change in the first language model's generation tendency for non-preferred answers relative to the baseline model;

[0163] P(y_w|x) / π(y_w|x) represents the probability preference of the current first language model relative to the baseline model when generating preferred answers. When this ratio is greater than 1, it indicates that the current first language model has a stronger tendency to generate facts relative to the baseline model; when this ratio is less than 1, it indicates that the current first language model has a lower tendency to generate facts relative to the baseline model; similarly, P(y_r|x) / π(y_r|x) represents the probability preference of the first language model relative to the baseline model when generating non-preferred answers.

[0164] S104-3: Based at least on the loss function value of the reinforcement algorithm, update the network parameters of the first language model to obtain a target language model.

[0165] In this embodiment, by maximizing the loss function value of the reinforcement algorithm, the probability of the first language model generating highly factual expected text can be increased, and the probability of the first language model generating low-factuality undesirable text can be reduced, while the basic capabilities of the first language model are maintained through the constraints of the baseline model.

[0166] In this embodiment, the first language model is trained for factual reinforcement using a reinforcement algorithm loss function, so that the trained target language model can give priority to more factual vocabulary during the text generation process, greatly reducing the risk of factual errors in the generated text, thereby significantly improving the quality and factuality of the text generation task, especially when dealing with complex and fact-intensive texts.

[0167] In a feasible implementation, in order to further improve the training effect of the first language model, the first language model can be trained based on the loss function value of the strengthening algorithm, combined with the cross entropy loss function value and the regularization term parameter. In a specific implementation, the language model training method can also include the following steps:

[0168] S301: For any preference data in the preference data set, input a prompt word of the preference data into a first language model, output a second predicted text, and determine a cross entropy loss function value based on the generation probability of each word in the second predicted text.

[0169] In the specific implementation, the cross entropy loss function value can be calculated according to the following cross entropy loss function:

[0170] L CE =-∑ ω∈W logP model (ω|context) (11);

[0171] Among them, L CE represents the cross entropy loss function value; W represents the word set of the second predicted text; ω represents the word of the second predicted text; Pmodel (ω|context) is the probability of the first language model generating word ω based on the context.

[0172] In this embodiment, by calculating the cross entropy loss function value according to the generation probability of each word in the second predicted text, the difference between the probability distribution predicted by the first language model and the actual label can be effectively measured, thereby quantifying the accuracy of the text generated by the first language model. By minimizing the cross entropy loss function value, the first language model can learn more accurate word generation probabilities, thereby improving the accuracy of the prediction.

[0173] S302: Determine regularization item parameters based on the factuality score of each word in the second predicted text.

[0174] In this embodiment, the regularization term parameters shown below are constructed based on the factuality scores of each word in the second predicted text:

[0175] R=λ(1-∑ ω∈W P fact (ω)) (12);

[0176] Where R represents the regularization parameter; W represents the word set of the second predicted text; ω represents the word of the second predicted text; P fact (ω) represents the factual verification probability of word ω in the second predicted text; λ is the weight coefficient of the regularization term.

[0177] In this embodiment, by adding regularization term parameters as factual constraints, it is possible to not only prevent the first language model from overfitting and improve the generalization ability of the model, but also encourage the facts generated by the first language model to match the factual information in the known fact database.

[0178] In this implementation, in combination with the cross entropy loss function value and the regularization term parameter, S104-3 may specifically include the following sub-steps:

[0179] S104-3-1: Determine a second loss function value based on the reinforcement algorithm loss function value, the cross entropy loss function value and the regularization term parameter.

[0180] In a specific implementation, the second loss function value can be calculated according to the following formula:

[0181] L2=L DPO +L CE +R (13);

[0182] Where L2 represents the second loss function value; L DPO Represents the loss function value of the reinforcement algorithm; L CE represents the cross entropy loss function value; R represents the regularization term parameter.

[0183] In this embodiment, corresponding weights can also be set for the reinforcement algorithm loss function value, the cross entropy loss function value and the regularization term parameter, respectively, and then based on the weight, the reinforcement algorithm loss function value, the cross entropy loss function value and the regularization term parameter are weightedly summed to calculate the second loss function value.

[0184] S104-3-2: Based on the second loss function value, update the network parameters of the first language model to obtain a target language model.

[0185] In this embodiment, by combining the reinforcement algorithm loss function value, the cross entropy loss function value and the regularization term parameter, a more comprehensive second loss function can be obtained, which significantly improves the factuality of the target language model in the text generation task.

[0186] In this embodiment, the target language model is obtained by combining the plural space text mapping with the factual reinforcement algorithm training, which not only solves the common factual inaccuracy problem in traditional text generation methods, but also improves the overall quality of the generated text. The plural space text mapping improves the processing effect of the text similarity task, and the factual reinforcement algorithm ensures the factual accuracy of the text generated by the target language model. In this way, the quality and factual accuracy of the content generated by the target language model can be effectively ensured when processing complex and fact-intensive texts.

[0187] In the second aspect, based on the same inventive concept, refer to Figure 2 The embodiment of the present invention provides a language model training device 200, and the language model training device 200 includes:

[0188] A first training module 201 is used to train the initial language model based on the basic data set to obtain a first language model;

[0189] A text acquisition module 202, used to acquire a plurality of answer text pairs output by the first language model according to a plurality of prompt words;

[0190] The data construction module 203 is used to determine the factuality score of each answer text in the answer text pair corresponding to any prompt word; mark the answer text with a higher factual score in the answer text pair as a preferred answer, and mark the answer text with a lower factual score in the answer text pair as a non-preferred answer, so as to construct a preference data set;

[0191] The second training module 204 is used to train the first language model based on the preference data set and a preset factual reinforcement algorithm to obtain a target language model.

[0192] In one embodiment of the present invention, the first training module 201 includes:

[0193] A first prediction submodule is used to obtain, for any training text in the basic data set, a first predicted text output by the initial language model for the training text;

[0194] A plural mapping submodule, used to determine a first plural representation of the first predicted text in the plural space, and a second plural representation of the text label of the training text in the plural space;

[0195] A first loss determination submodule, for a submodule, for determining an angle loss function value based on the first complex number representation and the second complex number representation;

[0196] The first updating submodule is used to update the network parameters of the initial language model based at least on the angle loss function value to obtain a first language model.

[0197] In one embodiment of the present invention, the complex mapping submodule includes:

[0198] an angle difference determination unit, configured to determine an angle difference between the first complex number representation and the second complex number representation;

[0199] The first loss determination unit is used to determine the angle loss function value based on the angle difference and a preset angle loss function.

[0200] In one embodiment of the present invention, the language model training device 200 includes:

[0201] A cosine loss function value determination module, used to determine a cosine value between a training text and a first predicted text, and determine a cosine loss function value based on the cosine value and a preset cosine loss function;

[0202] The module for determining the loss function value of negative samples in a batch is used to determine the training text as a positive sample, and randomly sample the basic data set to obtain multiple negative samples; based on the positive sample, multiple negative samples and a preset loss function of negative samples in a batch, determine the loss function value of negative samples in a batch;

[0203] The first update submodule includes:

[0204] A first loss unit, used to determine a first loss function value based on an angle loss function value, a cosine loss function value, and an intra-batch negative sample loss function value;

[0205] The first updating unit is used to update the network parameters of the initial language model based on the first loss function value to obtain the first language model.

[0206] In one embodiment of the present invention, the first loss unit includes:

[0207] A weight acquisition subunit, used to acquire a first weight of an angle loss function value, a second weight of a cosine loss function value, and a third weight of a negative sample loss function value within a batch;

[0208] The first loss determination subunit is used to determine the first loss function value based on the angle loss function value, the first weight, the cosine loss function value, the second weight, the in-batch negative sample loss function value and the third weight.

[0209] In one embodiment of the present invention, the second training module 204 includes:

[0210] a probability determination submodule, for inputting the preference data into the first language model for any preference data in the preference data set, and determining a first probability that the first language model generates a preferred answer and a second probability that the first language model generates a non-preferred answer;

[0211] A second loss determination submodule, configured to determine a reinforcement algorithm loss function value based on the first probability, the second probability, and the reinforcement algorithm loss function corresponding to the factual reinforcement algorithm;

[0212] The second updating submodule is used to update the network parameters of the first language model based on at least the loss function value of the reinforcement algorithm to obtain a target language model.

[0213] In one embodiment of the present invention, the language model training device 200 further includes:

[0214] A loss determination module is used for inputting a prompt word of the preference data into the first language model for any preference data in the preference data set, outputting a second predicted text, determining a cross entropy loss function value based on a generation probability of each word in the second predicted text; and determining a regularization term parameter based on a factual score of each word in the second predicted text;

[0215] The second update submodule includes:

[0216] A third loss determination unit, used to determine a second loss function value based on a reinforcement algorithm loss function value, a cross entropy loss function value, and a regularization term parameter;

[0217] The second updating unit is used to update the network parameters of the first language model based on the second loss function value to obtain a target language model.

[0218] It should be noted that the specific implementation of a language model training device 200 in an embodiment of the present invention refers to the specific implementation of the language model training method proposed in the first aspect of the above-mentioned embodiment of the present invention, and will not be repeated here.

[0219] In the third aspect, based on the same inventive concept, an embodiment of the present invention provides a computer-readable storage medium having an executable program stored thereon, which, when executed by a processor, implements the language model training method proposed in the first aspect of the present invention.

[0220] It should be noted that the specific implementation of the computer-readable storage medium of the embodiment of the present invention refers to the specific implementation of the language model training method proposed in the first aspect of the embodiment of the present invention, and will not be repeated here.

[0221] Fourthly, refer to Figure 3 Based on the same inventive concept, an embodiment of the present invention provides an electronic device 300, including:

[0222] Memory 301, used for storing executable programs;

[0223] Processor 302;

[0224] When the executable program is executed by the processor 302, the language model training method proposed in the first aspect of the present invention is implemented.

[0225] It should be noted that the specific implementation of the electronic device 300 in the embodiment of the present invention refers to the specific implementation of the language model training method proposed in the first aspect of the embodiment of the present invention, which will not be repeated here.

[0226] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0227] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0228] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0229] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0230] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0231] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0232] The above is a detailed introduction to a language model training method, device, storage medium and electronic device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A language model training method, characterized in that: The method comprises: Based on the basic data set, the initial language model is trained to obtain the first language model; Acquire a plurality of answer text pairs output by the first language model according to a plurality of prompt words; For the answer text pair corresponding to any prompt word, determine the factual score of each answer text in the answer text pair; mark the answer text with a higher factual score in the answer text pair as a preferred answer, and mark the answer text with a lower factual score in the answer text pair as a non-preferred answer, so as to construct a preference data set; Based on the preference data set and a preset factual reinforcement algorithm, the first language model is trained to obtain a target language model.

2. The language model training method according to claim 1, characterized in that: Based on the basic data set, the initial language model is trained to obtain the first language model, including: For any training text in the basic data set, obtaining a first predicted text output by the initial language model for the training text; Determine a first plural representation of the first predicted text in a plural space, and a second plural representation of the text label of the training text in the plural space; Determining an angle loss function value based on the first complex representation and the second complex representation; Based at least on the angle loss function value, the network parameters of the initial language model are updated to obtain the first language model.

3. The language model training method according to claim 2, characterized in that: Determining an angle loss function value based on the first complex number representation and the second complex number representation includes: determining an angular difference between the first complex representation and the second complex representation; Based on the angle difference and a preset angle loss function, the angle loss function value is determined.

4. The language model training method according to claim 2, characterized in that: The method further comprises: Determine a cosine value between the training text and the first predicted text, and determine a cosine loss function value based on the cosine value and a preset cosine loss function; Determine the training text as a positive sample, and randomly sample the basic data set to obtain a plurality of negative samples; determine the value of the intra-batch negative sample loss function based on the positive sample, the plurality of negative samples and a preset intra-batch negative sample loss function; At least based on the angle loss function value, updating the network parameters of the initial language model to obtain the first language model includes: Determine a first loss function value based on the angle loss function value, the cosine loss function value, and the in-batch negative sample loss function value; Based on the first loss function value, the network parameters of the initial language model are updated to obtain the first language model.

5. The language model training method according to claim 4, characterized in that: Determining a first loss function value based on the angle loss function value, the cosine loss function value, and the in-batch negative sample loss function value includes: Obtaining a first weight of the angle loss function value, a second weight of the cosine loss function value, and a third weight of the intra-batch negative sample loss function value; The first loss function value is determined based on the angle loss function value, the first weight, the cosine loss function value, the second weight, the in-batch negative sample loss function value and the third weight.

6. The language model training method according to claim 1, characterized in that: Based on the preference data set and a preset factual reinforcement algorithm, the first language model is trained to obtain a target language model, including: For any preference data in the preference data set, input the preference data into the first language model, and determine a first probability that the first language model generates the preferred answer and a second probability that the first language model generates the non-preferred answer; Determine a reinforcement algorithm loss function value based on the first probability, the second probability, and the reinforcement algorithm loss function corresponding to the factual reinforcement algorithm; Based at least on the loss function value of the reinforcement algorithm, the network parameters of the first language model are updated to obtain the target language model.

7. The language model training method according to claim 6, characterized in that: The method further comprises: For any preference data in the preference data set, input the prompt word of the preference data into the first language model, output a second predicted text, determine the cross entropy loss function value based on the generation probability of each word in the second predicted text; and determine the regularization term parameter based on the factual score of each word in the second predicted text; At least based on the loss function value of the reinforcement algorithm, updating the network parameters of the first language model to obtain the target language model includes: Determine a second loss function value based on the reinforcement algorithm loss function value, the cross entropy loss function value and the regularization term parameter; Based on the second loss function value, the network parameters of the first language model are updated to obtain the target language model.

8. A language model training device, characterized in that: The device comprises: A first training module, used to train the initial language model based on the basic data set to obtain a first language model; A text acquisition module, used to acquire a plurality of answer text pairs output by the first language model according to a plurality of prompt words; A data set construction module is used to determine the factuality score of each answer text in the answer text pair corresponding to any prompt word; mark the answer text with a higher factual score in the answer text pair as a preferred answer, and mark the answer text with a lower factual score in the answer text pair as a non-preferred answer, so as to construct a preference data set; The second training module is used to train the first language model based on the preference data set and a preset factual reinforcement algorithm to obtain a target language model.

9. A computer-readable storage medium having an executable program stored thereon, characterized in that: When the executable program is executed by the processor, the language model training method as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory for storing an executable program; processor; When the executable program is executed by the processor, the language model training method as described in any one of claims 1 to 7 is implemented.

Images