Language model optimization method and device, electronic equipment and storage medium

Abstract

The invention relates to a language model optimization method and device, electronic equipment and a storage medium, and the method comprises the steps: building a word frequency queue based on the occurrence frequency of each word in a training corpus; a first reference word and a second reference word of a target word are determined from the word frequency queue, and the second reference word is a word with the frequency higher than that of the target word in the word frequency queue; determining a positive example and a counter example of the target word from the first reference word and the second reference word, and constructing a contrast learning loss function based on the positive example and the counter example; according to the comparative learning loss function, parameters of a language model are iteratively updated, and the language model is constructed based on the training corpus. According to the method disclosed by the embodiment of the invention, the prediction effect of the optimized language model can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of natural language, and in particular to a method and device for optimizing a language model, an electronic device, and a storage medium. Background Art

[0002] The training goal of the causal language model is to predict the word at time t+1 based on the word at time t. In the prior art, the causal language model adopts this training method.

[0003] However, the causal language model focuses on the quality of overall language generation. The problem is that it is often difficult to express the embedded representation of low-frequency words in the corpus. Specifically, when the trained language model is verified using the test set, most of the mispredicted words are low-frequency words, while the prediction accuracy of high-frequency words is very high.

[0004] In the related art, for other types of language models, there is also the problem of low prediction accuracy for low-frequency words. Summary of the invention

[0005] In view of this, the present disclosure proposes a technical solution for optimizing a language model.

[0006] According to one aspect of the present disclosure, a method for optimizing a language model is provided, comprising:

[0007] Build a word frequency queue based on the frequency of occurrence of each word in the training corpus;

[0008] Determine a first reference word and a second reference word of a target word from the word frequency queue, wherein the second reference word is a word with a higher frequency than the target word in the word frequency queue;

[0009] Determining positive examples and negative examples of the target word from the first reference words and the second reference words, and constructing a contrastive learning loss function based on the positive examples and the negative examples;

[0010] According to the contrastive learning loss function, parameters of a language model are iteratively updated, and the language model is constructed based on the training corpus.

[0011] In a possible implementation, determining the first reference word and the second reference word of the target word from the word frequency queue includes:

[0012] From the word frequency queue, a word whose semantic distance to the target word is less than a preset semantic distance and a word whose semantic distance to the target word is greater than the preset semantic distance are determined as a first candidate word and a second candidate word respectively;

[0013] determining the first reference word from the first candidate words;

[0014] determining the second reference word from the second candidate words;

[0015] The step of determining a positive example and a negative example of the target word from the first reference word and the second reference word comprises:

[0016] The first reference word and the second reference word are determined as the positive example and the negative example respectively.

[0017] In a possible implementation manner, determining the first reference word from the first candidate words includes:

[0018] From the first candidate words, determine the word with the smallest frequency difference with the target word in the word frequency queue as the first reference word.

[0019] In a possible implementation, constructing the contrastive learning loss function includes:

[0020] Determine a semantic distance between a first target word vector of the target word and a first reference word vector of the first reference word as a first distance;

[0021] Determine the semantic distance between the first target word vector and the second reference word vector as a second distance; wherein the first target word vector, the first reference word vector and the second reference word vector are determined from a second word embedding matrix obtained by performing attention encoding on the first word embedding matrix of the language model;

[0022] The contrastive learning loss function is determined according to the first distance and the second distance.

[0023] In a possible implementation, determining the semantic distance between the first target word vector of the target word and the first reference word vector of the first reference word as the first distance includes:

[0024] Determine the semantic distance between the third target word vector and the first reference word vector as the first distance; wherein the third target word vector is obtained by concatenating the first target word vector and the second target word vector; and the second target word vector is obtained by connecting features of multiple network layers obtained by performing feature extraction on the first word embedding matrix to an output layer through the same pooling operation;

[0025] The step of determining the semantic distance between the first target word vector and the second reference word vector as a second distance includes:

[0026] The semantic distance between the third target word vector and the second reference word vector is determined as the second distance.

[0027] In a possible implementation, the iterative updating of the parameters of the language model according to the contrastive learning loss function includes:

[0028] Based on a first total loss function, the parameters of the language model are adjusted; wherein the first total loss function is obtained by adding a cross entropy loss function and the contrastive learning loss function; and the cross entropy loss function is determined based on a word index corresponding to the training corpus and the third target word vector.

[0029] In a possible implementation, the cross entropy loss function is calculated based on the normalized discounted cumulative gain;

[0030] The adjusting the parameters of the language model based on the first total loss function includes:

[0031] Obtaining a second total loss function according to the normalized discounted cumulative gain and the first total loss function;

[0032] According to the second total loss function, parameters of the language model are adjusted.

[0033] According to another aspect of the present disclosure, there is provided a language model optimization device, comprising:

[0034] A word frequency queue construction unit is used to construct a word frequency queue based on the frequency of occurrence of each word in the training corpus;

[0035] A reference word determination unit, used to determine a first reference word and a second reference word of a target word from the word frequency queue, wherein the second reference word is a word with a higher frequency than the target word in the word frequency queue;

[0036] A loss function construction unit, used to determine positive examples and negative examples of the target word from the first reference words and the second reference words, and to construct a contrastive learning loss function based on the positive examples and the negative examples;

[0037] An iterative updating unit is used to iteratively update the parameters of a language model according to the contrastive learning loss function, wherein the language model is constructed based on the training corpus.

[0038] In a possible implementation, the reference word determination unit is used to:

[0039] From the word frequency queue, a word whose semantic distance to the target word is less than a preset semantic distance and a word whose semantic distance to the target word is greater than the preset semantic distance are determined as a first candidate word and a second candidate word respectively;

[0040] determining the first reference word from the first candidate words;

[0041] determining the second reference word from the second candidate words;

[0042] The step of determining a positive example and a negative example of the target word from the first reference word and the second reference word comprises:

[0043] The first reference word and the second reference word are determined as the positive example and the negative example respectively.

[0044] In a possible implementation, the reference word determination unit is used to:

[0045] From the first candidate words, determine the word with the smallest frequency difference with the target word in the word frequency queue as the first reference word.

[0046] In a possible implementation, the loss function construction unit is used to:

[0047] Determine a semantic distance between a first target word vector of the target word and a first reference word vector of the first reference word as a first distance;

[0048] Determine the semantic distance between the first target word vector and the second reference word vector as a second distance; wherein the first target word vector, the first reference word vector and the second reference word vector are determined from a second word embedding matrix obtained by performing attention encoding on the first word embedding matrix of the language model;

[0049] The contrastive learning loss function is determined according to the first distance and the second distance.

[0050] In a possible implementation, the loss function construction unit is used to:

[0051] Determine the semantic distance between the third target word vector and the first reference word vector as the first distance; wherein the third target word vector is obtained by concatenating the first target word vector and the second target word vector; and the second target word vector is obtained by connecting features of multiple network layers obtained by performing feature extraction on the first word embedding matrix to an output layer through the same pooling operation;

[0052] The step of determining the semantic distance between the first target word vector and the second reference word vector as a second distance includes:

[0053] The semantic distance between the third target word vector and the second reference word vector is determined as the second distance.

[0054] In a possible implementation, the iterative updating unit is used to:

[0055] Based on a first total loss function, the parameters of the language model are adjusted; wherein the first total loss function is obtained by adding a cross entropy loss function and the contrastive learning loss function; and the cross entropy loss function is determined based on a word index corresponding to the training corpus and the third target word vector.

[0056] In a possible implementation, the cross entropy loss function is calculated based on the normalized discounted cumulative gain;

[0057] Iterative update unit, used to:

[0058] Obtaining a second total loss function according to the normalized discounted cumulative gain and the first total loss function;

[0059] According to the second total loss function, parameters of the language model are adjusted.

[0060] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0061] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0062] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0063] In the disclosed embodiment, a first reference word and a second reference word with a higher frequency than the target word are determined from the word frequency queue, and a positive example and a negative example of the target word are set according to the first reference word and the second reference word, and the language model is optimized by comparative learning. The optimization process of the language model introduces the frequency of words in the training corpus. Since the prediction accuracy of the second reference word with a higher frequency is higher than that of the target word, the language model optimization process using the second reference word as a positive example / negative example can improve the prediction accuracy of low-frequency words without changing the prediction accuracy of high-frequency words, thereby improving the prediction effect of the optimized language model.

[0064] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0065] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0067] Figure 1 A flowchart of a method for optimizing a language model according to an embodiment of the present disclosure is shown.

[0068] Figure 2 A schematic diagram showing extraction of a target object according to an embodiment of the present disclosure is shown.

[0069] Figure 3 A block diagram of a language model optimization device according to an embodiment of the present disclosure is shown.

[0070] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0071] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0072] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0073] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0074] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0075] Language model is the most basic task in natural language processing. In practical applications, input methods, intelligent customer service, and conversational robots are all direct applications of language models. In today's era of pre-trained large models and big data, language models are the cornerstone of the entire natural language processing task. Researchers train large-scale language models by training huge corpora, thereby fine-tuning downstream tasks. Large-scale language models have already played a great role in search engines, personalized recommendation systems, and advertising systems. Therefore, optimizing language models is a very important proposition.

[0076] The task of language model is to measure the probability of a sentence in a certain corpus. There are two main types of pre-training tasks, namely masked language model and causal language model. Masked language model is mainly used for language understanding tasks, while causal language model is mainly used for language generation tasks.

[0077] Currently, existing technologies for solving low-frequency words focus more on masked language models, such as Dict-BERT. How to significantly improve the low-frequency word prediction performance of the causal language model without affecting the high-frequency word prediction performance has become a technical problem that needs to be solved urgently. In order to better illustrate the present disclosure and highlight the main purpose of the present disclosure, the specific embodiments of this article use the causal language model as a description object. Those skilled in the art should understand that for other language models whose word prediction accuracy is related to the word frequency in the training corpus, the present disclosure can also be implemented flexibly.

[0078] Figure 1A flow chart of a method for optimizing a language model according to an embodiment of the present disclosure is shown, and the method can be applied to an optimization device for a language model, and the optimization device for a language model can be a terminal device, a server, or other processing devices, etc. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.

[0079] In some possible implementations, the language model optimization method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0080] like Figure 1 As shown, the optimization method of the language model may include:

[0081] In step S11, a word frequency queue is constructed based on the frequency of occurrence of each word in the training corpus.

[0082] The training corpus can be used in the training process of the language model before optimization. The disclosure does not limit the form of the training corpus, and can be a single sentence, a paragraph composed of sentences, or an article composed of paragraphs. The disclosure does not limit the source of the training corpus, and can use crawler technology to crawl news, comments, encyclopedias from various websites, as well as open source data sets on the Internet.

[0083] In one example, before using the training corpus, it is necessary to process the training corpus. Specifically, the processing process of the training corpus may include: corpus cleaning, word segmentation, stemming, stop word removal, and duplicate removal. Specifically, corpus cleaning may be to retain useful data and delete noisy data, including manual duplicate removal, alignment, deletion, annotation, etc.; word segmentation may be to decompose the original text in the text into fine-grained words, for example, by rule-based or statistics-based word segmentation methods. Among them, English is segmented according to spaces, and Chinese is segmented according to specific word groups; stemming may be to find the stem of a word and restore the word to its root; stop word removal may be to remove words that frequently appear in the corpus but do not provide much information in the document (for example, words like "a", "the", "ah", punctuation marks, modal particles, and words like "的" do not provide semantics in the document and will also add noise to the document data); duplicate removal may be to use a duplicate removal algorithm to remove duplicate corpus. Among them, lemmatization and stemming have roughly the same effect, both of which convert words into root forms. However, the difference is that the root words obtained by lemmatization often belong to valid words in the language, while the root words obtained by stemming may not. The present disclosure does not limit the selection of stemming and lemmatization, and can be selected according to the actual situation. In one example, the lightweight part-of-speech merging method of stemming can be used to improve the speed of obtaining the corpus.

[0084] In one example, the training corpus can be divided according to a certain ratio and divided into a training set, a validation set, and a test set respectively. The present disclosure does not specifically limit the ratio of the division of the training corpus, and can be determined according to the amount of the training corpus. In one example, the ratio of the training corpus divided into a training set, a validation set, and a test set can be 8:1:1.

[0085] Input the divided training set into the language model, use the validation set for parameter tuning, and after multiple iterations, the language model can be trained. The main work of the present invention is to optimize the trained language model.

[0086] The word frequency queue is a queue built based on the frequency of all words in the training corpus. Specifically, the test set in the training corpus can be input into the trained language model, and the words at each moment can be predicted, and the words predicted incorrectly can be recorded; the word frequencies of all correct words corresponding to the incorrectly predicted words are counted, and multiple dividing points are divided, such as 100, 200, 300..., which respectively represent the number of words with a word frequency less than 100 and the number of words with a word frequency less than 200, and so on. As many dividing points as possible are divided and a line graph is drawn; according to a large number of experiments, the slope of the line graph curve is getting smaller and smaller, indicating that the growth rate is getting slower and slower, and it can be known that most of the incorrectly predicted words are low-frequency words, and the prediction accuracy of high-frequency words is very high. This is because the amount of data of high-frequency words in the training corpus is large, so high-frequency words are more fully trained, while low-frequency words are just the opposite.

[0087] In one example, a line graph can be drawn based on the word frequency queue. Furthermore, the optimization effect can be observed through a line graph composed of the prediction results of the language model before and after optimization.

[0088] In step S12, a first reference word and a second reference word of the target word are determined from the word frequency queue, wherein the second reference word is a word in the word frequency queue with a higher frequency than the target word.

[0089] In step S13, positive examples and negative examples of the target word are determined from the first reference words and the second reference words, and a contrastive learning loss function is constructed based on the positive examples and negative examples.

[0090] In step S14, the parameters of the language model are iteratively updated according to the contrastive learning loss function, and the language model is constructed based on the training corpus.

[0091] The target word is a word with a relatively low frequency in the word frequency queue. The purpose of optimizing the language model in the present invention is to improve the prediction performance of low-frequency words in the language model. Specifically, at least one word whose prediction accuracy needs to be improved can be determined from the word frequency queue according to user needs as the target word.

[0092] The first reference word and the second reference word are words that serve as positive examples and negative examples of the target word in the contrastive learning loss function, respectively. The present disclosure does not specifically limit the method of determining positive examples and negative examples from the first reference word and the second reference word, and can be selected according to actual conditions. Specifically, the first reference word can be used as a positive example and the second reference word can be used as a negative example; or the first reference word can be used as a negative example and the second reference word can be used as a positive example.

[0093] Furthermore, the second reference word is a word in the word frequency queue with a higher frequency than the target word. In one example, based on the position of the target word in the word frequency queue, all words with a higher frequency than the target word can be determined as candidate words for the target word. Furthermore, a word can be randomly selected from the candidate words as the second reference word. The present disclosure does not specifically limit the method of obtaining the second reference word from the word frequency queue, and it can be selected according to actual conditions.

[0094] The present disclosure does not specifically limit the first reference word. In the word frequency queue, any word that can constitute a positive example and a negative example of the target word in the contrastive learning process together with the second reference word can be used. Specifically, the first reference word can be set to a word whose semantic distance with the target word is greater than the semantic distance between the second reference word and the target word, and then the first reference word and the second reference word can be used as the negative example and the positive example of the target word respectively; the first reference word can also be set to a word whose semantic distance with the target word is less than the semantic distance between the second reference word and the target word, and then the first reference word and the second reference word can be used as the positive example and the negative example of the target word respectively. Further, a contrastive learning loss function can be constructed based on the positive example and the negative example. Then, the parameters of the language model can be adjusted according to the contrastive learning loss function to optimize the language model.

[0095] Specifically, the contrastive learning loss function can be used to obtain the contrastive learning loss function value, and the language model is back-propagated according to the contrastive learning loss function value to calculate the gradient value of the language model, and then the gradient value is used to update the language model parameters.

[0096] In contrastive learning, when the batch size of the training corpus is large, the optimization process of the language model can cover more diverse negative examples, thereby providing meaningful learning performance. In one example, in order to train a neural network with a larger batch size, an activation check technique can be used, which can trade time for space without recording every intermediate state, and only needs to be recalculated during back propagation.

[0097] The above process is a single model training on a single graphics card, and the semantic generalization of the model needs to be demonstrated by a large model. Language generation ability is also one of the manifestations of semantic generalization ability. In order to train a large model, multiple graphics cards need to be used for parallel training. In one example, the model can be trained on multiple graphics cards simultaneously in a data parallel manner. Each graphics card retains the same model, and gradient propagation is performed through reduction operations. The average value is taken and then the model on each graphics card is updated. Through distributed training, the convergence speed of large models under big data can be accelerated, and the performance of the model can be greatly improved.

[0098] In the disclosed embodiment, a first reference word and a second reference word with a higher frequency than the target word are determined from the word frequency queue, and positive and negative examples of the target word are set according to the first reference word and the second reference word, and the language model is optimized by comparative learning. The optimization process of the language model introduces the frequency of words in the training corpus. Since the prediction accuracy of the second reference word with a higher frequency is higher than that of the target word, the language model optimization process using the second reference word as a positive / negative example can improve the prediction accuracy of low-frequency words without changing the prediction accuracy of high-frequency words, thereby improving the prediction effect of the optimized language model.

[0099] In a possible implementation, determining the first reference word and the second reference word of the target word from the word frequency queue includes:

[0100] From the word frequency queue, a word whose semantic distance to the target word is less than a preset semantic distance and a word whose semantic distance to the target word is greater than the preset semantic distance are determined as a first candidate word and a second candidate word respectively;

[0101] determining the first reference word from the first candidate words;

[0102] determining the second reference word from the second candidate words;

[0103] The step of determining a positive example and a negative example of the target word from the first reference word and the second reference word comprises:

[0104] The first reference word and the second reference word are determined as the positive example and the negative example respectively.

[0105] Specifically, a preset semantic distance can be set in the semantic distance between each word in the word frequency queue and the target word, and the semantic distance between each word and the target word can be divided by the preset semantic distance. In one example, a word in the word frequency queue whose semantic distance with the target word is less than the preset semantic distance can be determined as the first candidate word; and a word in the word frequency queue whose semantic distance with the target word is greater than the preset semantic distance can be determined as the second candidate word.

[0106] Furthermore, a first reference word can be determined from the first candidate word, and a second reference word can be determined from the second candidate word, and then the first reference word and the second reference word are used as positive examples and negative examples of the target word, respectively, to construct a contrastive learning loss function.

[0107] In one example, two preset semantic distances can also be set. Specifically, the words whose semantic distances with the target word are less than the first preset semantic distance can be set as the first candidate words; the words whose semantic distances with the target word are greater than the second preset semantic distance can be set as the second candidate words. Among them, the first preset semantic distance is less than the second preset semantic distance. The present disclosure can make the first reference word, which is a positive example of the target word, closer to the target word, and the distance between the second reference word, which is a negative example of the target word, and the target word is farther away, that is, the contrast between the positive example and the negative example of the target word is increased, thereby improving the effect of comparative learning.

[0108] In the disclosed embodiment, by presetting the semantic distance, the words in the word frequency queue are divided into the first candidate word and the second candidate word, and then the first reference word and the second reference word are determined from the first candidate word and the second candidate word respectively, and the first reference word and the second reference word are used as the positive example and the negative example of the target word, and the language model is optimized by comparative learning. Since the prediction accuracy of the second reference word is higher than that of the target word, the language model optimization process using the second reference word as the positive example / negative example improves the prediction accuracy of the low-frequency word while not changing the prediction accuracy of the high-frequency word, thereby improving the prediction effect of the optimized language model.

[0109] In a possible implementation manner, determining the first reference word from the first candidate words includes:

[0110] From the first candidate words, determine the word with the smallest frequency difference with the target word in the word frequency queue as the first reference word.

[0111] Among them, in the comparative learning process disclosed in the present invention, it is necessary to calculate the semantic distance between the target word and the reference word (the first reference word and the second reference word). As mentioned above, the predictive performance of a word in a language model is related to the frequency of the word. When the frequency difference between two words is the smallest, the semantic distance between the two words in the semantic space formed by the language model is closer to the true value. In one example, when determining the first reference word, the word with the smallest frequency difference with the target word can be selected from the first candidate words with semantics close to the target word as the first reference word, so that the semantic distance between the first reference word and the target word is closer.

[0112] In the disclosed embodiment, the first reference word is not only close to the semantics of the target word, but also close to the word frequency of the target word, so that the semantic distance between the target word and the first reference word is closer to the true value, that is, the target word is closer to the first reference word, and the corresponding target word is further away from the second reference word. This process increases the contrast between the positive and negative examples of the target word, thereby improving the prediction accuracy of the language model optimized by the contrastive learning loss function.

[0113] In a possible implementation, constructing the contrastive learning loss function includes:

[0114] Determine a semantic distance between a first target word vector of the target word and a first reference word vector of the first reference word as a first distance;

[0115] Determine a semantic distance between the first target word vector and a second reference word vector of the second reference word as a second distance; wherein the first target word vector, the first reference word vector, and the second reference word vector are determined from a second word embedding matrix obtained by performing attention encoding on the first word embedding matrix of the language model;

[0116] The contrastive learning loss function is determined according to the first distance and the second distance.

[0117] Among them, the first word embedding matrix can be obtained from the trained language model. The process of determining the first word embedding matrix can include: segmenting the cleaned training corpus into sentences, which can be based on punctuation marks such as periods and exclamation marks; dividing the entire training corpus into batches, packaging multiple sentences into a batch of sentences, and inputting the batch of sentences into the language model together; after the sentences are segmented by the word segmenter, they are converted into word indexes, and filled with [PAD] symbols to pad them to the maximum length, and converted into a tensor type; the word index is input into the embedding search layer, which finds the word embedding vector of the corresponding word according to the index, extracts the word embedding vector, and splices the word embedding vectors of all words in the training corpus into the first word embedding matrix.

[0118] Furthermore, the first word embedding matrix can be processed based on the attention mechanism. The process can include initializing the weight matrix, the query matrix, and the value matrix. Afterwards, the first word embedding matrix can be matrix multiplied with the weight matrix, the query matrix, and the value matrix to obtain three word embedding matrices representing different semantic spaces.

[0119] Among them, the weight matrix and the query matrix perform matrix multiplication to obtain the attention matrix, which represents the attention coefficient between each word, indicating that each word will pay attention to different words according to its own semantics; next, the attention matrix and the value matrix perform matrix multiplication, and after this operation, the second word embedding matrix encoded by attention will be obtained. Furthermore, after the matrix multiplication of the attention matrix and the value matrix, it can also be encoded by two layers of feedforward neural network, and processed by layer normalization and residual connection.

[0120] Since the causal language model is composed of multiple layers of identical networks, the above two steps of attention calculation and feedforward neural network encoding can be performed multiple times, and what is finally generated is the second word embedding matrix after multi-layer network encoding and with contextual semantics.

[0121] Furthermore, the vectors of the target word, the first reference word, and the second reference word can be determined from the second word embedding matrix, and then the contrastive learning loss function can be determined according to the first distance between the first target word vector and the first reference word vector, and the second distance between the first target word vector and the second reference word vector. The present disclosure does not specifically limit the method for determining the contrastive learning loss function according to the first distance and the second distance, and can be selected according to actual conditions.

[0122] In the embodiment of the present disclosure, since the attention mechanism can overcome the disadvantage that the system performance decreases with the increase of input length, the ability of the second word embedding matrix obtained by attention encoding the first word embedding matrix to capture long-range dependency information is enhanced, that is, it has contextual semantics, and ultimately the language model obtained by comparative learning based on the first distance and the second distance in the second word embedding matrix has better prediction ability.

[0123] In a possible implementation, determining the semantic distance between the first target word vector of the target word and the first reference word vector of the first reference word as the first distance includes:

[0124] Determine the semantic distance between the third target word vector and the first reference word vector as the first distance; wherein the third target word vector is obtained by concatenating the first target word vector and the second target word vector; and the second target word vector is obtained by connecting features of multiple network layers obtained by performing feature extraction on the first word embedding matrix to an output layer through the same pooling operation;

[0125] The step of determining the semantic distance between the first target word vector and the second reference word vector as a second distance includes:

[0126] The semantic distance between the third target word vector and the second reference word vector is determined as the second distance.

[0127] Specifically, in addition to using the attention mechanism to process the first word embedding matrix so that the obtained second word embedding matrix has contextual semantics, a neural network can also be used to fuse the features of the first word embedding matrix at multiple network levels. Specifically, the fusion method can be to connect the features of multiple network levels to the output layer through a unified pooling operation, integrate the results of the unified pooling operation through the output layer, obtain a combined vector, and then obtain the second target word vector based on the combined vector.

[0128] Since compressed interaction networks are often used in recommendation scenarios to automatically combine the features of recommendation data, in one example, compressed interaction networks can be used in the feature extraction of the aforementioned word embedding vectors. Specifically, the compressed interaction network can automatically combine the word embedding representations of each word in the first word embedding matrix at a high order through tensor calculations, so that the context of low-frequency words can fully interact, and finally encode through convolutional layers to fully model the interaction of word embedding representations, thereby further strengthening the output vector representation of low-frequency words.

[0129] In one example, a window language module using a compressed interaction network model can be set in the language model, and the window language module predicts the current word through the context to further optimize the language model. Through sampling, the window language model can learn low-frequency words multiple times.

[0130] Further, in one example, the first target word vector and the second target word vector can be concatenated to obtain a third target word vector. Furthermore, the first distance and the second distance can be obtained according to the third target word vector, so as to perform comparative learning and optimize the language model through the first distance and the second distance. It should be understood that the two processing methods of attention encoding and unified pooling operation on the first word embedding matrix can be implemented separately or jointly (such as the concatenation of the first target word vector and the second target word vector mentioned above), and the present disclosure does not make specific limitations on this.

[0131] In the disclosed embodiment, a unified pooling operation is performed so that the third target word vector not only has contextual semantics but also includes feature interactions of different orders, so that the context of low-frequency words can further fully interact. This process strengthens the output vector representation of low-frequency words, further improves the prediction effect of the language model on low-frequency words, and thus improves the prediction performance of the language model.

[0132] In a possible implementation, the iterative updating of the parameters of the language model according to the contrastive learning loss function includes:

[0133] Based on a first total loss function, the parameters of the language model are adjusted; wherein the first total loss function is obtained by adding a cross entropy loss function and the contrastive learning loss function; and the cross entropy loss function is determined based on a word index corresponding to the training corpus and the third target word vector.

[0134] Specifically, the performance (such as accuracy, etc.) of the language model after the attention mechanism and the unified pooling operation can be represented by the training loss. In one example, the training loss of this part can be calculated using the cross entropy loss function.

[0135] In one example, the cross entropy loss function can be added to the contrastive learning loss function to obtain a first total loss function. Typically, the training loss of a model is inversely proportional to the prediction accuracy of the model. The smaller the training loss, the higher the prediction accuracy of the model. For the present disclosure, the lower the total training loss obtained by the first total loss function of the language model, the higher the accuracy of the optimized language model. In one example, the trained language model can be back-propagated based on the total training loss obtained to calculate the gradient value of the language model, and then the gradient value can be used to update the parameters of the language model.

[0136] In the disclosed embodiment, the loss function is back-propagated by combining the cross entropy loss function and the contrast loss function to obtain the first total loss function, thereby optimizing the trained language model.

[0137] In a possible implementation, the cross entropy loss function is calculated based on the normalized discounted cumulative gain;

[0138] The adjusting the parameters of the language model based on the first total loss function includes:

[0139] Obtaining a second total loss function according to the normalized discounted cumulative gain and the first total loss function;

[0140] According to the second total loss function, parameters of the language model are adjusted.

[0141] Specifically, while using the concatenated vector to calculate the cross entropy loss function, the normalized discounted cumulative gain (NDCG) can be calculated. After obtaining the first total loss function, the NDCG is multiplied by the first total loss function to obtain the second total loss function. The second total loss function can then be used to adjust the language model parameters to achieve language model optimization.

[0142] In the disclosed embodiment, since the NDCG indicator can reduce the importance and weight of words as the frequency increases, the NDCG indicator is added to the first total loss function in the form of multiplication, so that the optimized language model can consider the frequency of words as a whole, thereby further improving the prediction accuracy of the language model.

[0143] Application scenario examples

[0144] The problem with causal language models is that it is often difficult to express low-frequency words in the corpus well in terms of embedded representation. Specifically, when the trained causal language model is verified in the test set, most of the words that are predicted incorrectly are low-frequency words, while the prediction accuracy of high-frequency words is very high.

[0145] Figure 2 A schematic diagram showing an application example according to the present disclosure is shown as follows: Figure 2 As shown, the embodiment of the present disclosure proposes an optimization method for a causal language model. This optimization method can improve the prediction effect of low-frequency words by modeling the concept of frequency in the causal language model. The optimization process of the causal language model can be:

[0146] like Figure 2 As shown, the optimization process of the causal language model can be roughly divided into five steps.

[0147] The first step is to obtain the first word embedding matrix of the trained causal language model. Specifically, the causal language model is trained through the training corpus, and the word index converted from the training corpus is input into the embedding search layer of the trained causal language model to obtain the word embedding vector of each word in the training corpus, and the word embedding vectors are spliced ​​into the first word embedding matrix.

[0148] The second step is to determine the cross entropy loss function. Specifically, the first word embedding matrix is ​​input into the attention processing module and the window language module of the causal language model respectively, and the first target word vector corresponding to the target word is obtained according to the attention network in the attention processing module, and the second target word vector corresponding to the target word is obtained according to the compression interaction network in the window language module, and the first target word vector and the second target word vector are concatenated into a third target word vector. According to the word index and the third target word vector, the cross entropy loss function is determined. While calculating the cross entropy loss function, the NDCG indicator is calculated.

[0149] The third step is to determine the contrastive learning loss function. Specifically, a word frequency queue is constructed according to the frequency of occurrence of each word in the training corpus. The word in the word frequency queue whose semantic distance to the target word is less than the preset semantic distance and whose frequency is closest to the target word is determined as the first reference word; from the words whose semantic distance to the target word is greater than the preset semantic distance, a word with a higher frequency than the target word is randomly selected as the second reference word. The contrastive learning loss function is determined based on the first distance between the third target word vector and the first reference word, and the second distance between the third target word vector and the second reference word.

[0150] Step 4: Determine the total training loss. Specifically, add the cross entropy loss function and the contrastive learning loss function to get the first total loss function, and multiply the first total loss function by the NDCG index to get the second total loss function. Calculate the total training loss based on the value output by the second total loss function.

[0151] Step 5: Model optimization. Specifically, according to the total training loss, the causal language model is back-propagated, the gradient value of the causal language model is calculated, and the gradient value is used to update the parameters of the causal language model.

[0152] Before and after the optimization of the causal language model, the word frequencies of all correct words corresponding to the predicted wrong words can be counted respectively, and multiple dividing points, such as 100, 200, 300..., respectively represent the number of words with a word frequency less than 100, the number of words with a word frequency less than 200, and so on, divide as many dividing points as possible, and draw a line graph. It can be seen from the line graph that the slope of the curve of the optimized line graph is increased compared with the slope of the line graph before optimization, indicating that with the increase of word frequency, the growth rate of the number of predicted wrong words is accelerated, and the difference in prediction accuracy between low-frequency words and high-frequency words becomes smaller, that is, the optimization method of the causal language model disclosed in the present invention can effectively improve the prediction accuracy of low-frequency words, thereby improving the prediction effect of the optimized causal language model.

[0153] In the disclosed embodiment, a first reference word and a second reference word with a higher frequency than the target word are determined from the word frequency queue, and positive and negative examples of the target word are set according to the first reference word and the second reference word, and the language model is optimized by comparative learning. The optimization process of the language model introduces the frequency of words in the training corpus. Since the prediction accuracy of the second reference word with a higher frequency is higher than that of the target word, the language model optimization process using the second reference word as a positive / negative example can improve the prediction accuracy of low-frequency words without changing the prediction accuracy of high-frequency words, thereby improving the prediction effect of the optimized language model.

[0154] It should be noted that the language model optimization method of the embodiment of the present disclosure is not limited to application in the optimization of the above-mentioned causal language model, but can be applied to any language model, and the present disclosure does not impose any limitation on this.

[0155] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0156] In addition, the present disclosure also provides a language model optimization device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any language model optimization method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0157] Figure 3 A block diagram of an optimization device for a language model according to an embodiment of the present disclosure is shown. The optimization device for the language model may be a terminal device, a server, or other processing devices, etc. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.

[0158] In some possible implementations, the language model optimization device may be implemented by a processor calling computer-readable instructions stored in a memory.

[0159] like Figure 3 As shown, the language model optimization device 30 may include:

[0160] A word frequency queue construction unit 31 is used to construct a word frequency queue based on the frequency of occurrence of each word in the training corpus;

[0161] A reference word determination unit 32, used to determine a first reference word and a second reference word of a target word from the word frequency queue, wherein the second reference word is a word with a higher frequency than the target word in the word frequency queue;

[0162] A loss function construction unit 33, used to determine positive examples and negative examples of the target word from the first reference words and the second reference words, and to construct a contrastive learning loss function based on the positive examples and the negative examples;

[0163] The iterative updating unit 34 is used to iteratively update the parameters of the language model according to the contrastive learning loss function, and the language model is constructed based on the training corpus.

[0164] In a possible implementation, the reference word determination unit is used to:

[0165] From the word frequency queue, a word whose semantic distance to the target word is less than a preset semantic distance and a word whose semantic distance to the target word is greater than the preset semantic distance are determined as a first candidate word and a second candidate word respectively;

[0166] determining the first reference word from the first candidate words;

[0167] determining the second reference word from the second candidate words;

[0168] The step of determining a positive example and a negative example of the target word from the first reference word and the second reference word comprises:

[0169] The first reference word and the second reference word are determined as the positive example and the negative example respectively.

[0170] In a possible implementation, the reference word determination unit is used to:

[0171] From the first candidate words, determine the word with the smallest frequency difference with the target word in the word frequency queue as the first reference word.

[0172] In a possible implementation, the loss function construction unit is used to:

[0173] Determine a semantic distance between a first target word vector of the target word and a first reference word vector of the first reference word as a first distance;

[0174] Determine the semantic distance between the first target word vector and the second reference word vector as a second distance; wherein the first target word vector, the first reference word vector and the second reference word vector are determined from a second word embedding matrix obtained by performing attention encoding on the first word embedding matrix of the language model;

[0175] The contrastive learning loss function is determined according to the first distance and the second distance.

[0176] In a possible implementation, the loss function construction unit is used to:

[0177] Determine the semantic distance between the third target word vector and the first reference word vector as the first distance; wherein the third target word vector is obtained by concatenating the first target word vector and the second target word vector; and the second target word vector is obtained by connecting features of multiple network layers obtained by performing feature extraction on the first word embedding matrix to an output layer through the same pooling operation;

[0178] The step of determining the semantic distance between the first target word vector and the second reference word vector as a second distance includes:

[0179] The semantic distance between the third target word vector and the second reference word vector is determined as the second distance.

[0180] In a possible implementation, the iterative updating unit is used to:

[0181] Based on a first total loss function, the parameters of the language model are adjusted; wherein the first total loss function is obtained by adding a cross entropy loss function and the contrastive learning loss function; and the cross entropy loss function is determined based on a word index corresponding to the training corpus and the third target word vector.

[0182] In a possible implementation, the cross entropy loss function is calculated based on the normalized discounted cumulative gain;

[0183] Iterative update unit, used to:

[0184] Obtaining a second total loss function according to the normalized discounted cumulative gain and the first total loss function;

[0185] According to the second total loss function, the parameters of the language model are adjusted. In some embodiments, the functions or modules included in the apparatus provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment, and its specific implementation can refer to the description of the above method embodiment, which will not be repeated here for the sake of brevity.

[0186] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0187] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0188] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0189] Figure 4 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the device 1900 may be provided as a server or a terminal device. Figure 4 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0190] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0191] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0192] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0193] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0194] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0195] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0196] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0197] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0198] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0199] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0200] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for optimizing a language model, characterized in that: include: Build a word frequency queue based on the frequency of occurrence of each word in the training corpus; Determine a first reference word and a second reference word of a target word from the word frequency queue, wherein the second reference word is a word with a higher frequency than the target word in the word frequency queue; Determining positive examples and negative examples of the target word from the first reference words and the second reference words, and constructing a contrastive learning loss function based on the positive examples and the negative examples; According to the contrastive learning loss function, parameters of a language model are iteratively updated, and the language model is constructed based on the training corpus.

2. The method according to claim 1, characterized in that: Determining the first reference word and the second reference word of the target word from the word frequency queue includes: From the word frequency queue, a word whose semantic distance to the target word is less than a preset semantic distance and a word whose semantic distance to the target word is greater than the preset semantic distance are determined as a first candidate word and a second candidate word respectively; determining the first reference word from the first candidate words; determining the second reference word from the second candidate words; The step of determining a positive example and a negative example of the target word from the first reference word and the second reference word comprises: The first reference word and the second reference word are determined as the positive example and the negative example respectively.

3. The method according to claim 2, characterized in that The step of determining the first reference word from the first candidate words comprises: From the first candidate words, determine the word with the smallest frequency difference with the target word in the word frequency queue as the first reference word.

4. The method according to claim 2, characterized in that: The constructing of the contrastive learning loss function includes: Determine a semantic distance between a first target word vector of the target word and a first reference word vector of the first reference word as a first distance; Determine a semantic distance between the first target word vector and a second reference word vector of the second reference word as a second distance; wherein the first target word vector, the first reference word vector, and the second reference word vector are determined from a second word embedding matrix obtained by performing attention encoding on the first word embedding matrix of the language model; The contrastive learning loss function is determined according to the first distance and the second distance.

5. The method according to claim 4, characterized in that The step of determining the semantic distance between the first target word vector of the target word and the first reference word vector of the first reference word as a first distance includes: Determine the semantic distance between the third target word vector and the first reference word vector as the first distance; wherein the third target word vector is obtained by concatenating the first target word vector and the second target word vector; and the second target word vector is obtained by connecting features of multiple network layers obtained by performing feature extraction on the first word embedding matrix to an output layer through the same pooling operation; The step of determining the semantic distance between the first target word vector and the second reference word vector as a second distance includes: The semantic distance between the third target word vector and the second reference word vector is determined as the second distance.

6. The method according to claim 5, characterized in that The iterative updating of the parameters of the language model according to the contrastive learning loss function comprises: Based on a first total loss function, the parameters of the language model are adjusted; wherein the first total loss function is obtained by adding a cross entropy loss function and the contrastive learning loss function; and the cross entropy loss function is determined based on a word index corresponding to the training corpus and the third target word vector.

7. The method according to claim 6, characterized in that The cross entropy loss function is calculated based on the normalized discounted cumulative gain; The adjusting the parameters of the language model based on the first total loss function includes: Obtaining a second total loss function according to the normalized discounted cumulative gain and the first total loss function; According to the second total loss function, parameters of the language model are adjusted.

8. A language model optimization device, characterized in that: include: A word frequency queue construction unit is used to construct a word frequency queue based on the frequency of occurrence of each word in the training corpus; A reference word determination unit, used to determine a first reference word and a second reference word of a target word from the word frequency queue, wherein the second reference word is a word with a higher frequency than the target word in the word frequency queue; A loss function construction unit, used to determine positive examples and negative examples of the target word from the first reference words and the second reference words, and to construct a contrastive learning loss function based on the positive examples and the negative examples; An iterative updating unit is used to iteratively update the parameters of a language model according to the contrastive learning loss function, wherein the language model is constructed based on the training corpus.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images