Method for training a text prediction model, text prediction method and device

Abstract

Embodiments of the present application disclose a method for training a text prediction model, a text prediction method and device. The present application uses a training sample set to train a large language model to obtain a text prediction model. This method essentially uses the annotated samples to train the large language model (LLM). In the updating process of the second key matrix and the second value matrix, the first key matrix and the first value matrix generated by the second key matrix and the second value matrix obtained in the last iteration and the current input feature matrix are used. The historical information is retained and the information of the current input text is maintained, so that the large language model can fully understand and learn the annotated samples, thereby improving the text prediction effect based on the large language model in the scenario learning scene. Moreover, this forward optimization model greatly reduces the model parameters that need to be updated, reduces the cost of model training, and improves the efficiency.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method for training a text prediction model, a text prediction method, and a device. Background Art

[0002] The goal of NLP (Natural Language Processing) research is to enable machines to understand human language. Large Language Models (LLMs) are a core tool in the field of NLP. These are deep learning models with large numbers of parameters (typically hundreds of millions or more).

[0003] LLMs are widely used in the field of text prediction due to their exceptional learning capabilities. In-context learning is one of the text prediction methods currently employed by LLMs. Given labeled data, LLMs observe and summarize it to make predictions on unlabeled data. Because in-context learning typically involves no training, its performance is poor when applied to text prediction. Therefore, there is an urgent need for methods to improve the performance of LLM-based text prediction in context learning scenarios. Summary of the Invention

[0004] In view of this, the present application provides a method for training a text prediction model, a text prediction method and an apparatus, so as to improve the text prediction effect based on LLM in a situational learning scenario.

[0005] This application provides the following solutions:

[0006] In a first aspect, a method for training a text prediction model is provided, the method comprising:

[0007] Obtaining a training data set, wherein the training data set includes an input text sample and an output label corresponding to the input text sample;

[0008] Using a text sequence including an input text sample and an output label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model uses a large language model, and the large language model includes a multi-layer serialized transformer network;

[0009] In the training, each Transformer network is executed as the current layer Transformer network respectively: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix and the second key matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third key matrix, and the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network; the first key matrix is ​​used to update the second key matrix, and the updated second key matrix is ​​used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the first value matrix is ​​used to update the second value matrix, and the updated second value matrix is ​​used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

[0010] According to an achievable method in an embodiment of the present application, the large language model further includes an embedding network, and the text prediction model further includes a prediction network;

[0011] The embedding network is used to embed the text sequence;

[0012] If the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network;

[0013] The prediction network is used to predict the output label corresponding to the input text sample using the feature representation output by the last layer of the Transformer network.

[0014] According to an achievable method in an embodiment of the present application, updating the second bond matrix using the first bond matrix includes: using the first bond matrix to update the second bond matrix obtained by the current layer Transformer network in the previous iteration by using momentum gradient descent;

[0015] Updating the second value matrix using the first value matrix includes: using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration using momentum gradient descent.

[0016] According to an implementable method in an embodiment of the present application, using the first bond matrix to update the second bond matrix obtained by the current layer Transformer network in the previous round of iteration using a momentum gradient descent method includes: using the first bond matrix and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain a bond matrix gradient; using the bond matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the bond matrix gradient to perform weighted summation to obtain the bond matrix momentum obtained by the current layer Transformer network in the current round of iteration; using the bond matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the updated second bond matrix;

[0017] Using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration using momentum gradient descent includes: using the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain the value matrix gradient; using the value matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the value matrix gradient to perform weighted summation to obtain the value matrix momentum obtained by the current layer Transformer network in the current round of iteration; using the value matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the updated second value matrix.

[0018] According to an achievable method in an embodiment of the present application, after each round of iteration is completed, if it is determined that the preset training end condition is met, the second key matrix and the second value matrix obtained from the current iteration of each layer of the Transformer network are respectively stored as the second key matrix and the second value matrix of each layer of the Transformer network obtained by training.

[0019] In a second aspect, a text prediction method is provided, the method comprising:

[0020] Get input text;

[0021] Inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes a multi-layer serialized Transformer network;

[0022] Each Transformer network is executed as the current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

[0023] According to an achievable method in an embodiment of the present application, the large language model further includes an embedding network, and the text prediction model further includes a prediction network;

[0024] The embedding network is used to embed the text sequence;

[0025] If the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network;

[0026] The prediction network is used to predict the output label corresponding to the input text using the feature representation output by the last layer of the Transformer network.

[0027] In a third aspect, a method for training a text prediction model is provided, the method comprising:

[0028] Obtaining a training data set, wherein the training data set includes an input text sample and a sentiment category label corresponding to the input text sample;

[0029] Using a text sequence including an input text sample and a sentiment category label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model uses a large language model, and the large language model includes a multi-layer serialized Transformer network;

[0030] In the training, each Transformer network is executed as the current layer Transformer network respectively: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix and the second key matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third key matrix, and the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network; the first key matrix is ​​used to update the second key matrix, and the updated second key matrix is ​​used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the first value matrix is ​​used to update the second value matrix, and the updated second value matrix is ​​used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

[0031] In a fourth aspect, a sentiment analysis method is provided, the method comprising:

[0032] Get input text;

[0033] Inputting a text sequence containing the input text into a text prediction model, and obtaining a sentiment category label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes a multi-layer serialized Transformer network;

[0034] Each Transformer network is executed as the current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

[0035] In a fifth aspect, a text prediction method is provided, which is executed by a cloud server, and the method includes:

[0036] Get input text from the user terminal;

[0037] Inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model;

[0038] Determining corresponding service content based on the output tag, and sending the service content to the user terminal;

[0039] The text prediction model adopts a large language model, which includes a multi-layer serialized Transformer network;

[0040] Each Transformer network is executed as the current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

[0041] In a sixth aspect, a device for training a text prediction model is provided, the device comprising:

[0042] A sample acquisition unit is configured to acquire a training data set, wherein the training data set includes an input text sample and an output label sample corresponding to the input text sample;

[0043] The model training unit is configured to use a text sequence including an input text sample and an output label corresponding to the input text sample as an input of a text prediction model to train the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes a multi-layer serial conversion Transformer network; in the training, each Transformer network is respectively executed as a current layer Transformer network: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​spliced ​​with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third key matrix Matrix, and concatenate the first value matrix with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third value matrix; use the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation of the output of the current layer Transformer network; use the first key matrix to update the second key matrix, and use the updated second key matrix as the second key matrix obtained by the current layer Transformer network in the current round of iteration; use the first value matrix to update the second value matrix, and use the updated second value matrix as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

[0044] In a seventh aspect, a text prediction device is provided, the device comprising:

[0045] A text acquisition unit configured to acquire input text;

[0046] The text prediction unit is configured to input a text sequence containing the input text into a text prediction model, and obtain an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes a multi-layer serial conversion Transformer network; wherein each Transformer network is executed as a current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

[0047] According to an eighth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0048] According to a ninth aspect, there is provided an electronic device, comprising:

[0049] one or more processors; and

[0050] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0051] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0052] 1) This application uses a training sample set to train LLM to obtain a text prediction model. This method actually uses labeled samples to train LLM. In the process of updating the second key matrix and the second value matrix, the first key matrix and the first value matrix generated by the second key matrix and the second value matrix obtained in the previous round of iteration and the current input feature matrix are used. This not only retains historical information but also maintains the information of the current input text, so that LLM can fully understand and learn the labeled samples, thereby improving the text prediction effect of LLM.

[0053] 2) In the LLM training process, the second key matrix and second value matrix used in each iteration are obtained from the previous iteration and are updated for use in the next iteration. This forward optimization model only requires optimizing the second key matrix and second value matrix of each Transformer network, greatly reducing the model parameters that need to be updated, reducing the cost of model training, and improving efficiency.

[0054] 3) In the embodiment of the present application, the second key matrix and the second value matrix of each Transformer network are updated by using momentum gradient descent, which can speed up the gradient descent, make the iteration more efficient, and avoid falling into the local minimum.

[0055] 4) In the prediction process, this application uses the second key matrix and second value matrix of each Transformer network obtained after training the text prediction model. Compared with ordinary scenario learning, the second key matrix and second value matrix of each Transformer network contain the text prediction model's better observation and understanding of the labeled sample data, which can significantly improve the prediction accuracy.

[0056] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 This is a schematic diagram of the principle of traditional LLM-based scenario learning;

[0059] Figure 2 A diagram of the system architecture applicable to the embodiments of the present application;

[0060] Figure 3 A flowchart of a method for training a text prediction model provided in an embodiment of the present application;

[0061] Figure 4 A structural diagram of a text prediction model provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of a Transformer network provided in an embodiment of the present application;

[0063] Figure 6 A flowchart of a text prediction method provided in an embodiment of the present application;

[0064] Figure 7 A schematic diagram of the principle of scenario learning based on LLM provided in an embodiment of the present application;

[0065] Figure 8 A schematic block diagram of a model training device provided in an embodiment of the present application;

[0066] Figure 9 A schematic block diagram of a text prediction device provided in an embodiment of the present application;

[0067] Figure 10 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0069] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0070] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0071] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0072] The traditional LLM-based situational learning process is as follows Figure 1 As shown in [1], labeled sample data and the input text to be predicted are fed into the LLM. The LLM observes and summarizes the labeled sample data, and then predicts the output label corresponding to the input text. This scenario learning process does not involve a training phase, and the text prediction effect is poor.

[0073] If a text prediction model is pre-trained based on LLM using traditional backpropagation training, that is, using labeled sample data to train the LLM and using the loss function to generate gradients to update the model parameters, the traditional backpropagation training method will be extremely costly and inefficient due to the large number of LLM model parameters.

[0074] In view of this, the present application proposes a new approach, which uses a forward training method to pre-train a text prediction model based on LLM. In order to facilitate the understanding of the present application, the system architecture on which the present application is based is first briefly described. Figure 2 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 2 As shown in , the system architecture includes a model training device for training a text prediction model in an offline manner, and a text prediction device for predicting input text online.

[0075] Among them, after obtaining the training data set, the model training device can use the method provided in the embodiment of the present application to perform model training to obtain a text prediction model.

[0076] The text prediction device uses the trained text prediction model to predict the input text, and the prediction result is the output label corresponding to the input text. For example, the input text is predicted to obtain categories such as emotion and intent.

[0077] The model training device and text prediction device can be deployed on separate servers, on the same server or server cluster, or on separate or shared cloud servers. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service ecosystem. It addresses the management difficulties and limited scalability of traditional physical hosts and virtual private servers (VPS). The model training device and text prediction device can also be deployed on a computer terminal with significant computing power.

[0078] In addition to performing text prediction online, the text prediction device can also perform text prediction offline, for example, performing batch text prediction on texts in a database as input texts, and storing the prediction results in the database for subsequent query or call.

[0079] It should be understood that Figure 2 The number of model training devices, text prediction devices, and text prediction models in the embodiment is merely illustrative. Any number of model training devices, text prediction devices, and text prediction models may be provided according to implementation requirements.

[0080] Figure 3 A flowchart of a method for training a text prediction model provided in an embodiment of the present application, which can be performed by Figure 2 The model training device in the system shown is executed. Figure 3 As shown in , the method may include the following steps:

[0081] Step 302: Obtain a training data set, where the training data set includes an input text sample and an output label sample corresponding to the input text sample.

[0082] Step 304: Using a text sequence including an input text sample and an output label corresponding to the input text sample as input to a text prediction model to train the text prediction model; wherein the text prediction model adopts an LLM, which includes a multi-layer serially connected Transformer network. During training, each Transformer network is executed as the current layer Transformer network: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the third key matrix, and the first value matrix is ​​concatenated with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network; the first key matrix is ​​used to update the above second key matrix, and the updated second key matrix is ​​used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the first value matrix is ​​used to update the above second value matrix, and the updated second value matrix is ​​used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

[0083] It can be seen from the above process that the present application uses the training sample set to train the LLM to obtain a text prediction model. This method actually uses the labeled samples to train the LLM. In the process of updating the second key matrix and the second value matrix, the first key matrix and the first value matrix generated by the second key matrix and the second value matrix obtained in the previous round of iteration and the current input feature matrix are used. This not only retains historical information but also maintains the information of the current input text, so that the LLM can fully understand and learn the labeled samples, thereby improving the text prediction effect of the LLM.

[0084] Furthermore, during LLM training, the second key matrix and second value matrix used in each iteration are obtained from the previous iteration and updated for the next iteration. This forward optimization model only requires optimizing the second key matrix and second value matrix of each Transformer network, significantly reducing the number of model parameters that need to be updated, lowering the cost of model training, and improving efficiency.

[0085] It should be noted that the terms "first" and "second" in this disclosure do not restrict size, order, or quantity, but are merely used to distinguish between them in terms of name. For example, "first key matrix," "second key matrix," and "third key matrix" are used to distinguish three key matrices in terms of name. Another example is "first value matrix," "second value matrix," and "third value matrix" are used to distinguish three value matrices in terms of name.

[0086] Each step in the above process is described in detail below. First, the above step 302, ie, "obtaining a training data set," is described in detail in conjunction with an embodiment.

[0087] The training data set obtained in this step includes multiple training samples. The training samples include input text samples and output label samples corresponding to the input text samples. Each training sample can be expressed as (x i ,y i ), where x i is the i-th input text sample, y i is x i The corresponding output label sample is x i The output labels for the annotation.

[0088] The input text samples mentioned above can be sentences, paragraphs, articles, phrases, and so on. The output labels can correspond to different content in different application scenarios and can be category labels or text labels. For example, in a sentiment analysis scenario, the output label can be a sentiment type label, i.e., the sentiment type annotated for the input text sample. Another example is an intent recognition scenario, where the output label can be an intent type label, i.e., the intent type annotated for the input text sample. Another example is an automatic question-answering scenario, where the output label can be an answer text label, i.e., the answer text annotated for the question, treating the input text sample as a question.

[0089] The above-mentioned output label samples can be annotated manually or automatically using existing mining methods, and this application does not impose any restrictions on this.

[0090] The following describes in detail the above step 304, i.e., "using a text sequence including an input text sample and an output label corresponding to the input text sample as an input of a text prediction model to train the text prediction model," in conjunction with an embodiment.

[0091] In the embodiment of the present application, the text prediction model is implemented based on LLM, which is a pre-trained large language model. The embodiment of the present application further optimizes the model based on LLM so that LLM can quickly understand the labeled sample set training data set. In order to make full use of the existing language understanding ability of LLM, the text sequence is constructed using each input text sample and the output label corresponding to the input text sample. For example, the input text sample x is used as the training data set. i And the corresponding output label y i Splice to get the text sequence T i Input text prediction model is LLM. i and y i When x i and y i For example, the input text sample "The food is so delicious" and the output label "positive" can form the text sequence "Evaluation: The food is so delicious. Sentiment: Positive". "Evaluation" and "Sentiment" are indicative information of the input text sample and the output label.

[0092] The overall framework of the text prediction model used in the embodiment of the present application can be as follows: Figure 4 As shown in , it mainly includes feature extraction network and prediction network.

[0093] The feature extraction network is used to extract feature representations from the input text sequence. The prediction network is used to predict the output label of the input text sample using the feature representations extracted by the feature extraction network. In addition to this framework, other frameworks such as encoder-decoder frameworks can also be used. We will not list them one by one here, but all frameworks are based on LLM implementation, such as feature extraction network based on LLM implementation, encoder based on LLM implementation, etc. In the embodiment of this application, only LLM is used. Figure 4 The framework shown is taken as an example.

[0094] Since the text prediction model of the present application is implemented based on LLM, the feature extraction network can specifically include an embedding network and a multi-layer serial Transformer network.

[0095] The embedding network is used to perform embedding processing on the input text sequence. More specifically, the embedding processing can be performed on each token (element) of the input text sequence. Each token in a text sequence refers to the element that constitutes the text sequence. For a text sequence, the text sequence is divided into character or word sequences, and the characters or words, as well as the start symbol and separator in the text sequence are all tokens.

[0096] The token-based embedding process mentioned above includes at least word embedding and position embedding. Word embedding involves encoding each token into a word vector to obtain a word vector representation. Position embedding involves encoding the position of each token in the text sequence to be predicted to obtain a position representation.

[0097] The Transformer network is a model that uses a self-attention mechanism to encode each input token to convert it into a feature representation. The forward learning method provided in the embodiment of the application only requires modifying the self-attention part in the Transformer.

[0098] Figure 5 A schematic diagram of a Transformer network provided in an embodiment of the present application is shown in FIG. Figure 5 As shown in the figure, each Transformer network includes "horizontal" processing of feature representations and "vertical" processing of model parameters. It should be noted that the "horizontal" and "vertical" directions here are just for convenience of description and do not have actual directional meanings.

[0099] like Figure 5 In the "horizontal" processing process, the current Transformer network first uses the feature representation output by the previous layer of network Determine the first bond matrix K X , the first value matrix V X And the first query matrix Q. If the current Transformer network is the first layer Transformer network, then the above is the feature representation of the embedding network output. If the current Transformer network is not the first layer Transformer network, then the above It represents the feature representation of the output of the previous layer of Transformer network. In the figure, the current layer of Transformer network is the lth layer of Transformer network as an example.

[0100] The subscript t in the figure represents the tth iteration in the text prediction model training process. Each iteration collects a batch of training samples from the training dataset and uses them as input for text prediction model training. This process is similar to traditional model training and will not be described in detail here.

[0101] The first bond matrix K X , the first value matrix V X and the first query matrix Q X It can be determined using the following formula:

[0102]

[0103]

[0104]

[0105] Among them, W Q 、W k and W v is the weight matrix, which is the parameter that LLM has learned during the pre-training process.

[0106] Then the first key matrix K X The second key matrix obtained by the current layer Transformer network in the previous round of iteration Splice to get the third key matrix K, and the first value matrix V X The second value matrix obtained by the current layer Transformer network in the previous round of iteration The third value matrix V is obtained by concatenation.

[0107] Among them, K and V can be expressed as:

[0108]

[0109]

[0110] Among them, || represents the splicing process.

[0111] The second key matrix and second value matrix of each Transformer network are model parameters that need to be learned during the forward training process. They are continuously updated in each round of iteration. Each Transformer network corresponds to a second key matrix and a second value matrix.

[0112] Then use the third key matrix K, the third value matrix V and the first query matrix Q to process the self-attention mechanism to obtain the feature representation of the current layer Transformer network output The processing of the self-attention mechanism is the processing of the attention module and the FFN module, which will not be detailed here and can be expressed as:

[0113]

[0114] Among them, Transformer() is the processing function of the attention module and FFN module.

[0115] In the above “horizontal” processing, the first bond matrix K X The second key matrix obtained by the current layer Transformer network in the previous round of iteration Splice and convert the first value matrix V X The second value matrix obtained by the current layer Transformer network in the previous round of iteration Splicing can be seen as mixing the information of "past and present" to obtain the third key matrix K and the third value matrix V, which is in line with the essence of situational learning.

[0116] For the “vertical” processing, we first use the first key matrix K X For the second bond matrix Update the updated second key matrix As the second key matrix obtained by the current layer Transformer network in the current round of iteration, it is used for the next round of iteration. X For the second value matrix Update the updated second value matrix As the second value matrix obtained by the current layer Transformer network in the current round of iteration, it is used for the next round of iteration. It can be seen that the update process of this parameter is a forward update process.

[0117] As one possible implementation, the second key matrix and the second value matrix can be updated using momentum gradient descent. Momentum gradient descent is a method in which the direction of the parameter update in each iteration is opposite to the direction of the momentum used in that iteration. Alternatively, the direction of the parameter update in each iteration can be considered a weighted combination of the opposite direction of the current gradient and the direction of the previous parameter update. In other words, the direction of the parameter update depends not only on the current gradient direction but also on the direction of previous parameter updates.

[0118] Taking the update of the second key matrix as an example, we can first use the first key matrix K X and the second key matrix obtained by the current layer Transformer network in the previous round of iteration Perform element-by-element difference to obtain the key matrix gradient Then use the key matrix momentum obtained by the current layer Transformer network in the previous iteration and bond matrix gradient Perform weighted summation to obtain the key matrix momentum obtained by the current layer Transformer network in the current round of iteration Then use the key matrix momentum obtained by the current layer Transformer network in the current round iteration and the second key matrix obtained by the current layer Transformer network in the previous round of iteration Get the updated second key matrix The whole process can be expressed as follows:

[0119]

[0120]

[0121]

[0122] wherein β and η are momentum weight and update step respectively, both are preset hyperparameters, which can adopt empirical value or experimental value, for example, β takes 0.9 and η takes 0.01.

[0123] The updating process of the second value matrix is as follows: firstly, the first value matrix V X and the second value matrix V obtained by the current layer Transformer network in the last round of iteration are used to perform element-by-element difference, to obtain the value matrix gradient Then, the value matrix momentum obtained by the current layer Transformer network in the last round of iteration is used to perform weighted summation with the value matrix gradient to obtain the value matrix momentum obtained by the current layer Transformer network in the current round of iteration. Then, the value matrix momentum obtained by the current layer Transformer network in the current round of iteration is used to perform element-by-element difference with the second value matrix V obtained by the current layer Transformer network in the last round of iteration, to obtain the updated second value matrix V

[0124]

[0125]

[0126]

[0127] In the model training in the embodiments of the present application, the input is the input text sample and the output label, and the Transformer network learns the relationship between the input text sample and the output label through self-attention processing. The key matrix and the value matrix in the above process are the key carriers for expressing these relationships. The key matrix gradient and the value matrix gradient can be regarded as the relationship difference between the input text sample and the output label learned in two rounds of iteration. Through continuous forward updating, the obtained relationship tends to be consistent.

[0128] The updating process actually mixes the second key matrix and the second value matrix obtained in the last iteration with the first key matrix and the first value matrix generated by the current input feature matrix, that is, the historical information is retained and the information of the current input text is maintained, the parameters are updated smoothly, and the text prediction capability of the model is improved.

[0129] As can be seen from the above embodiments, only the updating of the two main parameters, the second key matrix and the second value matrix, is involved in the updating process of the entire model, and the parameters involved in the basic network of the model, such as the embedding network, the self-attention module, the FFN module and the prediction network, do not need to be updated, which greatly reduces the amount of parameters to be updated, reduces the calculation cost and improves the efficiency.

[0130] After each iteration is completed, it can be determined whether a preset training end condition is met. If yes, the iteration is stopped and the training is ended; otherwise, the next iteration is performed. After the training is ended, the second key matrix and the second value matrix obtained in the current iteration of each layer of the Transformer network are stored as the second key matrix and the second value matrix of each layer of the Transformer network obtained by training, respectively. and the second value matrix respectively. and are used in the text prediction process.

[0131] The training end condition can include but is not limited to that the accuracy on the validation set reaches a preset accuracy threshold, the number of iterations reaches a preset iteration threshold, and the like.

[0132] In the embodiments of the present application, the momentum gradient descent method is used to update the second key matrix and the second value matrix of each Transformer network, which can accelerate the gradient descent speed, make the iteration more efficient, and avoid falling into a local minimum. In addition, in addition to the above-mentioned momentum gradient descent method, other gradient descent methods can also be used, which are not listed one by one here.

[0133] Figure 6 A text prediction method flowchart is provided in the embodiments of the present application, which can be executed by the text prediction device in the system architecture shown in Figure 2 . As shown in Figure 6 , the method can include the following steps:

[0134] Step 602: obtaining an input text.

[0135] The input text involved in this step is a text without annotated corresponding output label, which can be a sentence, a paragraph, an article, a phrase, etc.

[0136] Step 604: Input the text sequence containing the input text into the text prediction model to obtain the output label corresponding to the input text predicted by the text prediction model; the text prediction model uses a large language model, which includes multiple layers of serially connected transformer networks. Each transformer network is executed as a current layer transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix, and the first query matrix; the first key matrix is ​​concatenated with the pre-trained second key matrix of the current layer transformer network to obtain a third key matrix, and the first value matrix is ​​concatenated with the pre-trained second value matrix of the current layer transformer network to obtain a third value matrix; the third key matrix, the third value matrix, and the first query matrix are used to perform self-attention processing to obtain the feature representation output by the current layer transformer network.

[0137] The input text alone can be used as a text sequence. Alternatively, the input text and its accompanying information can be combined to form a text sequence. Furthermore, the input text can be combined with preset prompt text to form a text sequence. The prompt text can be pre-set based on the needs of the actual scenario. For example, when performing emotion recognition, a phrase such as "The emotion expressed by this sentence is [Mask]" can be used. [Mask] represents the mask portion, and the text prediction model, through encoding and prediction of the text sequence, effectively predicts the content of the [mask] portion.

[0138] The structure of the text prediction model can refer to the relevant records in the previous embodiment. Figure 4 In the framework shown, a text sequence is input into an embedding network, which performs embedding processing on the input text sequence. It is then encoded through multiple connected Transformer networks to obtain a feature representation of the text sequence. Finally, a prediction network uses the feature representation of the text sequence to predict the output label corresponding to the input text.

[0139] Among them, the processing process of each Transformer network can be as follows Figure 7 As shown in , if the feature representation of the output of the previous layer of network is First use Determine the first bond matrix K X , the first value matrix V X And the first query matrix Q. If the current Transformer network is the first layer Transformer network, then the above is the feature representation of the embedding network output. If the current Transformer network is not the first layer Transformer network, then the above It represents the feature representation of the output of the previous layer of Transformer network. In the figure, the current layer of Transformer network is the lth layer of Transformer network as an example.

[0140] The first bond matrix K X , the first value matrix V X and the first query matrix Q X It can be determined using the following formula:

[0141]

[0142]

[0143]

[0144] Then the first key matrix K X The second key matrix of the current layer Transformer network obtained by pre-training Splice to get the third key matrix K, and the first value matrix V X The second value matrix of the current layer Transformer network obtained by pre-training The third value matrix V is obtained by concatenation.

[0145] Among them, K and V can be expressed as:

[0146]

[0147]

[0148] Then use the third key matrix K, the third value matrix V and the first query matrix Q to process the self-attention mechanism to obtain the feature representation of the current layer Transformer network output The processing of the self-attention mechanism is the processing of the attention module and the FFN module, which will not be detailed here and can be expressed as:

[0149]

[0150] From the above prediction process, we can see that the prediction process uses the and Compared with ordinary situational learning, and It includes a better observation and understanding of the labeled sample data by the text prediction model, which can significantly improve the prediction accuracy.

[0151] From the above training and prediction process, it can be seen that the principle of scenario learning adopted by the embodiment of the present application is as follows: Figure 7As shown in , we first use the sample data in the training sample set to train the LLM, and finally get the second key matrix K of all Transformer networks. T and the second value matrix V T During the prediction process, the trained LLM is used to predict the input text. During the prediction process, each Transformer needs to use the pre-trained K T and V T Perform self-attention processing. For the specific process, please refer to the relevant records in the previous embodiment. Figure 1 This shows that there are significant differences in existing situational learning.

[0152] The above method provided in the embodiment of the present application can be applied to various application scenarios, including but not limited to sentiment analysis, intention recognition, question-answering processing, etc. The following describes the method provided in the above embodiment by taking sentiment analysis as an example.

[0153] First, we obtain labeled sample data to construct a training dataset. The training dataset includes multiple training samples, each of which includes an input text sample and the sentiment category label corresponding to the input text sample. For example, as shown in Table 1:

[0154] Table 1

[0155] Input text sample Emotion category label The food is so delicious positive The food is terrible negative Bad dish negative This dish has a rich soup positive Fragrant and delicious positive …… ……

[0156] The emotion categories provided in Table 1 above are based on "positive," "neutral," and "negative" as examples. These categories can also be further categorized into coarser or finer-grained emotion categories, such as "excited," "happy," "peaceful," "sad," "angry," and "furious."

[0157] The text prediction model is trained by taking a text sequence containing an input text sample and a sentiment category label as input. For example, consider the text sequence "Review: The food is delicious. Sentiment category: positive."

[0158] The text prediction model is obtained by optimizing the pre-trained LLM. The structure and training process of the text prediction model can be found in the relevant description of the above method embodiment. The core content is that the LLM includes a multi-layer series transformer network. During training, each Transformer network is executed as the current layer Transformer network: in the current round of iteration, the feature representation of the output of the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix obtained in the previous round of iteration of the current layer Transformer network to obtain the third key matrix, and the first value matrix is ​​concatenated with the second value matrix obtained in the previous round of iteration of the current layer Transformer network to obtain the third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention processing to obtain the feature representation of the output of the current layer Transformer network; the first key matrix is ​​used to update the above second key matrix, and the updated second key matrix is ​​used as the second key matrix obtained in the current round of iteration of the current layer Transformer network; the first value matrix is ​​used to update the above second value matrix, and the updated second value matrix is ​​used as the second value matrix obtained in the current round of iteration of the current layer Transformer network. Other details are not repeated here.

[0159] During model training, the Transformer network uses self-attention processing to learn the relationship between input text samples and sentiment category labels. The key matrices and value matrices in the above process are the key carriers for expressing these relationships. The key matrix gradient and value matrix gradient can be regarded as the difference in the relationship between the input text samples and sentiment category labels learned in two rounds of iterations. Through continuous forward updates, the obtained relationships are made consistent.

[0160] During the model training process, the text prediction model retains historical information in each iteration, thereby fully understanding and learning the labeled sample data. After the model training is completed, the second bond matrix of each layer of the Transformer network obtained by training is and the second value matrix Store for use in sentiment analysis.

[0161] When using a trained text prediction model for sentiment analysis, for unlabeled input text, such as "This dish is so delicious that people can't stop eating it", one possible approach is to input "Evaluation: This dish is so delicious that people can't stop eating it" as a text sequence into the text prediction model to obtain the sentiment category corresponding to the input text predicted by the text prediction model.

[0162] As another feasible method, the text sequence obtained by concatenating the input text and the prompt text can be input into the text prediction model to obtain the emotion category corresponding to the input text predicted by the text prediction model. The prompt text can be, for example, "The emotion expressed by this sentence is [Mask]". For example, after concatenating the input text "This dish has a rich broth" with the prompt text, we can obtain the text sequence "This dish has a rich broth, and the emotion expressed by this sentence is [Mask]". [Mask] is the mask part. The text prediction model actually predicts the content of the [mask] part by encoding and predicting the text sequence. The prediction result is actually mapped to a specific emotion category in the emotion category space.

[0163] Similarly, the text prediction model uses a large language model. Its structure and principle can be found in the relevant records in the previous embodiment, so I will not elaborate on it here. The core content is that the large language model includes multiple layers of serially connected Transformer networks. Among them, each Transformer network is executed as the current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​compared with the pre-trained second key matrix of the current layer Transformer network. Concatenate to obtain the third key matrix, and combine the first value matrix with the second value matrix of the current layer Transformer network obtained by pre-training The third value matrix is ​​obtained by splicing; the third key matrix, the third value matrix and the first query matrix are used to process the self-attention mechanism to obtain the feature representation of the output of the current layer Transformer network.

[0164] As one possible implementation, the above-described method for training a text prediction model or text prediction method can be executed by a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service ecosystem. It addresses the management difficulties and limited scalability of traditional physical hosts and virtual private servers (VPS) services.

[0165] After receiving input text from a user terminal, the cloud server feeds the text sequence containing the input text into a text prediction model, obtains the output label corresponding to the input text predicted by the text prediction model, determines the corresponding service content based on the output label, and sends the service content to the user terminal. For example, the cloud server receives text input by a user in an intelligent customer service system and performs sentiment analysis on the text. Based on the sentiment category obtained from the sentiment analysis, the cloud server then provides the user with the corresponding service content. For example, if the user is identified as angry, the call can be transferred to a human customer service representative for processing. If the user is identified as happy, a promotional message can be sent to the user, and so on.

[0166] The principle and structure of the text prediction model can be found in the relevant descriptions in the previous method embodiments, and will not be elaborated here.

[0167] The aforementioned terminal devices may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and personal computers (PCs). Smart mobile devices may include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices may include smart TVs, smart speakers, and smart refrigerators. Wearable devices may include smart watches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0168] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] According to another embodiment, there is provided a device for training a text prediction model, the device corresponding to Figure 2 The model training device in the system shown. Figure 8 A schematic block diagram of the model training device according to one embodiment is shown as follows: Figure 8 As shown, the apparatus 800 may include: a sample acquisition unit 801 and a model training unit 802. The main functions of each unit are as follows:

[0170] The sample acquisition unit 801 is configured to acquire a training data set, where the training data set includes an input text sample and an output label sample corresponding to the input text sample.

[0171] The model training unit 802 is configured to use a text sequence including an input text sample and an output label corresponding to the input text sample as an input of a text prediction model to train the text prediction model. Among them, the text prediction model adopts LLM, which includes multiple layers of serially connected conversion Transformer networks; during training, each Transformer network is executed as the current layer Transformer network: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the third key matrix, and the first value matrix is ​​concatenated with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network; the first key matrix is ​​used to update the second key matrix, and the updated second key matrix is ​​used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the first value matrix is ​​used to update the second value matrix, and the updated second value matrix is ​​used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

[0172] Furthermore, the above-mentioned LLM may further include an embedding network, and the above-mentioned text prediction model may further include a prediction network.

[0173] The embedding network is used to embed text sequences.

[0174] If the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network.

[0175] The prediction network is used to use the feature representation output by the last layer of the Transformer network to predict the output label corresponding to the input text sample.

[0176] As one of the feasible methods, in the current layer Transformer network, the first key matrix can be used to update the second key matrix obtained by the current layer Transformer network in the previous round of iteration by using the momentum gradient descent method; and the first value matrix can be used to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration by using the momentum gradient descent method.

[0177] As one of the feasible methods, in the current layer Transformer network, the first bond matrix and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration can be used to perform element-by-element difference to obtain the bond matrix gradient; the bond matrix momentum and the bond matrix gradient obtained by the current layer Transformer network in the previous round of iteration are weighted summed to obtain the bond matrix momentum obtained by the current layer Transformer network in the current round of iteration; the bond matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration are used to obtain the updated second bond matrix.

[0178] The first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration can be used to perform element-by-element difference to obtain the value matrix gradient; the value matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the value matrix gradient are weighted summed to obtain the value matrix momentum obtained by the current layer Transformer network in the current round of iteration; the value matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second value matrix obtained by the current layer Transformer network in the previous round of iteration are used to obtain the updated second value matrix.

[0179] Furthermore, after each round of iteration is completed, if it is determined that the preset training end condition is met, the model training unit 802 stores the second key matrix and the second value matrix obtained by the current iteration of each layer of the Transformer network as the second key matrix and the second value matrix of each layer of the Transformer network obtained by training.

[0180] According to yet another embodiment, a text prediction device is provided. Figure 9 A schematic block diagram of a text prediction device according to an embodiment is shown. Figure 9 As shown, the apparatus 900 may include: a text acquisition unit 901 and a text prediction unit 902. The main functions of each unit are as follows:

[0181] The text acquisition unit 901 is configured to acquire input text.

[0182] The text prediction unit 902 is configured to input a text sequence containing input text into a text prediction model, and obtain an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts LLM, and the LLM includes a multi-layer serial conversion Transformer network; wherein each Transformer network is executed as a current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

[0183] Furthermore, LLM also includes an embedding network, and the above text prediction model also includes a prediction network.

[0184] The embedding network is used to embed text sequences.

[0185] If the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network.

[0186] The prediction network is used to use the feature representation output by the last layer of the Transformer network to predict the output label corresponding to the input text.

[0187] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0189] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0190] And an electronic device comprising:

[0191] one or more processors; and

[0192] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0193] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods described in the aforementioned method embodiments when executed by a processor.

[0194] in, Figure 10 The electronic device architecture is shown as an example, and may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, and the memory 1020 may be communicatively connected via a communication bus 1030.

[0195] Among them, the processor 1010 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0196] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system 1021 for controlling the operation of the electronic device 1000, and a basic input and output system (BIOS) 1022 for controlling the low-level operations of the electronic device 1000. In addition, a web browser 1023, a data storage management system 1024, and a model training device / text prediction device 1025, etc. can also be stored. The above-mentioned model training device / text prediction device 1025 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0197] The input / output interface 1013 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0198] The network interface 1014 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0199] The bus 1030 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1010 , the video display adapter 1011 , the disk drive 1012 , the input / output interface 1013 , the network interface 1014 , and the memory 1020 ).

[0200] It should be noted that although the above device only shows the processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, the memory 1020, the bus 1030, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0201] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0202] The above is a detailed introduction to the technical solutions provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. A method for training a text prediction model, characterized in that: The method comprises: Obtaining a training data set, wherein the training data set includes an input text sample and an output label corresponding to the input text sample; Using a text sequence including an input text sample and an output label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model uses a large language model, and the large language model includes a multi-layer serialized transformer network; In the training, each Transformer network is executed as the current layer Transformer network respectively: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix and the second key matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third key matrix, and the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network; the first key matrix is ​​used to update the second key matrix, and the updated second key matrix is ​​used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the first value matrix is ​​used to update the second value matrix, and the updated second value matrix is ​​used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

2. The method according to claim 1, characterized in that The large language model further includes an embedding network, and the text prediction model further includes a prediction network; The embedding network is used to embed the text sequence; If the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network; The prediction network is used to predict the output label corresponding to the input text sample using the feature representation output by the last layer of the Transformer network.

3. The method according to claim 1, characterized in that Updating the second bond matrix using the first bond matrix includes: using the first bond matrix to update the second bond matrix obtained in the previous iteration of the current layer Transformer network by using momentum gradient descent; Updating the second value matrix using the first value matrix includes: using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration using momentum gradient descent.

4. The method according to claim 3, characterized in that Using the first bond matrix to update the second bond matrix obtained by the current layer Transformer network in the previous round of iteration using the momentum gradient descent method includes: using the first bond matrix and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain the bond matrix gradient; using the bond matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the bond matrix gradient to perform weighted summation to obtain the bond matrix momentum obtained by the current layer Transformer network in the current round of iteration; using the bond matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the updated second bond matrix; Using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration using momentum gradient descent includes: using the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain the value matrix gradient; using the value matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the value matrix gradient to perform weighted summation to obtain the value matrix momentum obtained by the current layer Transformer network in the current round of iteration; using the value matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the updated second value matrix.

5. The method according to any one of claims 1 to 4, characterized in that After each round of iteration is completed, if it is determined that the preset training end condition is met, the second key matrix and the second value matrix obtained by the current iteration of each layer of the Transformer network are respectively stored as the second key matrix and the second value matrix of each layer of the trained Transformer network.

6. A text prediction method, characterized in that: The method comprises: Get input text; Inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes a multi-layer serialized Transformer network; Each Transformer network is executed as the current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

7. The method according to claim 6, characterized in that The large language model further includes an embedding network, and the text prediction model further includes a prediction network; The embedding network is used to embed the text sequence; If the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network; The prediction network is used to predict the output label corresponding to the input text using the feature representation output by the last layer of the Transformer network.

8. A method for training a text prediction model, characterized in that: The method comprises: Obtaining a training data set, wherein the training data set includes an input text sample and a sentiment category label corresponding to the input text sample; Using a text sequence including an input text sample and a sentiment category label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model uses a large language model, and the large language model includes a multi-layer serialized Transformer network; In the training, each Transformer network is executed as the current layer Transformer network respectively: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix and the second key matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third key matrix, and the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration are spliced ​​to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network; the first key matrix is ​​used to update the second key matrix, and the updated second key matrix is ​​used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the first value matrix is ​​used to update the second value matrix, and the updated second value matrix is ​​used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

9. A sentiment analysis method, characterized in that: The method comprises: Get input text; Inputting a text sequence containing the input text into a text prediction model, and obtaining a sentiment category label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes a multi-layer serialized Transformer network; Each Transformer network is executed as the current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

10. A text prediction method, executed by a cloud server, characterized in that: The method comprises: Get input text from the user terminal; Inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; Determining corresponding service content based on the output tag, and sending the service content to the user terminal; The text prediction model adopts a large language model, which includes a multi-layer serialized Transformer network; Each Transformer network is executed as the current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

11. A device for training a text prediction model, characterized in that: The device comprises: A sample acquisition unit is configured to acquire a training data set, wherein the training data set includes an input text sample and an output label sample corresponding to the input text sample; The model training unit is configured to use a text sequence including an input text sample and an output label corresponding to the input text sample as an input of a text prediction model to train the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes a multi-layer serial conversion Transformer network; in the training, each Transformer network is respectively executed as a current layer Transformer network: in the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​spliced ​​with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third key matrix Matrix, and concatenate the first value matrix with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third value matrix; use the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation of the output of the current layer Transformer network; use the first key matrix to update the second key matrix, and use the updated second key matrix as the second key matrix obtained by the current layer Transformer network in the current round of iteration; use the first value matrix to update the second value matrix, and use the updated second value matrix as the second value matrix obtained by the current layer Transformer network in the current round of iteration.

12. A text prediction device, characterized in that: The device comprises: A text acquisition unit configured to acquire input text; The text prediction unit is configured to input a text sequence containing the input text into a text prediction model, and obtain an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes a multi-layer serial conversion Transformer network; wherein each Transformer network is executed as a current layer Transformer network: the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is ​​concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is ​​concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 10.

Images