Text prediction model training method and device, and text prediction method and device
By using forward training methods in large language models, the second key matrix and value matrix of the Transformer network are updated, and the problem of poor text prediction effect under scenario learning is solved, improving prediction accuracy and training efficiency.
Patent Information
- Application Number
- PCT/SG2024/050244
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-23
- Filing Date
- 2024-04-09
- Publication Date
- 2024-11-07
AI Technical Summary
Existing large language models have poor text prediction results in scenario learning scenarios, and the lack of effective training processes leads to low prediction accuracy.
Using forward training method, by obtaining input text samples and output labels in the training dataset, using a multi-layer connected Transformer network for model training, updating the second key matrix and the second value matrix, retaining historical information and maintaining the current input text information, thereby improving the understanding and learning ability of the model.
It improves the text prediction accuracy of large language models in scenario learning scenarios, reduces the model training cost and computational complexity, avoids the local minimum value trap, and significantly improves iteration efficiency.
Smart Images

Figure SG2024050244_07112024_PF_FP_ABST
Abstract
Description
[0001] Cross-Reference to Patent Application No. 202310459343.3, filed with the Patent Office of the People's Republic of China on April 23, 2023, entitled "Method for Training a Text Prediction Model, Method for Training a Text Prediction Model, and Apparatus," the entire contents of which are incorporated herein by reference. Technical Field: This disclosure relates to the field of natural language processing, and more particularly to a method for training a text prediction model, method for training a text prediction model, and apparatus for training a text prediction model. Background: The goal of natural language processing (NLP) research is to enable machines to understand human language. Large language models (LLMs) are a core tool in the field of natural language processing and refer to deep learning models with large-scale parameters (typically hundreds of millions or more).
[0002] Due to its exceptionally high learning capabilities, LLMs are widely used in the field of text prediction. In-context learning is one of the text prediction methods currently employed by LLMs. In-context learning involves observing and summarizing labeled data, and then predicting unlabeled data. Since in-context learning typically involves no training, its application to text prediction suffers from poor results. Therefore, there is a pressing need for methods to improve the text prediction performance of LLMs in context learning scenarios. In light of this, the present disclosure provides a method, method, and apparatus for training a text prediction model to improve the text prediction performance of LLMs in context learning scenarios. The present disclosure provides the following solutions: In a first aspect, a method for training a text prediction model is provided, the method comprising: obtaining a training data set, the training data set comprising an input text sample and an output label corresponding to the input text sample; using a text sequence comprising the input text sample and the output label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model adopts a large language model, the large language model comprising a multi-layer serialized Transformer network; in the training, each Transformer network is respectively executed as a current layer Transformer network: in a current round of iteration, using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third key matrix, and concatenating the first value matrix with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third value matrix; performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current layer Transformer network; The second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current iteration round. The second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current iteration round.According to one possible implementation of the disclosed embodiment, the large language model further includes an embedding network, and the text prediction model further includes a prediction network; the embedding network is configured to perform embedding processing on the text sequence; if the current-layer Transformer network is a first-layer Transformer network, the feature representation output by the previous-layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous-layer network is the feature representation output by the previous-layer Transformer network; and the prediction network is configured to predict an output label corresponding to an input text sample using the feature representation output by the last-layer Transformer network. According to one possible implementation of the disclosed embodiment, updating the second key matrix using the first key matrix includes: updating the second key matrix obtained by the current-layer Transformer network in the previous iteration using momentum gradient descent using the first key matrix; and updating the second value matrix using the first value matrix includes: updating the second value matrix obtained by the current-layer Transformer network in the previous iteration using momentum gradient descent using the first value matrix.According to an implementable manner in an embodiment of the present disclosure, using the first key matrix to update the second key matrix obtained by the current layer Transformer network in the previous round of iteration using a momentum gradient descent method includes: using the first key matrix and the second key matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain a key matrix gradient; using the key matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the key matrix gradient to perform weighted summation to obtain the key matrix momentum obtained by the current layer Transformer network in the current round of iteration; using the key matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the updated second key matrix; using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration using a momentum gradient descent method includes: using the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain a value matrix gradient; The value matrix momentum obtained by the current layer Transformer network in the previous iteration and the value matrix gradient are weightedly summed to obtain the value matrix momentum obtained by the current layer Transformer network in the current iteration. The updated second value matrix is obtained by using the value matrix momentum obtained by the current layer Transformer network in the current iteration and the second value matrix obtained by the current layer Transformer network in the previous iteration. According to one achievable method in an embodiment of the present disclosure, after each iteration, if it is determined that a preset training end condition has been met, the second key matrix and second value matrix obtained by the current iteration of each layer Transformer network are stored as the second key matrix and second value matrix of each layer of the trained Transformer network, respectively.In a second aspect, a text prediction method is provided, the method comprising: obtaining input text; inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected transformer networks; each transformer network is respectively executed as a current layer transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer transformer network obtained by pre-training to obtain a third value matrix; using the third key matrix, the third value matrix, and the first query matrix to perform self-attention mechanism processing to obtain the feature representation output by the current layer transformer network. According to an implementable method of an embodiment of the present disclosure, the large-scale language model further includes an embedding network, and the text prediction model further includes a prediction network; the embedding network is used to perform embedding processing on the text sequence; if the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network; and the prediction network is used to use the feature representation output by the last layer Transformer network to predict the output label corresponding to the input text.In a third aspect, a method for training a text prediction model is provided, the method comprising: obtaining a training dataset, the training dataset comprising an input text sample and a sentiment category label corresponding to the input text sample; using a text sequence comprising the input text sample and the sentiment category label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model adopts a large language model, the large language model comprising multiple layers of serially connected Transformer networks; in the training, each Transformer network is respectively executed as a current layer Transformer network: in a current round of iteration, using a feature representation output by a previous layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with a second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third key matrix, and concatenating the first value matrix with a second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third value matrix; performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current layer Transformer network; and updating the second key matrix using the first key matrix. The updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.In a fourth aspect, a sentiment analysis method is provided, the method comprising: obtaining input text; inputting a text sequence containing the input text into a text prediction model, and obtaining a sentiment category label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected Transformer networks: each Transformer network is executed as a current layer Transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; using the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network. In a fifth aspect, a text prediction method is provided, which is executed by a cloud server, and the method includes: obtaining input text from a user terminal; inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; determining corresponding service content based on the output label, and sending the service content to the user terminal; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected transformer networks; each transformer network is executed as a current layer transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer transformer network obtained by pre-training to obtain a third value matrix; using the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation output by the current layer transformer network.In a sixth aspect, a device for training a text prediction model is provided, the device comprising: a sample acquisition unit, configured to acquire a training data set, the training data set comprising an input text sample and an output label sample corresponding to the input text sample; a model training unit, configured to use a text sequence comprising an input text sample and an output label corresponding to the input text sample as an input of a text prediction model to train the text prediction model; wherein the text prediction model adopts a large language model, the large language model comprises a multi-layer serialized conversion Transformer network; in the training, each Transformer network is respectively executed as a current layer Transformer network: in the current round of iteration, the feature representation output by the previous layer network is used to determine a first key matrix, a first value matrix and a first query matrix; the first key matrix is concatenated with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third key matrix, and the first value matrix is concatenated with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third value matrix; the third key matrix, The third value matrix and the first query matrix are processed by a self-attention mechanism to obtain a feature representation of the output of the current layer Transformer network; the second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.In a seventh aspect, a text prediction device is provided, comprising: a text acquisition unit configured to acquire input text; a text prediction unit configured to input a text sequence containing the input text into a text prediction model and obtain an output label corresponding to the input text predicted by the text prediction model; the text prediction model employing a large language model comprising multiple layers of serially connected Transformer networks; wherein each Transformer network is separately configured as a current-layer Transformer network and performs the following steps: determining a first key matrix, a first value matrix, and a first query matrix using a feature representation output by a previous-layer network; concatenating the first key matrix with a pre-trained second key matrix of the current-layer Transformer network to obtain a third key matrix; and concatenating the first value matrix with a pre-trained second value matrix of the current-layer Transformer network to obtain a third value matrix; and performing a self-attention mechanism using the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current-layer Transformer network. In an eighth aspect, a computer-readable storage medium is provided, storing a computer program that, when executed by a processor, implements the steps of any one of the methods described in the first aspect. According to a ninth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being configured to store program instructions, which, when read and executed by the one or more processors, execute the steps of any of the methods described in the first aspect. According to specific embodiments provided herein, the present disclosure provides the following technical effects:
[0003] 1) This disclosure utilizes a training sample set to train the LLM to obtain a text prediction model. This approach essentially utilizes labeled samples to train the LLM. During the update of the second key matrix and second value matrix, the first key matrix and first value matrix generated by combining the second key matrix and second value matrix obtained in the previous iteration with the current input feature matrix are utilized. This preserves both historical information and information about the current input text, enabling the LLM to fully understand and learn from the labeled samples, thereby improving the LLM's text prediction performance.
[0004] 2) During LLM training in this disclosure, the second key matrix and second value matrix used in each iteration are obtained from the previous iteration and updated for the next iteration. This forward optimization model approach only requires optimizing the second key matrix and second value matrix of each Transformer network, significantly reducing the number of model parameters that need to be updated, lowering model training costs and improving efficiency.
[0005] 3) In the disclosed embodiments, the second key matrix and the second value matrix of each Transformer network are updated by using a momentum gradient descent method, which can speed up the gradient descent, make the iteration more efficient, and avoid falling into local minimums.
[0006] 4) The present disclosure utilizes the second key matrix and second value matrix of each Transformer network, obtained after training the text prediction model, during the prediction process. Compared to conventional contextual learning, the second key matrix and second value matrix of each Transformer network reflect the text prediction model's better observation and understanding of the labeled sample data, significantly improving prediction accuracy. Of course, any product implementing the present disclosure does not necessarily need to achieve all of the advantages described above. BRIEF DESCRIPTION OF THE DRAWINGS To more clearly illustrate the embodiments of the present disclosure or technical solutions in the prior art, the following briefly describes the drawings required for the embodiments. Obviously, the drawings described below represent only some embodiments of the present disclosure. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Figure 1 is a schematic diagram of the principles of traditional contextual learning based on LLM; Figure 2 is a diagram of the system architecture applicable to embodiments of the present disclosure; Figure 3 is a flowchart of the method for training a text prediction model provided by embodiments of the present disclosure; Figure 4 is a schematic diagram of the structure of the text prediction model provided by embodiments of the present disclosure; Figure 5 is a schematic diagram of the principles of a Transformer network provided by embodiments of the present disclosure; Figure 6 is a flowchart of the text prediction method provided by embodiments of the present disclosure; Figure 7 is a schematic diagram of the principles of contextual learning based on LLM provided by embodiments of the present disclosure; Figure 8 is a schematic block diagram of a model training device provided by embodiments of the present disclosure; Figure 9 is a schematic block diagram of a text prediction device provided by embodiments of the present disclosure; and Figure 10 is a schematic block diagram of an electronic device provided by embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. It should be understood that the described embodiments are only some of the embodiments of the present disclosure, and not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure are within the scope of protection of the present disclosure. The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used in the disclosed embodiments and the appended claims, the singular forms "a," "an," "said," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Depending on the context, the term "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting." Similarly, depending on the context, the phrases "if determining" or "if detecting (a stated condition or event)" can be interpreted as "when determining" or "in response to determining" or "when detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)." The traditional contextual learning process based on LLM is shown in Figure 1. Labeled sample data and the input text to be predicted are input into the LLM. The LLM observes and summarizes the labeled sample data, thereby predicting the output label corresponding to the input text. This contextual learning process does not involve a training phase, resulting in poor text prediction results. If a text prediction model is pre-trained based on the LLM using traditional backpropagation training, that is, the LLM is trained using labeled sample data, and the model parameters are updated using gradients generated by the loss function. Due to the large number of model parameters in LLM, traditional backpropagation training methods are extremely costly and inefficient. In light of this, the present disclosure proposes a novel approach: pre-training a text prediction model based on LLM using forward training. To facilitate understanding of the present disclosure, the system architecture underlying the present disclosure is briefly described. Figure 2 illustrates an exemplary system architecture applicable to embodiments of the present disclosure. As shown in Figure 2, the system architecture includes a model training device that trains a text prediction model offline, and a text prediction device that performs online predictions on input text. After obtaining a training dataset, the model training device can perform model training using the methods provided in embodiments of the present disclosure to obtain a text prediction model. The text prediction device uses the trained text prediction model to predict input text, and the resulting prediction result is the output label corresponding to the input text. For example, the input text can be predicted to obtain categories such as sentiment and intent. The model training device and the text prediction device can be deployed as independent servers, on the same server or server cluster, or on separate or shared cloud servers. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service ecosystem. It addresses the management difficulties and scalability limitations of traditional physical hosts and virtual private servers (VPS). The model training device and text prediction device can also be installed on a computer terminal with strong computing power.In addition to performing text prediction online, the text prediction device can also perform text prediction offline. For example, batch text prediction can be performed on text in the data rhino as input text, and the prediction results can be stored in the data rhino for subsequent query or access. It should be understood that the number of model training devices, text prediction devices, and text prediction models shown in Figure 2 is merely illustrative. Any number of model training devices, text prediction devices, and text prediction models can be provided depending on implementation requirements. Figure 3 is a flowchart of a method for training a text prediction model provided in an embodiment of the present disclosure. This method can be performed by the model training device in the system shown in Figure 2. As shown in Figure 3, the method may include the following steps: Step 302: Obtain a training dataset, the training dataset comprising an input text sample and an output label sample corresponding to the input text sample. Step 304: Use a text sequence comprising the input text sample and the output label sample corresponding to the input text sample as input to a text prediction model to train the text prediction model. The text prediction model employs an LLM, which comprises a multi-layer serially connected Transformer network. During training, each Transformer network is executed as the current layer Transformer network: In the current round of iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix and the first query matrix; the first key matrix is concatenated with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the third key matrix, and the first value matrix is concatenated with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the third value matrix; the third key matrix, the third value matrix and the first query matrix are used to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network; the above-mentioned second key matrix is updated by using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the above-mentioned second value matrix is updated by using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.As can be seen from the above process, the present disclosure utilizes a training sample set to train the LLM to obtain a text prediction model. This approach essentially utilizes labeled samples to train the LLM. During the update of the second key matrix and second value matrix, the first key matrix and first value matrix generated by combining the second key matrix and second value matrix obtained in the previous iteration with the current input feature matrix are utilized. This preserves both historical information and information about the current input text, enabling the LLM to fully understand and learn from the labeled samples, thereby improving its text prediction performance. Furthermore, during LLM training, the second key matrix and second value matrix used in each iteration are obtained from the previous iteration and updated for use in the next iteration. This forward optimization model approach only requires optimizing the second key matrix and second value matrix of each Transformer network, significantly reducing the number of model parameters that need to be updated, lowering the cost of model training, and improving efficiency. It should be noted that the terms "first" and "second" used in this disclosure do not restrict size, order, or number; they are merely used to distinguish them in terms of name. For example, "first key matrix," "second key matrix," and "third key matrix" are used to distinguish the three key matrices by name. Another example is "first value matrix," "second value matrix," and "third value matrix" are used to distinguish the three value matrices by name. Each step in the above process is described in detail below. First, step 302, "obtaining a training dataset," is described in detail in conjunction with an embodiment. The input text sample can be a sentence, paragraph, article, phrase, or the like. The output label can correspond to different content in different application scenarios and can be a category label or a text label. For example, in a sentiment analysis scenario, the output label can be a sentiment type label, i.e., annotating the input text sample with the sentiment type. Another example is in an intent recognition scenario, where the output label can be an intent type label, i.e., annotating the input text sample with the intent type. Another example is in an automatic question-answering scenario, where the output label can be an answer text label, i.e., annotating the answer text to the question using the input text sample as a question. The output label samples can be annotated manually or automatically using existing mining methods, and this disclosure does not limit this. The following describes in detail step 304, "using a text sequence including the input text sample and the output label corresponding to the input text sample as input to a text prediction model to train the text prediction model," in conjunction with embodiments. The text prediction model in the disclosed embodiments is implemented based on the LLM, a pre-trained large-scale language model. This disclosed embodiment further optimizes the LLM to enable it to quickly understand the labeled training dataset. To fully leverage the LLM's existing language understanding capabilities, during training, text sequences are constructed using each input text sample and its corresponding output label. For example, the input text sample X is concatenated with the corresponding output label to produce the text sequence 7; the input text prediction model is the LLM. When concatenating ∘ and ∘, information indicating x and ∘ can be added. For example, the input text sample "The food is so delicious" and the output label "positive" can form the text sequence "Evaluation: The food is so delicious. Sentiment: Positive." "Evaluation" and "Sentiment" indicate the input text sample and the output label. The overall framework of the text prediction model used in the disclosed embodiments can be shown in Figure 4, primarily comprising a feature extraction network and a prediction network. The feature extraction network is used to extract feature representations from the input text sequence. The prediction network is used to predict the output label of an input text sample using the feature representations extracted by the feature extraction network. In addition to this framework, other frameworks, such as encoder-decoder frameworks, can also be used. These frameworks are not listed here, but all are implemented based on the LLM. For example, the feature extraction network is implemented based on the LLM, the encoder is implemented based on the LLM, and so on. In the embodiments of this disclosure, only the framework shown in Figure 4 is used as an example. Since the text prediction model of this disclosure is implemented based on the LLM, the feature extraction network can specifically include an embedding network and a multi-layered series of Transformer networks.The embedding network is used to perform embedding processing on the input text sequence. More specifically, embedding can be performed on each token (element) in the input text sequence. Each token in a text sequence refers to the element that constitutes the text sequence. For a text sequence, the text sequence is segmented into character or word sequences. The characters or words, as well as the start symbol and separator in the text sequence, are all tokens. The token-based embedding processing mentioned above includes at least: word embedding and position embedding. Embedding encodes each token into a word vector to obtain a word vector representation. Position embedding encodes the position of each token in the text sequence to be predicted to obtain a position representation.
[0007] The Transformer network is a model that uses the self-attention mechanism to encode each Token of the input and convert it into a feature representation. The forward learning method provided in the embodiments of the present disclosure only needs to modify the self-attention part in the Transformer. FIG. 5 is a schematic diagram of the principle of a Transformer network provided in the embodiments of the present disclosure. As shown in FIG. 5, each Transformer network includes the processing of feature representation in the "horizontal" direction and the update processing of model parameters in the "vertical" direction. It should be noted that the "horizontal" and "vertical" here are the directions shown in the figure and are only names used for convenience of description and do not have the meaning of real directions. As shown in the "horizontal" processing process in FIG. 5, the current Transformer network first determines the first key matrix K*, the first value matrix V, and the first query matrix Q using the feature representation X output by the previous layer network. Among them, if the current Transformer network is the first layer Transformer network, then the above X is the feature representation output by the embedding network. If the current Transformer network is not the first layer Transformer network, then the above X is the feature representation output by the previous layer Transformer network. In the figure, the current layer Transformer network is taken as the i-th layer Transformer network as an example. The subscript i in the figure represents the i-th round of iteration in the training process of the text prediction model. In each round of iteration, a batch of training samples can be collected from the training dataset and input into the text prediction model for training. This part is similar to the training of traditional models and will not be described in detail. The above first key matrix K*, first value matrix V x and the first query matrix Q can be determined by the following formulas:
[0008] K = W * X (1)
[0009] V x = W v X t - 1 (2) Q = W Q X (3) Where W Q . W and W are weight matrices, which are parameters learned by the LLM during the pre-training process. Then, the first key matrix K is combined with the second key matrix obtained by the current layer Transformer network in the previous round of iteration. The Transformer network is in Among them, R and I can be expressed as:
[0010] K = {Mining Area (II) (4)}
[0011] ¥ = {呀||匕} (5) where || represents the splicing process. The second key matrix and the second value matrix of each Transformer network are model parameters to be learned during the forward training process, and are continuously updated in each iteration. Each Transformer network corresponds to a second key matrix and a second value matrix. Then, using the third key matrix K, the third value matrix / , and the first query matrix Q for self-attention mechanism processing, the feature representation X: output by the current layer Transformer network is obtained. The self-attention mechanism processing is the processing of the attention module and the FFN module, which will not be elaborated here and can be expressed as:
[0012] X t l = Transformer(K, V, Q) (6) where Transformer. is the processing function of the attention module and the FFN module. , The binary matrix 玲 is used as the second value matrix obtained by the current layer Transformer network in the current iteration for use in the next iteration. It can be seen that the update process of this parameter is a forward update process. As one possible implementation, the update of the above second key matrix and second value matrix can adopt the momentum gradient descent method. The so-called momentum gradient descent method means that in gradient descent, the direction of parameter update in each iteration is opposite to the direction of the momentum used in this iteration. It can also be considered that the direction of parameter update in each iteration is a weighted combination of the opposite direction of the current gradient and the previous parameter update direction. That is to say, the direction of parameter update depends not only on the direction of the current gradient, but also on the previous parameter update direction. Taking the update of the second key matrix as an example, the first key matrix K can be used first x and the current layer Transformer network The key matrix momentum 儿匕;】 and the key matrix gradient 6房 obtained by the current layer Transformer network in the previous iteration are used for The whole process can be shown by the following formula:
[0013] G K , = K X - K\_ X⑺ qi = (1“bi, + chi ⑻ d M (9) where ” and “liao” are the momentum weights and update steps respectively, both of which are preset hyperparameters and can adopt empirical values or experimental values. For example, 0 takes 0.9 and “liao” takes 0.01. The update process of the second value matrix is as follows: First, use the first value matrix V x and the second value matrix “liao” obtained by the current layer Transformer network in the previous iteration to perform element-wise subtraction to obtain the value matrix gradient 6ling. Then use the current layer Transformer to sum to obtain the value matrix momentum % / of the current layer Transformer network in the current iteration. Then use the value matrix momentum 纠』 obtained by the current layer Transformer network in the current iteration and the second value matrix “liao” obtained by the current layer Transformer network in the previous iteration to obtain the updated second value matrix 儿'. The whole process can be shown by the following formula: The Transformer network uses self-attention to learn the relationship between input text samples and output labels. The key and value matrices in the above process are key vectors for expressing these relationships. The key and value matrix gradients can be viewed as the differences in the relationships between input text samples and output labels learned over two iterations. Through continuous forward updates, the resulting relationships converge. The above update process effectively "mixes" the second key and value matrices obtained from the previous iteration with the first key and value matrices generated from the current input feature matrix to produce the updated second key and value matrices. This preserves both historical information and the current input text, enabling smooth parameter updates and improving the model's text prediction capabilities. As can be seen from the above embodiment, the entire model update process only involves updating the two main parameters: the second key and value matrices. Parameters of the model's basic networks, such as the embedding network, self-attention module, FFN module, and prediction network, do not need to be updated. This significantly reduces the number of parameters that require updating, lowers computational cost, and improves efficiency. After each round of iteration, it is determined whether the preset training termination criteria are met. If so, the iteration is stopped and training ends; otherwise, the next round of iteration is performed. After training, the second key matrix and second value matrix obtained from the current iteration of each layer of the Transformer network are stored as the second key matrix and second value matrix of each layer of the trained Transformer network, respectively. The second key matrix and second value matrix can be set for use in the text prediction process. Training termination criteria may include, but are not limited to: the accuracy on the validation set reaching a preset accuracy threshold, the number of iterations reaching a preset iteration threshold, etc. In the disclosed embodiment, momentum gradient descent is used to update the second key matrix and second value matrix of each Transformer network. This can accelerate gradient descent, improve iteration efficiency, and avoid falling into local minima. In addition to the momentum gradient descent method described above, other gradient descent methods can also be used, which are not listed here. Figure 6 is a flow chart of the text prediction method provided in the disclosed embodiment. This method can be performed by the text prediction device in the system architecture shown in Figure 2. As shown in FIG6 , the method may include the following steps: Step 602: Obtain input text. The input text involved in this step is text that is not marked with a corresponding output tag, and may be a sentence, paragraph, article, phrase, etc.Step 604: Input a text sequence containing input text into a text prediction model to obtain the output label corresponding to the input text predicted by the text prediction model. The text prediction model utilizes a large language model comprising multiple layers of serially connected Transformer networks. Each Transformer network serves as a current-layer Transformer network: The feature representation output by the previous layer network is used to determine a first key matrix, a first value matrix, and a first query matrix; the first key matrix is concatenated with the pre-trained second key matrix of the current-layer Transformer network to obtain a third key matrix; and the first value matrix is concatenated with the pre-trained second value matrix of the current-layer Transformer network to obtain a third value matrix; a self-attention mechanism is performed using the third key matrix, the third value matrix, and the first query matrix to obtain the feature representation output by the current-layer Transformer network. The input text alone can be used as the text sequence. Alternatively, the input text and information indicating the input text can form the text sequence. Alternatively, the input text and preset prompt text can be concatenated to form the text sequence. The prompt text can be pre-set based on the actual scenario requirements. For example, when performing emotion recognition, a phrase such as "The emotion expressed by this sentence is [Mask]" can be used. [Mask] is the mask portion. The text prediction model, through encoding and prediction of the text sequence, actually predicts the content of the [Mask] portion. The structure of the text prediction model can be found in the relevant descriptions of the previous embodiments. If the framework shown in Figure 4 is used, the text sequence is input into the embedding network, which performs embedding processing on the input text sequence. It is then encoded through multiple connected Transformer networks to obtain a feature representation of the text sequence. Finally, the feature representation of this sequence is used to predict the output label corresponding to the input text. The processing process of a Transformer network can be shown in Figure 7. If the previous layer network input is X, X is first used to determine the first key matrix K, the first value matrix K, and the first value matrix K. If the current Transformer network is a first-layer Transformer network, then X is the feature representation. If the current Transformer network is not a first-layer Transformer network, then The feature representation output by the nth layer Transformer network. In the figure, the current layer Transformer network is taken as the nth layer Transformer network as an example. The above first key matrix Ky, first value matrix V, and first query matrix Qx can be determined by the following formulas:
[0014] K x = W k X;'' (13)
[0015] V x =W v X^ (14)
[0016] (2=%X 「 (15) Then, the first key matrix K x is concatenated with the second key matrix K of the current layer Transformer network obtained by pre-training to obtain the third key matrix K, and the first value matrix V is concatenated with the second value matrix V of the current layer Transformer network obtained by pre-training to obtain the third value matrix V. Among them, K and W can be expressed as:
[0017] K={K T l \K x} (16)
[0018] V={Vi\V x} (17) Then, the self-attention mechanism is processed using the third key matrix K, the third value matrix V, and the first query matrix Q to obtain the feature representation X of the current layer Transformer network. The processing of the self-attention mechanism is the processing of the attention module and the FFN module, which will not be elaborated here and can be expressed as: From the above prediction process, it can be seen that the prediction process uses K and V obtained after model training. Compared with ordinary scenario learning, K T lThe summation includes a better observation and understanding of the text prediction model for the labeled sample data, which can significantly improve prediction accuracy. As can be seen from the above training and prediction process, the principle of contextual learning adopted by the disclosed embodiment is shown in Figure 7. The LLM is first trained using sample data from the training sample set, ultimately obtaining the second key matrix λ and the second value matrix κ for all Transformer networks. During the prediction process, the trained LLM is used to predict the input text. During the prediction process, each Transformer needs to use the pre-trained summation to perform self-attention processing. For details, please refer to the relevant description in the previous embodiment. This is significantly different from the existing contextual learning shown in Figure 1. The above method provided by the disclosed embodiment can be applied to various application scenarios, including but not limited to sentiment analysis, intent recognition, question-answering processing, etc. The following uses sentiment analysis as an example to describe the method provided by the above embodiment. First, labeled sample data is obtained to construct a training dataset. The training dataset includes multiple training samples, each of which includes an input text sample and the sentiment category label corresponding to the input text sample. For example, as shown in Table 1: Table 1 The emotion categories provided in Table 1 above are based on "positive", "neutral" and "negative" as examples, and can also be divided into other coarser or finer granularity emotion categories. For example, "excited", "happy",
[0019] "peaceful," "sad," "angry," "furious," and so on. A text prediction model is trained using a text sequence containing input text samples and sentiment category labels. For example, the text sequence "Review: The food is delicious. Sentiment category: positive" is constructed. This text prediction model is optimized based on the pre-trained LLM. The structure and training process of the text prediction model can be found in the relevant description of the method embodiment above. The core concept of the LLM is that it comprises a multi-layer, serially connected Transformer network. During training, each Transformer network acts as the current Transformer network. In the current iteration, the feature representation output by the previous layer is used to determine the first key matrix, the first value matrix, and the first query matrix. The first key matrix is concatenated with the second key matrix obtained in the previous iteration of the current Transformer network to obtain the third key matrix, and the first value matrix is concatenated with the second value matrix obtained in the previous iteration of the current Transformer network to obtain the third value matrix. A self-attention mechanism is applied to the third key matrix, the third value matrix, and the first query matrix to obtain the feature representation output by the current Transformer network. The second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained in the current Transformer network in the current iteration. The second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained in the current Transformer network in the current iteration. Other details are not detailed here. During model training, the Transformer network uses self-attention to learn the relationship between input text samples and sentiment category labels. The key and value matrices in this process are key carriers for expressing these relationships. The key and value matrix gradients can be viewed as the difference between the learned relationships between input text samples and sentiment category labels over two iterations. Through continuous forward updates, the resulting relationships converge. During model training, the text prediction model retains historical information in each iteration, thereby fully understanding and learning from the labeled sample data. After model training is complete, the second key matrix K: and second value matrix t of each layer of the Transformer network are stored for use in sentiment analysis.When using a trained text prediction model for sentiment analysis, for unlabeled input text, such as "This dish is so delicious," one possible implementation approach is to input the text sequence "Evaluation: This dish is so delicious" into the text prediction model to obtain the sentiment category predicted by the text prediction model. Another possible implementation approach is to input the text sequence obtained by concatenating the input text with the prompt text into the text prediction model to obtain the sentiment category predicted by the text prediction model. The prompt text can be, for example, "The sentiment expressed by this sentence is [Mask]." For example, concatenating the input text "This dish has a rich broth" with the prompt text yields the text sequence "This dish has a rich broth, and the sentiment expressed by this sentence is [Mask]." [Mask] is the mask. By encoding and predicting the text sequence, the text prediction model effectively predicts the [mask] portion, mapping the prediction result to a specific sentiment category in the sentiment category space. Similarly, the text prediction model uses a large language model. Its structure and principles are described in the relevant embodiments and are not detailed here. The core concept is that the large language model includes multiple layers of serially connected Transformer networks. Each Transformer network serves as the current-layer Transformer network. The current-layer Transformer network is pre-trained using the feature representation output by the previous layer network to determine the first key matrix. The first value matrix is concatenated with the pre-trained second value matrix ' of the current-layer Transformer network to obtain a third value matrix. A self-attention mechanism is performed using the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation of the output of the current-layer Transformer network. As one possible implementation, the above-mentioned method for training a text prediction model or text prediction method can be performed by a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service system that addresses the management difficulties and poor scalability of traditional physical hosts and virtual private servers (VPS) services. After receiving input text from a user terminal, the cloud server inputs the text sequence containing the input text into a text prediction model, obtains the output label corresponding to the input text predicted by the text prediction model, determines the corresponding service content based on the output label, and sends the service content to the user terminal. For example, text input by a user in an intelligent customer service system is obtained and sentiment analysis is performed on the text. Based on the sentiment category obtained by the sentiment analysis, the user is provided with corresponding service content. For example, if the user is identified as angry, the call can be transferred to a human customer service representative for processing. If it is determined that the user is happy, promotional information can be sent to the user, etc. The principles and structure of the text prediction model can be found in the relevant descriptions of the previous method embodiments and will not be elaborated here. The terminal devices mentioned above may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and PCs (Personal Computers). Smart mobile devices may include, for example, mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices may include smart TVs, smart speakers, smart refrigerators, etc. Wearable devices may include, for example, smart watches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality). The above description of specific embodiments of this specification is intended. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain implementations, multitasking and parallel processing are also possible or may be advantageous.According to another embodiment, a device for training a text prediction model is provided. This device corresponds to the model training device in the system shown in FIG2 . FIG8 shows a schematic block diagram of the model training device according to one embodiment. As shown in FIG8 , the device 800 may include a sample acquisition unit 801 and a model training unit 802. The main functions of each unit are as follows: The sample acquisition unit 801 is configured to acquire a training dataset comprising an input text sample and an output label sample corresponding to the input text sample. The model training unit 802 is configured to train the text prediction model by using a text sequence comprising the input text sample and the output label sample corresponding to the input text sample as input. The text prediction model uses an LLM, which includes multiple layers of serially connected Transformer networks. During training, each Transformer network is executed as the current layer Transformer network: in the current iteration, the feature representation output by the previous layer network is used to determine the first key matrix, the first value matrix, and the first query matrix; the first key matrix is concatenated with the second key matrix obtained by the current layer Transformer network in the previous iteration to obtain a third key matrix, and the first value matrix is concatenated with the second value matrix obtained by the current layer Transformer network in the previous iteration to obtain a third value matrix; the third key matrix, the third value matrix, and the first query matrix are processed using a self-attention mechanism to obtain the feature representation output by the current layer Transformer network; the second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current iteration; the second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current iteration. Furthermore, the LLM may further include an embedding network, and the text prediction model may further include a prediction network. The embedding network is used to embed text sequences. If the current Transformer network is the first-layer Transformer network, the feature representation output by the previous layer is used as the feature representation output by the embedding network. Otherwise, the feature representation output by the previous layer is used as the feature representation output by the previous layer Transformer network. The prediction network is used to predict the output label corresponding to the input text sample using the feature representation output by the last-layer Transformer network.As one possible implementation, in the current layer Transformer network, the first bond matrix can be used to update the second bond matrix obtained in the previous iteration of the current layer Transformer network using momentum gradient descent. The first value matrix can also be used to update the second value matrix obtained in the previous iteration of the current layer Transformer network using momentum gradient descent. As one possible implementation, in the current layer Transformer network, the first bond matrix and the second bond matrix obtained in the previous iteration of the current layer Transformer network can be element-by-element subtracted to obtain the bond matrix gradient. The bond matrix momentum obtained in the previous iteration of the current layer Transformer network and the bond matrix gradient can be weighted summed to obtain the bond matrix momentum obtained in the current iteration of the current layer Transformer network. The bond matrix momentum obtained in the current iteration of the current layer Transformer network and the second bond matrix obtained in the previous iteration of the current layer Transformer network can be used to obtain the updated second bond matrix. The first value matrix and the second value matrix obtained in the previous iteration of the current layer Transformer network can be element-by-element subtracted to obtain a value matrix gradient. The value matrix momentum obtained in the previous iteration of the current layer Transformer network and the value matrix gradient can be weighted summed to obtain the value matrix momentum obtained in the current iteration of the current layer Transformer network. The value matrix momentum obtained in the current iteration of the current layer Transformer network and the second value matrix obtained in the previous iteration of the current layer Transformer network can be used to obtain an updated second value matrix. Furthermore, after each iteration, if it is determined that a preset training termination condition has been met, the model training unit 802 stores the second key matrix and second value matrix obtained in the current iteration of each layer Transformer network as the second key matrix and second value matrix of each trained Transformer network layer, respectively. According to yet another embodiment, a text prediction device is provided. FIG9 shows a schematic block diagram of a text prediction device according to an embodiment. As shown in FIG9 , the device 900 may include: a text acquisition unit 901 and a text prediction unit 902. The main functions of each unit are as follows: The text acquisition unit 901 is configured to acquire input text.The text prediction unit 902 is configured to input a text sequence containing input text into a text prediction model and obtain an output label corresponding to the input text predicted by the text prediction model. The text prediction model employs an LLM, which includes multiple layers of serially connected Transformer networks. Each Transformer network acts as a current-layer Transformer network: a first key matrix, a first value matrix, and a first query matrix are determined using the feature representation output by the previous-layer network; the first key matrix is concatenated with the pre-trained second key matrix of the current-layer Transformer network to obtain a third key matrix, and the first value matrix is concatenated with the pre-trained second value matrix of the current-layer Transformer network to obtain a third value matrix; a self-attention mechanism is performed using the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current-layer Transformer network. Furthermore, the LLM also includes an embedding network, and the aforementioned text prediction model also includes a prediction network. The embedding network is used to embed the text sequence. If the current Transformer network is the first-layer Transformer network, the feature representation output by the previous-layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous-layer network is the feature representation output by the previous-layer Transformer network. The prediction network is used to predict the output label corresponding to the input text using the feature representation output by the last-layer Transformer network. The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are described briefly because they are generally similar to the method embodiments. For relevant details, refer to the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one location or distributed across multiple network units. Some or all of these modules may be selected to achieve the objectives of the present embodiments based on actual needs. Persons of ordinary skill in the art can understand and implement these embodiments without inventive effort.It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject. The various functional units provided in the embodiments of this disclosure can be executed in the methods of the above-mentioned embodiments or similar computing devices, or can be stored as part of a computer-readable storage medium. In addition, the embodiments of this disclosure also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods in the above-mentioned method embodiments. Optionally, when the computer program is executed by a processor, program code implements the following steps: obtaining a training data set, the training data set including an input text sample and an output label corresponding to the input text sample; using a text sequence including the input text sample and the output label corresponding to the input text sample as input to a text prediction model to train the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected transformer networks; in the training, each transformer network is respectively executed as a current layer transformer network: in a current round of iteration, using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with the second key matrix obtained by the current layer transformer network in the previous round of iteration to obtain a third key matrix, and concatenating the first value matrix with the second value matrix obtained by the current layer transformer network in the previous round of iteration to obtain a third value matrix; performing a self-attention mechanism processing using the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current layer transformer network; The second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current iteration round. The second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current iteration round.Optionally, when the above-mentioned computer program is executed by a processor, the program code implements the following steps: obtaining input text; inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected transformer networks; each transformer network is executed as a current layer transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer transformer network obtained by pre-training to obtain a third value matrix; using the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation output by the current layer transformer network.Optionally, the computer program, when executed by a processor, implements program code for the following steps: obtaining a training data set, the training data set comprising an input text sample and a sentiment category label corresponding to the input text sample; using a text sequence comprising the input text sample and the sentiment category label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model employs a large language model comprising a multi-layer serially connected Transformer network; in the training, each Transformer network is respectively executed as a current-layer Transformer network: in a current iteration, determining a first key matrix, a first value matrix, and a first query matrix using a feature representation output by a previous-layer network; concatenating the first key matrix with a second key matrix obtained by the current-layer Transformer network in the previous iteration to obtain a third key matrix, and concatenating the first value matrix with a second value matrix obtained by the current-layer Transformer network in the previous iteration to obtain a third value matrix; performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current-layer Transformer network; The second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current iteration round. The second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current iteration round.Optionally, when the above-mentioned computer program is executed by a processor, the program code implements the following steps: obtaining input text; inputting a text sequence containing the input text into a text prediction model, and obtaining the sentiment category label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected transformer networks: each transformer network is executed as a current layer transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer transformer network obtained by pre-training to obtain a third value matrix; using the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation output by the current layer transformer network. Optionally, when the above-mentioned computer program is executed by a processor, program code that implements the following steps: obtaining input text from a user terminal; inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; determining corresponding service content based on the output label, and sending the service content to the user terminal; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected transformer networks; each transformer network is executed as a current layer transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer transformer network obtained by pre-training to obtain a third value matrix; using the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation output by the current layer transformer network. Computer-readable storage media may include a data signal transmitted in baseband or as part of a carrier wave, carrying readable program code. Such a transmitted data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.A computer-readable storage medium can transmit, disseminate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained in the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical, radio frequency, or any suitable combination thereof. Also provided is an electronic device comprising: one or more processors; and a memory associated with the one or more processors, the memory storing program instructions that, when read and executed by the one or more processors, perform the steps of any of the aforementioned method embodiments. The present disclosure also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of any of the aforementioned method embodiments. FIG10 exemplarily illustrates the architecture of an electronic device, which may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020 may be communicatively connected via a communication bus 1030. The processor 1010 may be implemented using a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, configured to execute relevant programs to implement the technical solutions provided herein. The memory 1020 may be implemented using ROM (Read Only Memory), RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system 1021 configured to control the operation of the electronic device 1000, and a basic input / output system (BIOS) 1022 configured to control low-level operations of the electronic device 1000. Furthermore, the memory 1020 may store a web browser 1023, a data storage management system 1024, and a model training device / text prediction device 1025, etc. The model training device / text prediction device 1025 may be an application program that specifically implements the aforementioned steps in the embodiments of the present disclosure.In summary, when implementing the technical solutions provided by this disclosure through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010. The input / output interface 1013 is configured to connect to an input / output module to implement information input and output. The input / output module can be configured as a component within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., while output devices may include a display, speaker, vibrator, indicator light, etc. The network interface 1014 is configured to connect to a communication module (not shown in the figure) to enable communication between the device and other devices. The communication module can achieve communication through wired means (e.g., USB, network cable, etc.) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth, etc.). The bus 1030 comprises a pathway for transmitting information between various components of the device (e.g., the processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, and the memory 1020). It should be noted that although the above device only illustrates the processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, the memory 1020, and the bus 1030, in a specific implementation, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will appreciate that the above device may only include the components necessary to implement the disclosed solution, rather than all of the components shown in the figure. The present disclosure also provides a processor. FIG11 illustrates a schematic diagram of the structure of a processor used to implement the present disclosure. As shown in FIG11 , the processor 1100 is configured to run a program, wherein the program, when executed by the processor, executes the method in the above embodiment. In an embodiment of the present disclosure, the processor 1100 may execute a program of a method for training a text prediction model.Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: obtaining a training dataset, the training dataset including an input text sample and an output label corresponding to the input text sample; using a text sequence including the input text sample and the output label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected Transformer networks; in the training, each Transformer network is respectively executed as a current-layer Transformer network: in a current iteration, using a feature representation output by a previous-layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with a second key matrix obtained by the current-layer Transformer network in the previous iteration to obtain a third key matrix, and concatenating the first value matrix with a second value matrix obtained by the current-layer Transformer network in the previous iteration to obtain a third value matrix; performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current-layer Transformer network; The second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current iteration round. The second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current iteration round.Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: obtaining input text; inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; the text prediction model employing a large language model, the large language model including multiple layers of serially connected Transformer networks; each Transformer network acting as a current-layer Transformer network: determining a first key matrix, a first value matrix, and a first query matrix using a feature representation output by a previous-layer network; concatenating the first key matrix with a pre-trained second key matrix of the current-layer Transformer network to obtain a third key matrix, and concatenating the first value matrix with a pre-trained second value matrix of the current-layer Transformer network to obtain a third value matrix; and performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current-layer Transformer network.Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: obtaining a training data set, the training data set including an input text sample and a sentiment category label corresponding to the input text sample; using a text sequence including the input text sample and the sentiment category label corresponding to the input text sample as input to a text prediction model, and training the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected Transformer networks; in the training, each Transformer network is respectively executed as a current layer Transformer network: in a current round of iteration, using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with the second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third key matrix, and concatenating the first value matrix with the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third value matrix; performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current layer Transformer network; The second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current iteration round. The second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current iteration round.Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: obtaining input text; inputting a text sequence containing the input text into a text prediction model, and obtaining a sentiment category label corresponding to the input text predicted by the text prediction model; the text prediction model employing a large language model, the large language model including multiple layers of serially connected Transformer networks; each Transformer network being executed as a current-layer Transformer network: determining a first key matrix, a first value matrix, and a first query matrix using a feature representation output by a previous-layer network; concatenating the first key matrix with a pre-trained second key matrix of the current-layer Transformer network to obtain a third key matrix, and concatenating the first value matrix with a pre-trained second value matrix of the current-layer Transformer network to obtain a third value matrix; and performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current-layer Transformer network. Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: obtaining input text from a user terminal; inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; determining corresponding service content based on the output label, and sending the service content to the user terminal; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected transformer networks; each transformer network is respectively executed as a current layer transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer transformer network obtained by pre-training to obtain a third value matrix; and performing a self-attention mechanism processing using the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current layer transformer network. It can be seen from the above description of the embodiments that those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus a necessary general hardware platform.Based on this understanding, the technical solutions of the present disclosure, or the portion that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or portions of these embodiments of the present disclosure. The technical solutions provided by the present disclosure have been described in detail above, and the principles and implementation methods of the present disclosure have been illustrated herein using specific examples. The description of the above embodiments is intended only to facilitate understanding of the methods and core concepts of the present disclosure. Furthermore, those skilled in the art will appreciate that variations in the specific implementation methods and scope of application are possible based on the concepts of the present disclosure. In summary, the contents of this specification should not be construed as limiting the present disclosure. Industrial Applicability The solution provided by the embodiments of the present disclosure can be applied to the process of training a text prediction model, obtaining a training dataset, the training dataset including an input text sample and an output label corresponding to the input text sample; using a text sequence including the input text sample and the output label corresponding to the input text sample as input to the text prediction model, and training the text prediction model; wherein the text prediction model adopts a large language model, the large language model including multiple layers of serially connected transformer networks; during training, each transformer network is respectively executed as a current layer transformer network: in a current round of iteration, using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix, and a first query matrix; concatenating the first key matrix with the second key matrix obtained by the current layer transformer network in the previous round of iteration to obtain a third key matrix, and concatenating the first value matrix with the second value matrix obtained by the current layer transformer network in the previous round of iteration to obtain a third value matrix; performing a self-attention mechanism on the third key matrix, the third value matrix, and the first query matrix to obtain a feature representation output by the current layer transformer network; and updating the second key matrix using the first key matrix. The updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the second value matrix is updated using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current round of iteration, thereby solving the technical problem of low efficiency in training text prediction models.
Claims
Claims 1. A method for training a text prediction model, the method comprising: Acquire a training data set, the training data set comprising an input text sample and an output label corresponding to the input text sample; A text sequence including an input text sample and an output label corresponding to the input text sample is used as an input of a text prediction model to train the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes a plurality of layers of serially connected conversion Transformer networks; in the training, each Transformer network is respectively executed as a current layer Transformer network: in a current round of iteration, a first key matrix, a first value matrix and a first query matrix are determined using a feature representation output by a previous layer network; the first key matrix is concatenated with a second key matrix obtained by the current layer Transformer network in a previous round of iteration to obtain a third key matrix, and the first value matrix is concatenated with a second value matrix obtained by the current layer Transformer network in a previous round of iteration to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are processed by a self-attention mechanism to obtain a feature representation output by the current layer Transformer network; the second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the second value matrix is updated using the first value matrix, The updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.
2. The method according to claim 1, wherein: The large language model also includes an embedding network, and the text prediction model also includes a prediction network; the embedding network is used to embed the text sequence; if the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network; the prediction network is used to predict the output label corresponding to the input text sample using the feature representation output by the last layer Transformer network.
3. The method according to claim 1, wherein: Using the first key matrix to update the second key matrix includes: using the first key matrix to update the second key matrix obtained by the current layer Transformer network in the previous round of iteration using the momentum gradient descent method; using the first value matrix to update the second value matrix includes: using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration using the momentum gradient descent method.
4. The method according to claim 3, wherein: Using the first bond matrix to update the second bond matrix obtained by the current layer Transformer network in the previous round of iteration by using a momentum gradient descent method includes: using the first bond matrix and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration The key matrix is subtracted element by element to obtain the key matrix gradient; the key matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the key matrix gradient are weighted summed to obtain the key matrix momentum obtained by the current layer Transformer network in the current round of iteration; the key matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second key matrix obtained by the current layer Transformer network in the previous round of iteration are used to obtain the updated second key matrix; using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration by momentum gradient descent includes: using the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element subtraction to obtain the value matrix gradient; using the value matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the value matrix gradient are weighted summed to obtain the value matrix momentum obtained by the current layer Transformer network in the current round of iteration; using the current layer Transformer The value matrix momentum obtained by the network in the current round of iteration and the second value matrix obtained by the current layer Transformer network in the previous round of iteration are used to obtain the updated second value matrix.
5. The method according to any one of claims 1 to 4, wherein: After each round of iteration is completed, if it is determined that the preset training end condition is met, the second key matrix and the second value matrix obtained by the current iteration of each layer of the Transformer network are stored as the second key matrix and the second value matrix of each layer of the Transformer network obtained by training.
6. A text prediction method, the method comprising: Get input text; Inputting a text sequence containing the input text into a text prediction model, and obtaining an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected conversion Transformer networks; each Transformer network is respectively executed as a current layer Transformer network: using the feature representation output by the previous layer network to determine a first key matrix, a first value matrix and a first query matrix; concatenating the first key matrix with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and concatenating the first value matrix with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; using the third key matrix, the third value matrix and the first query matrix to perform self-attention mechanism processing to obtain the feature representation output by the current layer Transformer network.
7. The method according to claim 6, wherein: The large language model also includes an embedding network, and the text prediction model also includes a prediction network; the embedding network is used to embed the text sequence; if the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network; the prediction network is used to predict the output label corresponding to the input text using the feature representation output by the last layer Transformer network.
8. A method for training a text prediction model, the method comprising: Acquire a training data set, wherein the training data set includes an input text sample and a sentiment category label corresponding to the input text sample; A text sequence including an input text sample and an emotion category label corresponding to the input text sample is used as an input of a text prediction model to train the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected Transformer networks; in the training, each Transformer network is respectively executed as a current layer Transformer network: in a current round of iteration, a first key matrix, a first value matrix and a first query matrix are determined using a feature representation output by a previous layer network; the first key matrix is concatenated with a second key matrix obtained by the current layer Transformer network in a previous round of iteration to obtain a third key matrix, and the first value matrix is concatenated with a second value matrix obtained by the current layer Transformer network in a previous round of iteration to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are processed by a self-attention mechanism to obtain a feature representation output by the current layer Transformer network; the second key matrix is updated using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the second value matrix is updated using the first value matrix, The updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.
9. The method according to claim 8, wherein: The large language model also includes an embedding network, and the text prediction model also includes a prediction network; the embedding network is used to embed the text sequence; if the current layer Transformer network is the first layer Transformer network, the feature representation output by the previous layer network is the feature representation output by the embedding network; otherwise, the feature representation output by the previous layer network is the feature representation output by the previous layer Transformer network; the prediction network is used to predict the output label corresponding to the input text sample using the feature representation output by the last layer Transformer network.
10. The method according to claim 8, wherein: Using the first key matrix to update the second key matrix includes: using the first key matrix to update the second key matrix obtained by the current layer Transformer network in the previous round of iteration using the momentum gradient descent method; using the first value matrix to update the second value matrix includes: using the first value matrix to update the second value matrix obtained by the current layer Transformer network in the previous round of iteration using the momentum gradient descent method.
11. The method according to claim 10, wherein: Using the first bond matrix to update the second bond matrix obtained by the current layer Transformer network in the previous round of iteration by using the momentum gradient descent method includes: using the first bond matrix and the second bond matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain the bond matrix gradient; using the bond matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the bond matrix gradient to perform weighted summation to obtain the current layer The key matrix momentum obtained by the Transformer network in the current round of iteration; the updated second key matrix is obtained by using the key matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second key matrix obtained by the current layer Transformer network in the previous round of iteration; the second value matrix obtained by the current layer Transformer network in the previous round of iteration is updated by using the first value matrix by using the momentum gradient translation method, including: using the first value matrix and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to perform element-by-element difference to obtain the value matrix gradient; using the value matrix momentum obtained by the current layer Transformer network in the previous round of iteration and the value matrix gradient to perform weighted summation to obtain the value matrix momentum obtained by the current layer Transformer network in the current round of iteration; using the value matrix momentum obtained by the current layer Transformer network in the current round of iteration and the second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain the updated second value matrix.
12. A sentiment analysis method, the method comprising: Get input text; A text sequence containing the input text is input into a text prediction model to obtain a sentiment category label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected conversion Transformer networks; each Transformer network is respectively executed as a current layer Transformer network: a first key matrix, a first value matrix and a first query matrix are determined using the feature representation output by the previous layer network; the first key matrix is concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are processed by a self-attention mechanism to obtain a feature representation output by the current layer Transformer network.
13. A text prediction method, executed by a cloud server, comprising: Get input text from the user terminal; Input a text sequence containing the input text into a text prediction model, obtain an output label corresponding to the input text predicted by the text prediction model; determine the corresponding service content based on the output label, and send the service content to the user terminal; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected conversion Transformer networks; each Transformer network is respectively executed as a current layer Transformer network: a first key matrix, a first value matrix and a first query matrix are determined using the feature representation output by the previous layer network; the first key matrix is concatenated with the second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is concatenated with the second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are processed by a self-attention mechanism to obtain a feature representation output by the current layer Transformer network.
14. A device for training a text prediction model, the device comprising: A sample acquisition unit, configured to acquire a training data set, wherein the training data set includes an input text sample and an output label sample corresponding to the input text sample; A model training unit is configured to use a text sequence including an input text sample and an output label corresponding to the input text sample as an input of a text prediction model to train the text prediction model; wherein the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected conversion Transformer networks; in the training, each Transformer network is respectively used as a current layer Transformer network to perform: in a current round of iteration, a first key matrix, a first value matrix and a first query matrix are determined using a feature representation output by a previous layer network; the first key matrix is concatenated with a second key matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third key matrix, and the first value matrix is concatenated with a second value matrix obtained by the current layer Transformer network in the previous round of iteration to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are processed by a self-attention mechanism to obtain a feature representation output by the current layer Transformer network; The second key matrix is updated by using the first key matrix, and the updated second key matrix is used as the second key matrix obtained by the current layer Transformer network in the current round of iteration; the second value matrix is updated by using the first value matrix, and the updated second value matrix is used as the second value matrix obtained by the current layer Transformer network in the current round of iteration.
15. A text prediction device, the device comprising: A text acquisition unit, configured to acquire input text; A text prediction unit is configured to input a text sequence containing the input text into a text prediction model, and obtain an output label corresponding to the input text predicted by the text prediction model; the text prediction model adopts a large language model, and the large language model includes multiple layers of serially connected Transformer networks; wherein each Transformer network is respectively executed as a current layer Transformer network: a first key matrix, a first value matrix and a first query matrix are determined using the feature representation output by the previous layer network; the first key matrix is concatenated with a second key matrix of the current layer Transformer network obtained by pre-training to obtain a third key matrix, and the first value matrix is concatenated with a second value matrix of the current layer Transformer network obtained by pre-training to obtain a third value matrix; the third key matrix, the third value matrix and the first query matrix are processed by a self-attention mechanism to obtain a feature representation output by the current layer Transformer network.
16. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.
17. An electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being configured to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method according to any one of claims 1 to 13.
18. A computer program product, wherein: The method comprises a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13. 31 19. A computer program product, wherein: The method comprises a non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 13 when executed by a processor.
20. A computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented. 32
Citation Information
Patent Citations
Text sentiment classification method and device, electronic equipment and storage medium
CN113849592A
Text prediction method and model establishment method and device for text prediction
CN115563976A
Emotion analysis method, system and equipment based on deep learning
CN115982355A
Learned threshold token pruning for transformer neural networks
US20220374766A1
Cited By
Language model using method for sharing memory matrix
CN119537019A
A language model using method of sharing a memory matrix
CN119537019B
Key value compression method in self-attention mechanism, large language model and electronic equipment
CN120106150A
Key-value compression method in self-attention mechanism, large language model and electronic device
CN120106150B
Information processing method and device and electronic equipment
CN120542577A
