Text prediction model training method and device, equipment, medium and product

Abstract

The invention discloses a text prediction model training method and device, equipment, a medium and a product. The method comprises the following steps: acquiring sample prompt information; predicting through the first model to obtain predicted reply content corresponding to the sample prompt information, wherein the predicted reply content comprises at least two text units; generating dominant data respectively corresponding to the at least two text units; on the basis of the advantage data corresponding to the at least two text units, obtaining a loss value corresponding to the predicted reply content; and training the first model through the loss value to obtain a second model. The method can improve the training efficiency of the model.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and particularly to a training method, device, equipment, medium and product of a text prediction model. Background Art

[0002] Reinforcement Learning (RL) has been widely applied in the field of natural language processing (e.g., Large Language Model (LLM)), especially in complex reasoning tasks (such as long-chain reasoning), and more and more tasks rely on the step-by-step reasoning ability of the model.

[0003] In the related art, during the model training process implemented by reinforcement learning, the loss of the sample statement is determined according to the reward signal of the generated sample statement, so as to train the model through the loss.

[0004] However, when there are generation errors in the sample statement, the loss of the entire sample statement will be regarded as negative feedback, resulting in the model being unable to obtain effective reward feedback through this sample statement, and thus unable to optimize it effectively, thereby reducing the training efficiency. Summary of the Invention

[0005] Embodiments of this application provide a training method, device, equipment, medium and product of a text prediction model. The technical solutions provided by this application include the following aspects.

[0006] According to one aspect of the embodiments of this application, a training method of a text prediction model is provided, and the method includes:

[0007] Obtain sample prompt information, where the sample prompt information is used to guide the first model to output a predicted response content, and the first model is a machine learning model to be trained;

[0008] Predict, through the first model, the predicted response content corresponding to the sample prompt information, where the predicted response content includes at least two text units;

[0009] Generate advantage data corresponding to each of the at least two text units, where the advantage data is used to indicate the influence of the text unit on the prediction process when the first model predicts the predicted response content;

[0010] Based on the advantage data corresponding to each of the at least two text units, obtain the loss value corresponding to the predicted response content;

[0011] Train the first model through the loss value to obtain a second model, where the second model is used to perform text prediction tasks.

[0012] According to one aspect of the embodiments of the present application, there is provided a training device for a text prediction model, the device comprising:

[0013] An acquisition module, configured to acquire sample prompt information for guiding the first model to output a predicted response content, where the first model is a machine learning model to be trained;

[0014] A prediction module, configured to predict, through the first model, a predicted response content corresponding to the sample prompt information, where the predicted response content includes at least two text units;

[0015] A loss generation module, configured to generate advantage data corresponding to the at least two text units respectively, where the advantage data is used to indicate the influence of the text units on the prediction process when the first model predicts the predicted response content;

[0016] The loss generation module is further configured to obtain a loss value corresponding to the predicted response content based on the advantage data corresponding to the at least two text units respectively;

[0017] A training module, configured to train the first model through the loss value to obtain a second model, where the second model is used to perform a text prediction task.

[0018] According to one aspect of the embodiments of the present application, there is provided a terminal device, the terminal device comprising a processor and a memory, where a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned training method for a text prediction model.

[0019] According to one aspect of the embodiments of the present application, there is provided a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned training method for a text prediction model.

[0020] According to one aspect of the embodiments of the present application, there is provided a computer program product, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium, and a processor reads and executes the computer program from the computer-readable storage medium to implement the above-mentioned training method for a text prediction model.

[0021] The technical solution provided by the embodiments of the present application can bring the following beneficial effects:

[0022] When training the first model through reinforcement learning, during the training process, the first model predicts the predicted response content based on the input sample prompt information. When obtaining the loss based on the predicted response content, by independently calculating the advantage data corresponding to each text unit in the predicted response content, the loss value of the predicted response content is obtained through the advantage data corresponding to each text unit, so as to train the first model through the loss value. That is, since when determining the loss of the predicted response content, a loss calculation method at the text unit (token) level is adopted, the loss of the predicted response content can include the independent influence of each text unit on the predicted response content. Furthermore, model update can take into account the influence of each text unit on the predicted response content. Even if the overall predicted response content shows negative advantage, if there are text units showing positive advantage, it can still provide effective reference information for the training of the model, thereby improving the training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a computer system provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of a method for training a text prediction model provided by an embodiment of the present application;

[0025] Figure 3 is a flowchart of a method for training a text prediction model provided by an embodiment of the present application;

[0026] Figure 4 is a flowchart of a method for training a text prediction model provided by an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of the reinforcement learning process for the first model provided by an embodiment of the present application;

[0028] Figure 6 is a flowchart of a method for training a large language text prediction model provided by an embodiment of the present application;

[0029] Figure 7 is a block diagram of a training device for a text prediction model provided by an embodiment of the present application;

[0030] Figure 8 is a block diagram of a training device for a text prediction model provided by an embodiment of the present application;

[0031] Figure 9 is a block diagram of the structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings.

[0033] Reinforcement learning: It is a machine learning method that learns the optimal behavior strategy through the interaction between an agent and an environment. The agent takes actions in the environment, and the environment gives rewards or punishments according to the agent's actions. The agent continuously adjusts its strategy based on these feedbacks to maximize the long-term cumulative reward. The core of reinforcement learning lies in balancing exploration and exploitation, that is, while the agent tries new actions to discover better strategies, it also uses the known optimal strategies to obtain rewards. In this way, reinforcement learning can solve complex decision-making problems and is widely applied in fields such as large language models, robot control, games, financial investment, recommendation systems, etc. It is one of the important research directions in the field of artificial intelligence.

[0034] Basic text unit (Token): It refers to the smallest unit segmented from text data in the field of natural language processing, usually a word, a phrase, or a symbol (such as a punctuation mark).

[0035] Please refer to Figure 1 , which shows a schematic diagram of a computer system provided by an embodiment of this application.

[0036] The computer system 100 includes: a terminal 120 and a server 140.

[0037] The terminal 120 installs and runs an application program with an image restoration function. The device types of the terminal 120 include at least one of a smart phone, a laptop computer, a desktop computer, a tablet computer, a smart robot, an AR (Augmented Reality) device, a VR (Virtual Reality) device, etc.

[0038] The terminal 120 is connected to the server 140 through a wireless network or a wired network.

[0039] Those skilled in the art can know that the number of the above devices can be more or less. For example, the above devices can be only one, or the above devices can be dozens or hundreds, or a larger number. The embodiments of this application do not limit the number and types of the devices.

[0040] The server 140 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 140 provides a functional service for implementing the model training function. Optionally, the server 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or, the server 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or, a distributed computing architecture is adopted between the server 140 and the terminal 120 for collaborative computing.

[0041] It should be noted that the above-mentioned server 140 can be implemented as a physical server or a cloud server in the cloud. In some embodiments, the above-mentioned server 140 can also be implemented as a node in a blockchain system.

[0042] Optionally, the training method of the text prediction model provided in the embodiments of the present application is independently executed by the terminal 120 or the server 140, or can also be implemented by the cooperation of the terminal 120 and the server 140, which is not limited herein.

[0043] In one example, taking the training method of the text prediction model being independently executed by the terminal 120 as an example. Schematically, a training framework for implementing machine learning model training runs in the terminal 120. The training framework refers to software tools and platforms for constructing, training, and optimizing machine learning / deep learning models. It provides developers with a series of tools, libraries, and application programming interfaces (APIs), making the development and training of models more efficient and convenient. After the training framework in the terminal 120 loads the first model to be trained, it obtains sample prompt information, which is used to guide the first model to output a predicted response content; through the first model, the predicted response content corresponding to the sample prompt information is predicted, and the predicted response content includes at least two text units; advantage data corresponding to at least two text units is generated, and the advantage data is used to indicate the value of the text unit relative to the first model when predicting the predicted response content; based on the advantage data corresponding to at least two text units, the loss value corresponding to the predicted response content is obtained; the first model is trained through the loss value to obtain the second model.

[0044] In one example, take the case where the training method of the text prediction model is implemented by the cooperation of the terminal 120 and the server 140. Schematically, the terminal 120 uploads the model structure of the first model to be trained and the sample prompt information for model training to the server 140, and the server 140 provides the model training function for the terminal 120 by invoking the model training service. After the model training service in the server 140 loads the first model to be trained, it obtains the sample prompt information, which is used to guide the first model to output the predicted response content; the predicted response content corresponding to the sample prompt information is obtained through the prediction of the first model, and the predicted response content includes at least two text units; advantage data corresponding to at least two text units is generated, and the advantage data is used to indicate the value of the text unit relative to the first model when predicting the predicted response content; based on the advantage data corresponding to at least two text units, the loss value corresponding to the predicted response content is obtained; the first model is trained through the loss value to obtain the second model.

[0045] In some alternative embodiments, the trained first model can be deployed in the server 140. The terminal 120 indicates the prompt text to the server 140. After receiving the prompt text, the server 140 inputs the to-be-prompted text into the first model to obtain the response text, and the server 140 returns the response text to the terminal 120.

[0046] Please refer to Figure 2 , which shows the flowchart of the training method of the text prediction model provided by an embodiment of the present application. This method is executed by a computer device (including Figure 1 the terminal 120 and / or the server 140 in

[0047] Step 210, obtain the sample prompt information.

[0048] The sample prompt information (Prompt) is used to guide the first model to output the predicted response content. Among them, Prompt is an input form used to guide the first model to generate a specific output or perform a specific task. Prompt is usually a piece of text that contains context information, instructions, or questions to help the first model understand the user's needs and generate corresponding answers or perform corresponding operations. In some scenarios, Prompt also includes graphic information combined with images.

[0049] Among them, the above first model is a machine learning model to be trained. Optionally, the first model can be implemented as a natural language model, a large language model, an image text extraction model, a machine translation model, a medical Q&A model, a legal Q&A model, etc., which is not limited here.

[0050] Optionally, the sample prompt information can be implemented as a sample prompt statement, sample prompt words, sample prompt document, sample prompt graphic and text, etc., which are not limited herein.

[0051] In one example, when the first model is implemented as a large language model, the sample prompt information can be implemented as a sample prompt statement, which includes a question for the large language model to reply to the user's needs. For example, "How to improve work efficiency using the Pomodoro Technique?"; in another example, when the first model is implemented as an image text extraction model, the sample prompt information can be implemented as sample prompt graphic and text, which includes the target image for text extraction and the prompt text indicating the model to perform text extraction; in another example, when the first model is implemented as a machine translation model, the sample prompt information includes a sample prompt statement and sample prompt words. Among them, the sample prompt statement includes the original text statement to be translated, and the sample prompt words include the language type to be translated into, for example, "‘It's a beautiful day today!’,Japanese".

[0052] In some embodiments, the sample prompt information is data sampled from a sample set. Schematically, a sample set is obtained, which includes multiple sample prompt information, and the sample prompt information input to the first model is sampled from the sample set.

[0053] Optionally, the sampling methods of the sample prompt information include random sampling, grouped random sampling, weighted random sampling, etc., which are not limited herein.

[0054] In one example, taking the random sampling method as an example, the sample prompt information in the sample set corresponds to information numbers, and the information numbers of all sample prompt information form a number sampling range, for example, numbers 0 to 10000. A target number is randomly sampled from the number sampling range, and the sample prompt information corresponding to the target number is obtained. For example, the above target number is generated by using a random number generator. Among them, the random sampling method is adopted each time a sample is input to the first model.

[0055] In another example, taking the grouped random sampling method as an example, the sample prompt information in the sample set is divided into at least two sample groups, each sample group contains at least two sample prompt information, and the at least two sample groups respectively correspond to group numbers, forming a number sampling range. A target number is randomly sampled from the number sampling range, and the sample group corresponding to the target number is obtained. Batch training is completed through at least two sample prompt information in the sample group. For example, the above target number is generated by using a random number generator.

[0056] In another example, taking weighted random sampling as an example, the sample hint information in the sample set is divided into at least two sample groups, each sample group contains at least two sample hint information, and the at least two sample groups respectively correspond to group numbers. Before starting random sampling of the sample groups, initialize sampling weights for the at least two sample groups respectively, and the sampling weights are used to indicate the probability of selecting the corresponding sample group during random sampling. Use a random number generator configured with the sampling weights corresponding to each sample group to randomly sample a target number from the numbered sampling range, obtain the sample group corresponding to the target number, and complete batch training through at least two sample hint information in the sample group. Update the sampling weights corresponding to each sample group, where the sampling weight corresponding to the sample group with the sampled target number decreases, and the sampling weights corresponding to the sample groups with other numbers increase. In one example, the above-mentioned initialized sampling weights are the same initial weights for each sample group.

[0057] Optionally, obtain one sample hint information; or, obtain multiple sample hint information.

[0058] Step 220, predict the predicted response content corresponding to the sample hint information through the first model, and the predicted response content includes at least two text units.

[0059] Schematically, input the obtained sample hint information into the first model, and the first model generates the predicted response content based on the hint content of the sample hint information.

[0060] The first model refers to a machine learning model that can understand and generate human language. By learning the patterns and structures of language, it can predict the next word or phrase in a text sequence, thereby realizing various language-related tasks.

[0061] Optionally, the first model includes a large language model, an N-Gram model, a Recurrent Neural Network (RNN) model, a Long Short-Term Memory (LSTM) model, a Transformer model, etc., which are not limited here.

[0062] The predicted response content includes a text sequence output in natural language. Optionally, the predicted response content includes text content in at least one language, where the above languages include Chinese, English, Japanese, Korean, French, etc., which are not limited here. In some scenarios, the predicted response content also includes graphic and text content combined with images.

[0063] In one example, when the first model is implemented as a large language model, the predicted response content output by the first model based on the sample prompt information can be implemented as predicted text content. For example, when the sample prompt information input to the first model is "How to use the Pomodoro Technique to improve work efficiency?", the output predicted text content is "The Pomodoro Technique is a simple and easy time management method that improves work efficiency by dividing working time into multiple fixed-length Pomodoro times (usually 25 minutes). The following are specific steps to improve work efficiency using the Pomodoro Technique..."; in another example, when the first model is implemented as an image text prompt, the predicted response content output by the first model based on the sample prompt information can be implemented as predicted text content. For example, when the sample prompt information input to the first model is "How to use the Pomodoro Technique to improve work efficiency?", the output predicted text content is "The Pomodoro Technique is a simple and easy time management method that improves work efficiency by dividing working time into multiple fixed-length Pomodoro times (usually 25 minutes). The following are specific steps to improve work efficiency using the Pomodoro Technique..." When taking the model, the predicted reply content output by the first model based on the sample prompt information can be implemented as predicted graphic content, and the predicted graphic content includes the target image indicated by the sample prompt information or a partial screenshot of the target image, and the output text of the recognition result of the text content in the target image; in another example, when the first model is implemented as a machine translation model, the predicted reply content output by the first model based on the sample prompt information can be implemented as translated text content. For example, when the sample prompt information is "'The weather is so nice today!', Japanese", the corresponding output translated text content is "Today is a nice day!".

[0064] In some embodiments, after the predicted reply content is output by the first model, the predicted reply content is segmented to obtain at least two text units.

[0065] Optionally, the word segmentation rules for the predicted reply content may include at least one of the following:

[0066] Divide the predicted answer content into word-based text units;

[0067] Divide the predicted answer content into word-based text units;

[0068] Divide the predicted response content into phrase-based text units;

[0069] Divide the predicted answer content into short sentences as the basic text unit;

[0070] Use punctuation marks as dividing lines to divide the predicted responses.

[0071] Optionally, the word segmentation method for the predicted reply content can be implemented as at least one of the following: word segmentation through regular expressions; word segmentation through the Natural Language Toolkit (NLTK); word segmentation through Bidirectional Encoder Representations from Transformers (BERT), etc., which are not limited here.

[0072] Step 230: Generate advantage data corresponding to at least two text units respectively.

[0073] Among them, the advantage data is used to indicate the value of the text unit relative to the first model when predicting the predicted response content, that is, the advantage data is used to indicate the influence of the text unit on the prediction process when the first model predicts the predicted response content.

[0074] In the training process of the first model in the embodiments of the present application, reinforcement learning is used. Among them, reinforcement learning refers to a machine learning process in which an agent tries and learns in an environment, gradually optimizing its behavioral strategy to maximize the long-term cumulative reward. In the embodiments of the present application, the first model is the agent in the reinforcement learning process.

[0075] In reinforcement learning, advantage is used to measure the value of a specific action relative to the average action in a specific state. In the training process of the first model, the advantage data of the text unit is used to indicate the value of the text unit for the predicted response content when the first model is in the current state.

[0076] Optionally, the advantage data corresponding to the text unit includes positive advantage, negative advantage, and zero advantage. Among them, positive advantage represents a positive contribution of the text unit to the response effect of the currently output predicted response content; negative advantage represents a negative contribution of the text unit to the response effect of the currently output predicted response content; zero advantage represents that the text unit has no contribution to the response effect of the currently output predicted response content.

[0077] In some embodiments, the advantage data corresponding to at least two text units is determined by a pre-set advantage function. Schematically, obtain the pre-set advantage function, input at least two text units into the pre-set advantage function respectively, and obtain the advantage data corresponding to at least two text units respectively.

[0078] Optionally, the above pre-set advantage function includes a basic advantage function (the difference between the state-action reward signal and the state reward signal), a temporal difference error (TD error) function, a generalized advantage estimation (GAE) function, an n-step advantage function, a policy gradient-based advantage function, etc., which are not limited herein.

[0079] In some embodiments, the pre-set advantage function calculates the advantage data based on the reward signals corresponding to the text units. Illustratively, obtain the reward signals corresponding to at least two text units respectively; based on the reward signals corresponding to at least two text units respectively, generate the advantage data corresponding to at least two text units respectively.

[0080] The reward signal is used to indicate the quality impact of the text unit on the generated predicted response content. Optionally, the reward signal includes a positive reward, a negative reward, and a zero reward. Among them, the positive reward refers to a signal with a positive reward value, indicating that the text unit is beneficial to the generation quality of the predicted response content; the negative reward refers to a signal with a negative reward value, indicating that the text unit is harmful to the generation quality of the predicted response content; the zero reward refers to a signal with a zero reward value, indicating that the text unit has no direct impact on the generation quality of the predicted response content.

[0081] In some embodiments, perform normalization processing on the reward signals corresponding to at least two text units respectively through a pre-set advantage function to generate the advantage data corresponding to at least two text units respectively. Optionally, the implementation methods of the normalization processing include: at least one of Min-Max Normalization, Z-Score Normalization, Decimal Scaling Normalization, MaxAbs Normalization, median normalization, Log Transformation normalization, etc., which are not limited herein.

[0082] In an example, taking the above normalization processing implemented as Z-Score Normalization as an example, illustratively, determine the mean value of the reward signals and the standard deviation of the reward signals among at least two text units; based on the mean value of the reward signals and the standard deviation of the reward signals, perform normalization processing on the reward signals corresponding to at least two text units respectively to generate the advantage data corresponding to at least two text units respectively.

[0083] Specifically, the calculation process of the advantage data for the i-th text unit includes: subtracting the mean value of the reward signals from the reward signal of the i-th text unit to obtain the relative advantage data corresponding to the i-th text unit, where i is a positive integer; taking the quotient of the relative advantage data and the standard deviation of the reward signals as the advantage data of the i-th text unit.

[0084] Exemplarily, as shown in Formula 1, it shows the advantage function Advantage provided by an exemplary embodiment of the present application.

[0085]

[0086] Among them, reward i represents the reward signal of the i-th text unit, and rewards ans represents the reward signal corresponding to the predicted response content. mean() represents calculating the mean value, and std() represents calculating the standard deviation.

[0087] Optionally, the reward signal rewards corresponding to the above-mentioned predicted response content ans can be implemented as a reward signal obtained by taking the predicted response content as a whole; or, the reward signal rewards corresponding to the above-mentioned predicted response content ans can be implemented as a reward signal obtained by accumulating the reward signals corresponding to at least two text units respectively i respectively.

[0088] The advantage data of the text unit is generated through the reward signal, so that the advantage data of the text unit can directly represent the prediction effect under the current side model parameters of the first model, and can improve the learning efficiency of the model.

[0089] Step 240, based on the advantage data corresponding to at least two text units respectively, obtain the loss value corresponding to the predicted response content.

[0090] Among them, the loss value corresponding to the predicted response content is used to measure the prediction effect of the predicted response content.

[0091] Optionally, the advantage data corresponding to at least two text units respectively is used as the loss value corresponding to the predicted response content, that is, the prediction effect of the predicted response content is characterized by the advantage data corresponding to at least two text units respectively.

[0092] Optionally, the loss value corresponding to the predicted response content is obtained by accumulating the advantage data corresponding to at least two text units respectively, that is, the prediction effect of the response content is characterized by the sum of the advantage data corresponding to at least two text units.

[0093] In some embodiments, when obtaining the loss value corresponding to the predicted response content by accumulating the advantage data corresponding to at least two text units respectively, it is implemented in a weighted accumulation manner. Schematically, the unit weights corresponding to at least two text units are obtained, and after weighting and adding the advantage data corresponding to at least two text units respectively through the unit weights, the loss value corresponding to the predicted response content is obtained.

[0094] Optionally, the method for obtaining the unit weights corresponding to the two text units respectively can be implemented as follows: determining the unit weights based on the occurrence frequency of the text units in the predicted response content. Schematically, obtaining the frequency information corresponding to at least two text units respectively, where the frequency information is used to indicate the occurrence frequency of the text units in the corresponding predicted response content; normalizing the frequency information corresponding to at least two text units respectively to obtain the unit weights corresponding to the corresponding text units.

[0095] Optionally, the frequency information corresponding to the i-th text unit can be implemented as at least one of the following: the term frequency (TF), inverse document frequency (IDF), term frequency-inverse document frequency (TF-IDF), relative frequency, etc. of the i-th text unit, which is not limited herein.

[0096] In some embodiments, taking the training process of the first model as a supervised training process as an example, the sample prompt information corresponds to a sample response content, where the sample response content is a reference response given under the prompt of the sample prompt information. Schematically, the method for obtaining the loss value corresponding to the predicted response content is implemented as follows: obtaining the sample response content corresponding to the sample prompt information, where the sample response content includes at least two sample text units, obtaining the unit loss values corresponding to at least two text units respectively based on the alignment relationship between at least two sample text units in the sample response content and at least two text units in the predicted response content; converting the advantage data corresponding to at least two text units respectively into loss weights; and obtaining the loss value corresponding to the predicted response content by weighted summation of the unit loss values with the loss weights.

[0097] In some embodiments, the method for determining the alignment relationship between the sample text unit and the text unit can be implemented as follows: obtaining the semantic similarity between the n-th sample text unit and the m-th text unit; for the m-th text unit, selecting the k sample text units with the highest semantic similarity between them from at least two sample text units as the reference text units of the m-th text unit, thereby establishing the alignment relationship between the sample text unit and the text unit, where n, m, and k are all positive integers.

[0098] Optionally, the semantic similarity between the n-th sample text unit and the m-th text unit can be implemented as at least one of the following:

[0099] First, convert the nth sample text unit into a second word vector, and convert the mth text unit into a first word vector; use the vector distance between the second word vector and the first word vector as the semantic similarity between the nth sample text unit and the mth text unit.

[0100] Among them, word vectors are the vector representations corresponding to words or phrases in a high-dimensional space. Word vectors can capture the semantic and syntactic information of text units, enabling computers to process and understand language data more effectively.

[0101] Optionally, the generation methods of the second word vector and the first word vector can be implemented as at least one of the following: One-Hot Encoding, Word to Vector (Word2Vec) algorithm, Global Vectors for Word Representation (GloVe) algorithm, FastText algorithm, etc., which are not limited here.

[0102] Optionally, the vector distance between the second word vector and the first word vector can be implemented as at least one of the following: Euclidean Distance, Manhattan Distance, Cosine Similarity, Cosine Distance, Chebyshev Distance, Jaccard Similarity, Hamming Distance, etc., which are not limited here.

[0103] Second, obtain the semantic similarity between the nth sample text unit and the mth text unit through a preset semantic resource.

[0104] Among them, the semantic lexicon is a resource for describing and organizing lexical semantic information. It contains semantic information such as the definitions of words, synonyms, antonyms, hyponymy relations, etc. The semantic lexicon is usually organized in a structured manner. For example, a dictionary (listing words as entries and providing information such as definitions, synonyms, antonyms, etc.), a semantic network (representing the semantic relationships between words in the form of a graph, with nodes representing words and edges representing relationships).

[0105] In one example, taking the implementation of a semantic resource as a semantic network as an example, the first word node corresponding to the nth sample text unit and the second word node corresponding to the mth text unit are determined from the semantic network, and the path length from the first word node to the second word node in the semantic network is used as the semantic similarity between the nth sample text unit and the mth text unit.

[0106] Step 250: Train the first model with the loss value to obtain the second model.

[0107] Schematically, after obtaining the loss value, the gradient of the loss value with respect to the model parameters of the first model is calculated by the backpropagation algorithm, and the model parameters of the first model are updated according to the calculated gradient using a preset optimization algorithm to obtain the second model, which is used to perform the text prediction task.

[0108] Optionally, the above preset optimization algorithms include at least one of the Gradient Descent algorithm, Stochastic Gradient Descent (SGD) algorithm, Nesterov Accelerated Gradient (NAG) algorithm, Adaptive Gradient Algorithm (AdaGrad), Root Mean Square Propagation (RMSProp) algorithm, Adaptive Moment Estimation (Adam) algorithm, etc., which are not limited herein.

[0109] In summary, when training the first model through reinforcement learning, during the training process, the first model predicts the predicted response content through the input sample hint information. When obtaining the loss through the predicted response content, the advantage data corresponding to each text unit in the predicted response content is calculated independently, so as to obtain the loss value of the predicted response content through the advantage data corresponding to each text unit, and the first model is trained with the loss value. That is, since the loss calculation method at the text unit (token) level is adopted when determining the loss of the predicted response content, the loss of the predicted response content can include the independent value of each text unit for the predicted response content, and further, the model update can take into account the influence of each text unit on the predicted response content. Even if the predicted response content as a whole shows negative advantage, if there are text units showing positive advantage, it can still provide effective reference information for the training of the model, thus improving the training efficiency of the model.

[0110] In some alternative embodiments, the loss in the training process of the first model is determined by the reward signal corresponding to the text unit and the advantage data. Please refer to Figure 3 , which shows a flowchart of a method for training a text prediction model provided by an exemplary embodiment of the present application. The method may include at least one of the following steps (241-242).

[0111] Step 241, obtain the reward signals respectively corresponding to at least two text units.

[0112] Among them, the reward signal is used to indicate the influence of the text unit on the quality of the generated predicted response content. That is, the reward signal is used to measure the quality of the predicted response content generated by the first model under the current model parameters at the text unit level, so as to guide the first model to optimize its generation strategy.

[0113] Optionally, the reward signal includes a positive reward, a negative reward, and a zero reward. Among them, the positive reward refers to a signal with a positive reward value, indicating that the text unit is beneficial to the generation quality of the predicted response content; the negative reward refers to a signal with a negative reward value, indicating that the text unit is harmful to the generation quality of the predicted response content; the zero reward refers to a signal with a zero reward value, indicating that the text unit has no direct influence on the generation quality of the predicted response content.

[0114] Optionally, the reward signal includes at least one of an immediate reward, an accumulated reward, a sparse reward, a dense reward, etc. Among them, the immediate reward is the immediate feedback provided by the environment according to the action a of the agent t and the current state s t at each time step t; the accumulated reward refers to the sum of all immediate rewards from the current time step t to the end of the task; the sparse reward means that the reward signal is very sparse, that is, the reward is only given under specific conditions. The sparse reward enables the agent to explore more states and actions to find the path to obtain the reward; the dense reward means that the reward is given at each time step, usually used to provide more frequent feedback to help the agent learn faster.

[0115] In some embodiments, the above reward signal is obtained through a pre-set reward function. Schematically, obtain the pre-set reward function; input at least two text units into the reward function respectively to obtain the reward signals respectively corresponding to at least two text units.

[0116] Among them, the reward function is a function designed according to the influence of the text unit on the quality of the predicted response content. Optionally, the reward signal calculated by the reward function includes at least one of the following:

[0117] · Accuracy Reward

[0118] The accuracy reward is used to measure the matching degree between the predicted response text generated by the first model and the true sample response text. The accuracy reward of a text unit indicates the contribution of the text unit to the similarity between the predicted response content and the sample response content.

[0119] Optionally, the contribution of the text unit to the similarity between the predicted response content and the sample response content can be manifested in at least one of the semantic similarity contribution and the word order similarity contribution. Among them, the semantic similarity contribution indicates the contribution brought by the similarity between the semantics expressed by the text unit and the semantics of the predicted response content and the semantics of the sample response content; the word order similarity contribution indicates the contribution brought by the position of the text unit in the predicted response content to the word order similarity between the predicted response content and the sample response content.

[0120] In some embodiments, the calculation method of the semantic similarity contribution of a text unit is implemented as follows: obtaining the sample response text corresponding to the sample prompt information; segmenting the sample response text into at least two sample text units; obtaining the semantic similarity corresponding to each text unit and at least two sample text units, that is, obtaining the semantic similarity between the nth sample text unit and the mth text unit; for the mth text unit, obtaining the semantic similarity corresponding to the mth text unit and at least two sample text units, and determining the number of similar units of the sample text unit whose semantic similarity with the mth text unit reaches the first similarity threshold; converting the above number of similar units into a reward signal.

[0121] In an example, when the number of similar units corresponding to the mth text unit reaches the first quantity threshold, the reward signal of the mth text unit is output as the first positive reward; when the number of similar units corresponding to the mth text unit reaches the second quantity threshold and does not reach the first quantity threshold, the reward signal of the mth text unit is output as the first zero reward; when the number of similar units corresponding to the mth text unit does not reach the second quantity threshold, the reward signal of the mth text unit is output as the first negative reward. The reward signal reward corresponding to the mth text unit m As shown in Formula Two.

[0122] Formula Two:

[0123] Among them, N is the number of similar units of the mth text unit, Y is the first quantity threshold, and Z is the second quantity threshold.

[0124] Optionally, the semantic similarity between the nth sample text unit and the mth text unit can be implemented as at least one of the following:

[0125] First, convert the nth sample text unit into a second word vector, and convert the mth text unit into a first word vector; use the vector distance between the second word vector and the first word vector as the semantic similarity between the nth sample text unit and the mth text unit.

[0126] Among them, the word vector is the vector representation corresponding to a word or phrase in a high-dimensional space.

[0127] Optionally, the generation methods of the second word vector and the first word vector can be implemented as at least one of the following: one-hot encoding, Word2Vec, GloVe algorithm, FastText, etc., which are not limited here.

[0128] Optionally, the vector distance between the second word vector and the first word vector can be implemented as at least one of the following: Euclidean distance, Manhattan distance, cosine similarity, cosine distance, Chebyshev distance, Jaccard similarity, Hamming distance, etc., which are not limited here.

[0129] Second, obtain the semantic similarity between the nth sample text unit and the mth text unit through a preset semantic resource.

[0130] Among them, the semantic resource is a resource for describing and organizing lexical semantic information, which contains semantic information such as the definition, synonyms, antonyms, hyponymy relations of words. The semantic resource is usually organized in a structured manner. For example, a dictionary (with words as entries, providing information such as definitions, synonyms, antonyms, etc.), a semantic network (representing the semantic relations between words in the form of a graph, with nodes representing words and edges representing relations).

[0131] In an example, taking the semantic resource as a semantic network as an example, determine the first word node corresponding to the nth sample text unit and the second word node corresponding to the mth text unit in the semantic network, and use the path length from the first word node to the second word node in the semantic network as the semantic similarity between the nth sample text unit and the mth text unit.

[0132] In some embodiments, the calculation method of the contribution of the word order similarity of the text unit is implemented as follows: obtain the sample response text corresponding to the sample prompt information; obtain the longest common subsequence (Longest Common Subsequence, LCS) between the sample response text and the predicted response text; when the mth text unit appears in the longest common subsequence, output the reward signal of the mth text unit as the second positive reward; when the mth text unit does not appear in the longest common subsequence, output the reward signal of the mth text unit as the second negative reward or the second zero reward.

[0133] Among them, the longest common subsequence refers to the longest subsequence shared by two sequences. Optionally, the subsequence can be continuous or discontinuous in the original sequence.

[0134] · Fluency Reward

[0135] The fluency reward is used to measure the naturalness and grammatical correctness of the predicted response text generated by the first model. The fluency reward of a text unit indicates the contribution of the text unit to the fluency of the predicted response content.

[0136] In some embodiments, the reward signal for the fluency of a text unit is determined by calculating the usage of grammar in the predicted response content by the text unit. Schematically, when the grammar of the text unit in the predicted response content is correct, the reward signal of the text unit is a positive reward; when the grammar of the text unit in the predicted response content is incorrect, the reward signal of the text unit is a negative reward.

[0137] · Relevance Reward

[0138] The relevance reward is used to measure the relevance between the text generated by the first model and the input context. The relevance reward of a text unit indicates the contribution of the text unit to the context relevance of the predicted response content.

[0139] In some embodiments, the perplexity corresponding to at least two text units is obtained, and the reward signal corresponding to the text unit is output based on the perplexity corresponding to at least two text units respectively.

[0140] Among them, perplexity is an index to measure the prediction difficulty of a language model for text. A lower perplexity indicates that the language model has more confidence in predicting the text, usually meaning that the text is more fluent.

[0141] In one example, the perplexity corresponding to the m-th text unit is implemented by Formula 3.

[0142] Formula 3: Perplexity(w m |w 1:m-1 ) = P(w m |w 1:m-1 ) -1

[0143] Among them, P(w m |w 1:m-1 ) represents the probability that the first model predicts the m-th text unit under the condition of the first m - 1 text units, w m represents the m-th text unit, and w 1:m-1 represents the first m - 1 text units.

[0144] · Diversity Reward

[0145] The diversity reward is used to encourage the first model to generate diverse texts and avoid generating repetitive or single content. The diversity reward of a text unit indicates the contribution of the text unit to the diversity of the predicted response content.

[0146] In some embodiments, the diversity contribution of a text unit to the predicted response content is determined by calculating the variance of the word vectors of the text unit. Schematically, at least two text units are respectively converted into corresponding first word vectors; the variance between the first word vector corresponding to the m-th text unit and the first word vectors corresponding to at least two text units is obtained; and the reward signal corresponding to the m-th text unit is determined based on the variance corresponding to the m-th text unit.

[0147] In one example, when the variance corresponding to the m-th text unit reaches the first variance threshold, the reward signal of the m-th text unit is output as the third positive reward; when the variance corresponding to the m-th text unit reaches the second variance threshold and does not reach the first variance threshold, the reward signal of the m-th text unit is output as the third zero reward; when the variance corresponding to the m-th text unit does not reach the second variance threshold, the reward signal of the m-th text unit is output as the third negative reward.

[0148] Step 242, based on the advantage data corresponding to at least two text units and the reward signals corresponding to at least two text units respectively, obtain the loss value corresponding to the predicted response content.

[0149] In some embodiments, a corresponding unit loss value is calculated for each text unit respectively. Schematically, based on the advantage data corresponding to at least two text units and the reward signals corresponding to at least two text units respectively, obtain the unit loss values corresponding to at least two text units respectively, where the j-th unit loss value is calculated based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit, and j is a positive integer; fuse the unit loss values corresponding to at least two text units respectively to obtain the loss value corresponding to the predicted response content.

[0150] In some other embodiments, the unit loss values corresponding to at least two text units are used as the loss value corresponding to the predicted response content.

[0151] That is, since the optimization of each token is carried out independently, the model can obtain more refined feedback during the generation process, which not only improves the stability of training but also significantly enhances the training convergence speed. Especially in long-chain reasoning and generation tasks, token-level loss calculation enables the model to converge to a better strategy faster without relying on the balanced loss of the entire predicted response content.

[0152] Meanwhile, in long-chain reasoning tasks, incorrect tokens often affect the reasoning effect of the entire predicted response content. The sample-level loss calculation method used in related technologies may not be able to effectively suppress these low-quality tokens because their effects will be "diluted" by the contributions of other correct tokens. In the embodiments of the present application, through token-level loss calculation, each low-quality token can be punished in a timely manner, thereby reducing the generation of meaningless repetitions and verbose content and optimizing the quality of model generation.

[0153] In some embodiments, obtaining the unit loss value corresponding to at least two text units respectively based on the advantage data corresponding to at least two text units respectively and the reward signals corresponding to at least two text units respectively is implemented as: obtaining the advantage data corresponding to at least two text units respectively; obtaining the reward signals corresponding to at least two text units respectively; for the j-th text unit, multiplying the reward signal of the j-th text unit by the advantage data to obtain the unit loss value.

[0154] In some embodiments, in order to prevent the training instability caused by the excessive update of the strategy guided by the loss value, a clipping mechanism is introduced to limit the range of the loss when determining the loss value. Schematically, obtaining the i-th first candidate loss value based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit; obtaining a preset upper gradient threshold and a lower gradient threshold; obtaining the j-th second candidate loss value based on the upper gradient threshold and the advantage data corresponding to the j-th text unit; obtaining the j-th third candidate loss value based on the lower gradient threshold and the advantage data corresponding to the j-th text unit; determining the j-th unit loss value based on the comparison relationship among the j-th first candidate loss value, the j-th second candidate loss value, and the j-th third candidate loss value.

[0155] That is to say, through the clipping mechanism, the update amplitude brought by the loss corresponding to each token can be limited, preventing the gradient explosion or training instability caused by excessive model parameter updates, thereby maintaining the stability of training.

[0156] In summary, when training the first model through reinforcement learning, during the training process, the first model predicts the predicted response content through the input sample prompt information. When obtaining the loss through the predicted response content, by independently calculating the advantage data corresponding to each text unit in the predicted response content, the loss value of the predicted response content is obtained through the advantage data corresponding to each text unit, so as to train the first model through the loss value. That is, since when determining the loss of the predicted response content, a loss calculation method at the text unit (token) level is adopted, the loss of the predicted response content can include the independent value of each text unit for the predicted response content. Furthermore, the model update can take into account the influence of each text unit on the predicted response content. Even if the overall performance of the predicted response content shows a negative advantage, if there are text units showing a positive advantage, it can still provide effective reference information for the training of the model, thereby improving the training efficiency of the model.

[0157] In the embodiments of the present application, by refining the loss calculation granularity to each token level, it is ensured that each generated token can obtain separate reward feedback according to its contribution to the final result. If a certain token contributes to the error of the generated sample, it will be independently punished and will not lose the learning opportunity due to the error of the entire sample. Therefore, the model can obtain more effective training signals in a shorter time, significantly improving the training efficiency.

[0158] At the same time, adopting the token-level loss calculation method ensures that even if the generated predicted response content is incorrect, the model can still obtain immediate reward signals through the feedback of each token. This fine-grained reward feedback mechanism can ensure that each decision step (each token generation) in the reinforcement learning process can contribute to the optimization of the model, thereby accelerating the training process.

[0159] In some optional embodiments, during the training process of the first model, the model training is achieved by sampling multiple outputs of the model. Please refer to Figure 4 , which shows a flowchart of a training method for a text prediction model provided by an exemplary embodiment of the present application. The method may include at least one of the following steps (401-406).

[0160] Step 401, obtain sample prompt information.

[0161] The sample prompt information is used to guide the first model to output the predicted response content. Optionally, the sample prompt information can be implemented as a sample prompt statement, sample prompt words, sample prompt documents, sample prompt graphics and texts, etc., which are not limited herein.

[0162] In some embodiments, the sample prompt information is data sampled from a sample set. Schematically, a sample set is obtained, which includes multiple sample prompt information, and the sample prompt information input into the first model is sampled from the sample set.

[0163] Optionally, the sampling method of the sample prompt information includes random sampling, grouped random sampling, weighted random sampling, etc., which are not limited herein.

[0164] Step 402: Input the sample prompt information into the first model, and the first model performs at least two reply predictions to obtain at least two predicted reply contents.

[0165] In the embodiments of the present application, at least two predicted reply contents are obtained through the sample prompt information, where the at least two predicted reply contents are prediction results for the sample prompt information in different prediction rounds.

[0166] In some embodiments, at least two predicted reply contents are sampled from the first model, and each predicted reply content includes at least two text units. Exemplarily, as shown in Formula Four, it indicates that G predicted reply contents o are sampled from the first model through the sample prompt information q Sampling to obtain G predicted reply contents o i .

[0167] Formula Four:

[0168] In the embodiments of the present application, the at least two predicted reply contents sampled are used as candidate samples for the optimization process of the first model, that is, used to calculate the loss for updating the first model.

[0169] Step 403: Perform word segmentation on at least two predicted reply contents respectively to obtain at least two text units corresponding to the predicted reply contents.

[0170] Optionally, the word segmentation rules for the predicted reply contents may include at least one of the following:

[0171] · Divide the predicted reply content based on characters as text units;

[0172] · Divide the predicted reply content based on words as text units;

[0173] · Divide the predicted reply content based on phrases as text units;

[0174] · Divide the predicted reply content based on short sentences as text units;

[0175] · Divide the predicted reply content based on punctuation marks as dividing lines.

[0176] Optionally, the word segmentation method for the predicted response content can be implemented as at least one of the following: word segmentation by regular expression; word segmentation by NLTK; word segmentation by BERT, etc., which is not limited herein.

[0177] Step 404, obtaining the advantage data corresponding to at least two text units in each predicted response content through an advantage function.

[0178] In some embodiments, the advantage data corresponding to at least two text units is determined through a preset advantage function. Schematically, obtain the preset advantage function, and input at least two text units into the preset advantage function respectively to obtain the advantage data corresponding to at least two text units respectively.

[0179] Optionally, the above preset advantage function includes a basic advantage function (the difference between the state-action reward signal and the state reward signal), a TD error function, a GAE function, an n-step advantage function, a policy gradient-based advantage function, etc., which is not limited herein.

[0180] In some embodiments, the preset advantage function calculates the advantage data based on the reward signal corresponding to the text unit. Schematically, obtain the reward signals corresponding to at least two text units respectively; based on the reward signals corresponding to at least two text units respectively, generate the advantage data corresponding to at least two text units respectively.

[0181] In some embodiments, the preset advantage function performs normalization processing on the reward signals corresponding to at least two text units respectively to generate the advantage data corresponding to at least two text units respectively. Optionally, the implementation methods of the normalization processing include at least one of: min-max normalization, Z-score normalization, decimal scaling normalization, maximum absolute value normalization, median normalization, logarithmic transformation normalization, etc., which is not limited herein.

[0182] Specifically, the calculation process of the advantage data for the i-th text unit includes: subtracting the reward signal mean of the i-th text unit from the reward signal of the i-th text unit to obtain the relative advantage data corresponding to the i-th text unit, where i is a positive integer; taking the quotient of the relative advantage data and the standard deviation of the reward signal as the advantage data of the i-th text unit.

[0183] In some embodiments, the importance sampling ratio corresponding to the text unit is calculated as the reward signal corresponding to the text unit.

[0184] Step 405, obtaining the loss values corresponding to at least two predicted response contents based on the advantage data corresponding to at least two text units in each predicted response content.

[0185] In some embodiments, a corresponding unit loss value is calculated for each text unit. Schematically, based on the advantage data respectively corresponding to at least two text units and the reward signals respectively corresponding to at least two text units, the unit loss values respectively corresponding to at least two text units are obtained, where the j-th unit loss value is calculated based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit, and j is a positive integer; the unit loss values respectively corresponding to at least two text units are fused to obtain the loss value between the reference response content and the predicted response content.

[0186] In some embodiments, obtaining the unit loss values respectively corresponding to at least two text units based on the advantage data respectively corresponding to at least two text units and the reward signals respectively corresponding to at least two text units is implemented as: obtaining the advantage data respectively corresponding to at least two text units; obtaining the reward signals respectively corresponding to at least two text units; for the j-th text unit, multiplying the reward signal of the j-th text unit by the advantage data to obtain the unit loss value.

[0187] In some embodiments, in order to prevent the training instability caused by the excessive update of the strategy guided by the loss value, a clipping mechanism is introduced to limit the range of the loss when determining the loss value. Schematically, the i-th first candidate loss value is obtained based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit; the preset upper gradient threshold and lower gradient threshold are obtained; the j-th second candidate loss value is obtained based on the upper gradient threshold and the advantage data corresponding to the j-th text unit; the j-th third candidate loss value is obtained based on the lower gradient threshold and the advantage data corresponding to the j-th text unit; based on the comparison relationship among the j-th first candidate loss value, the j-th second candidate loss value and the j-th third candidate loss value, the j-th unit loss value is determined.

[0188] In some embodiments, the unit loss function corresponding to the text unit is as shown in Formula Five.

[0189] Formula Five: L i,t =min(r i,t (θ)A i,t ,clip(r i,t (θ),1 - ∈ low ,1 + ∈ high )A i,t )

[0190] Where r i,t (θ) is the importance sampling ratio of the t-th text unit in the i-th predicted response content, representing the ratio of the t-th text unit relative to the previous model parameter state (policy) of the first model, A i,t is the advantage data of the t-th text unit in the i-th predicted response content, ∈low is the lower gradient threshold, ∈ high is the upper gradient threshold.

[0191] In some embodiments, the set of unit loss values of each text unit of each predicted response content is regarded as the loss values respectively corresponding to at least two predicted response contents.

[0192] Step 406: Update the advantage function and the model parameters of the first model through the loss value to obtain a second model.

[0193] Illustratively, after obtaining the loss value, calculate the gradient of the loss value with respect to the model parameters of the first model through the backpropagation algorithm, and use a preset optimization algorithm to update the model parameters of the first model according to the calculated gradient to obtain a second model.

[0194] Optionally, the above preset optimization algorithm includes at least one of a gradient descent algorithm, a stochastic gradient descent algorithm, a Nesterov momentum algorithm, an adaptive gradient algorithm, a root mean square propagation algorithm, an adaptive moment estimation algorithm, etc., which are not limited herein.

[0195] In some embodiments, during the training process of the first model, a comparison process with a reference policy is also introduced. Illustratively, input the sample prompt information into the reference first model to obtain a reference response content; obtain the difference information between at least two predicted response contents and the reference response content respectively; update the model parameters of the first model based on the above difference information and the advantage data respectively corresponding to at least two text units in each predicted response content to obtain a second model.

[0196] Optionally, the difference information between at least two predicted response contents and the reference response content respectively can be indicated by the Kullback-Leibler divergence (K-L divergence), that is, calculate the KL divergence between at least two predicted response contents and the reference response content respectively, and update the model parameters of the first model based on the KL divergence between at least two predicted response contents and the reference response content respectively, and the advantage data respectively corresponding to at least two text units in each predicted response content to obtain a second model. Among them, during the process of updating the model parameters, the model parameters of the first model are updated with the goal of minimizing the KL divergence and maximizing the advantage data.

[0197] Such as Figure 5As shown in the figure, it shows a schematic diagram of the reinforcement learning process for the first model provided by an exemplary embodiment of the present application. The sample prompt information 510 outputs multiple predicted response contents 520 through the first model of the current state (model parameters). Each predicted response content 520 includes multiple text units 521. The reward signal 530 corresponding to each text unit 521 is calculated, and the advantage data 540 corresponding to each text unit 521 is obtained through the reward signal 530.

[0198] The sample prompt information 510 outputs multiple reference response contents 550 by referring to the first model. The KL divergence 560 between the multiple reference response contents 550 and the multiple predicted response contents 520 is calculated. The model parameters of the first model are updated through the KL divergence 560 and the advantage data 540 corresponding to each text unit 521, thereby realizing the reinforcement learning of the first model.

[0199] In summary, when training the first model through reinforcement learning, during the training process, the first model predicts the predicted response content through the input sample prompt information. When obtaining the loss through the predicted response content, by independently calculating the advantage data corresponding to each text unit in the predicted response content, the loss value of the predicted response content is obtained through the advantage data corresponding to each text unit, so as to train the first model through the loss value. That is, since when determining the loss of the predicted response content, a loss calculation method at the text unit (token) level is adopted, the loss of the predicted response content can include the independent value of each text unit for the predicted response content. Furthermore, the model update can take into account the influence of each text unit on the predicted response content. Even if the predicted response content as a whole shows negative advantage, if there are text units showing positive advantage, it can still provide effective reference information for the training of the model, thereby improving the training efficiency of the model.

[0200] In some optional embodiments, the model training method provided by the embodiments of the present application can be applied to the training scenario of a large language model. Among them, the large language model is a natural language model that can process and generate natural language text. Please refer to Figure 6 which shows a flowchart of the training method of the large language text prediction model provided by an exemplary embodiment of the present application. This method can include at least one of the following steps (610-650).

[0201] Step 610, obtain sample prompt information.

[0202] The sample prompt information is used to guide the large language model to output the predicted response content. Among them, the large language model is a natural language model that can implement the question and answer function. That is, for the question or simple text input by the user, the large language model can give the response content for the prompt words in the question or simple text.

[0203] The sample prompt information used as the input to the large language model includes sample prompt statements, sample prompt words, sample prompt documents, sample prompt images, etc., which are not limited herein.

[0204] Step 620: Use the large language model to predict the predicted response content corresponding to the sample prompt information.

[0205] In some embodiments, the sample prompt information is data sampled from a sample set. Schematically, obtain a sample set, which includes multiple sample prompt information, and sample the sample prompt information input to the large language model from the sample set.

[0206] Optionally, the sampling methods of the sample prompt information include random sampling, grouped random sampling, weighted random sampling, etc., which are not limited herein.

[0207] In some embodiments, when the sample prompt information is implemented as a sample prompt statement, the sample prompt information can be text generated by a pre-trained public large language model.

[0208] The predicted response content includes at least two text units. In some embodiments, after the large language model outputs the predicted response content, the predicted response content is tokenized to obtain at least two text units.

[0209] Optionally, the tokenization rules for the predicted response content may include at least one of the following:

[0210] · Divide the predicted response content based on characters as the basic text units;

[0211] · Divide the predicted response content based on words as the basic text units;

[0212] · Divide the predicted response content based on phrases as the basic text units;

[0213] · Divide the predicted response content based on short sentences as the basic text units;

[0214] · Divide the predicted response content based on punctuation marks as the dividing lines.

[0215] Optionally, the tokenization methods for the predicted response content can be implemented as at least one of the following: tokenization through regular expressions; tokenization through the Natural Language Toolkit; tokenization through a bidirectional encoder, etc., which are not limited herein.

[0216] Step 630: Generate the advantage data corresponding to at least two text units respectively.

[0217] Among them, the advantage data is used to indicate the value of the text unit relative to the large language model when predicting the predicted response content.

[0218] Optionally, the advantage data corresponding to the text units includes positive advantages, negative advantages, and zero advantages. Among them, a positive advantage represents that the text unit makes a positive contribution to the response effect of the predicted response content of the current output; a negative advantage represents that the text unit makes a negative contribution to the response effect of the predicted response content of the current output; a zero advantage represents that the text unit makes no contribution to the response effect of the predicted response content of the current output.

[0219] In some embodiments, the advantage data corresponding to at least two text units is determined by a preset advantage function. Schematically, the preset advantage function is obtained, and at least two text units are respectively input into the preset advantage function to obtain the advantage data corresponding to at least two text units.

[0220] In some embodiments, the preset advantage function calculates the advantage data based on the reward signals corresponding to the text units. Schematically, the reward signals corresponding to at least two text units are obtained; based on the reward signals corresponding to at least two text units, the advantage data corresponding to at least two text units is generated.

[0221] In some embodiments, the reward signals corresponding to at least two text units are normalized to generate the advantage data corresponding to at least two text units.

[0222] In some embodiments, the mean value of the reward signals and the standard deviation of the reward signals between at least two text units are determined; based on the mean value of the reward signals and the standard deviation of the reward signals, the reward signals corresponding to at least two text units are normalized to generate the advantage data corresponding to at least two text units.

[0223] In some embodiments, the difference between the reward signal of the i-th text unit and the mean value of the reward signals is obtained to get the relative advantage data corresponding to the i-th text unit, where i is a positive integer; the quotient of the relative advantage data and the standard deviation of the reward signals is used as the advantage data of the i-th text unit.

[0224] Step 640, based on the advantage data corresponding to at least two text units, obtain the loss value corresponding to the predicted response content.

[0225] Optionally, the advantage data corresponding to at least two text units is used as the loss value corresponding to the predicted response content, that is, the prediction effect of the predicted response content is characterized by the advantage data corresponding to at least two text units.

[0226] Optionally, the advantage data corresponding to at least two text units is accumulated to obtain the loss value corresponding to the predicted response content, that is, the prediction effect of the response content is characterized by the sum of the advantage data corresponding to at least two text units.

[0227] In some embodiments, obtain reward signals corresponding to at least two text units respectively, where the reward signals are used to indicate the quality impact of the text units on the generated predicted response content; based on the advantage data corresponding to the at least two text units respectively and the reward signals corresponding to the at least two text units respectively, obtain the loss value corresponding to the predicted response content.

[0228] In some embodiments, based on the advantage data corresponding to the at least two text units respectively and the reward signals corresponding to the at least two text units respectively, obtain unit loss values corresponding to the at least two text units respectively, where the j-th unit loss value is calculated based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit, and j is a positive integer; fuse the unit loss values corresponding to the at least two text units respectively to obtain the loss value corresponding to the predicted response content.

[0229] In some embodiments, obtain the i-th first candidate loss value based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit; obtain a preset upper gradient threshold and a lower gradient threshold; obtain the j-th second candidate loss value based on the upper gradient threshold and the advantage data corresponding to the j-th text unit; obtain the j-th third candidate loss value based on the lower gradient threshold and the advantage data corresponding to the j-th text unit; determine the j-th unit loss value based on the comparison relationship among the j-th first candidate loss value, the j-th second candidate loss value, and the j-th third candidate loss value.

[0230] In some embodiments, obtain unit weights corresponding to at least two text units respectively; after weighting and adding the advantage data corresponding to the at least two text units respectively by the unit weights, obtain the loss value corresponding to the predicted response content.

[0231] In some embodiments, obtain frequency information corresponding to at least two text units respectively, where the frequency information is used to indicate the appearance frequency of the text units in the corresponding predicted response content; normalize the frequency information corresponding to the at least two text units respectively to obtain the unit weights of the corresponding text units.

[0232] In some embodiments, obtain a preset reward function; input the at least two text units into the reward function respectively to obtain reward signals corresponding to the at least two text units respectively.

[0233] In some embodiments, sample response content corresponding to sample prompt information is obtained, and the sample response content includes at least two sample text units; based on the alignment relationship between the at least two sample text units in the sample response content and the at least two text units in the predicted response content, unit loss values corresponding to the at least two text units are obtained; the advantage data corresponding to the at least two text units are converted into loss weights; the unit loss values are weighted and summed through the loss weights to obtain the loss value corresponding to the predicted response content.

[0234] Step 650, training the large language model through the loss value to obtain the trained large language model.

[0235] Illustratively, after obtaining the loss value, the gradient of the loss value with respect to the model parameters of the large language model is calculated through the backpropagation algorithm, and the model parameters of the large language model are updated using a preset optimization algorithm according to the calculated gradient to obtain the trained large language model.

[0236] Optionally, the above preset optimization algorithm includes at least one of a gradient descent algorithm, a stochastic gradient descent algorithm, a Nesterov momentum algorithm, an adaptive gradient algorithm, a root mean square propagation algorithm, an adaptive moment estimation algorithm, etc., which is not limited herein.

[0237] The training method of the large language text prediction model provided by the embodiments of the present application can optimize the training of the large language model through token-level loss calculation, thereby improving the training efficiency of the large language model, and can complete multiple rounds of training and optimization in a shorter time, thereby accelerating the development cycle of the large language model on the product side and quickly responding to market demands. Moreover, the improvement of training efficiency can reduce the consumption of computing resources, thereby reducing the overhead of hardware resources and computing time and lowering the R & D cost of the model.

[0238] It should be noted that the training method of the text prediction model provided by the embodiments of the present application can also be applied to the model training process of other scenarios. For example, it can be applied to the training of language translation models in machine translation scenarios, the training of lyric generation models in songwriting scenarios, the training process of medical / legal Q&A models in medical / legal Q&A scenarios, etc., which is not limited herein.

[0239] In one example, taking the training of a language translation model applied to the machine translation scenario as an example, schematically, obtain a sample text to be translated, which is used to guide the language translation model to output a translation result in a specified language; obtain the predicted translation content corresponding to the sample text to be translated through the language translation model, and the predicted translation content includes at least two text units; generate advantage data corresponding to each of the at least two text units, and the advantage data is used to indicate the value of the text unit relative to the language translation model when predicting the predicted translation content; obtain the loss value corresponding to the predicted translation content based on the advantage data corresponding to each of the at least two text units; train the language translation model through the loss value to obtain the trained language translation model.

[0240] In some embodiments, obtain reward signals corresponding to each of the at least two text units, and the reward signals are used to indicate the influence of the text units on the quality of the generated predicted translation content; generate advantage data corresponding to each of the at least two text units based on the reward signals corresponding to each of the at least two text units.

[0241] In some embodiments, perform normalization processing on the reward signals corresponding to each of the at least two text units to generate advantage data corresponding to each of the at least two text units.

[0242] In some embodiments, determine the mean value and standard deviation of the reward signals between the at least two text units; perform normalization processing on the reward signals corresponding to each of the at least two text units based on the mean value and standard deviation of the reward signals to generate advantage data corresponding to each of the at least two text units.

[0243] In some embodiments, subtract the mean value of the reward signals from the reward signal of the i-th text unit to obtain the relative advantage data corresponding to the i-th text unit, where i is a positive integer; use the quotient of the relative advantage data and the standard deviation of the reward signals as the advantage data of the i-th text unit.

[0244] In some embodiments, obtain reward signals corresponding to each of the at least two text units, and the reward signals are used to indicate the influence of the text units on the quality of the generated predicted translation content; obtain the loss value corresponding to the predicted translation content based on the advantage data corresponding to each of the at least two text units and the reward signals corresponding to each of the at least two text units.

[0245] In some embodiments, based on the advantage data respectively corresponding to at least two text units and the reward signals respectively corresponding to the at least two text units, unit loss values respectively corresponding to the at least two text units are obtained, wherein the j-th unit loss value is calculated based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit, and j is a positive integer; the unit loss values respectively corresponding to the at least two text units are fused to obtain the loss value corresponding to the predicted translation content.

[0246] In some embodiments, the i-th first candidate loss value is obtained based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit; a preset upper gradient threshold and a lower gradient threshold are obtained; the j-th second candidate loss value is obtained based on the upper gradient threshold and the advantage data corresponding to the j-th text unit; the j-th third candidate loss value is obtained based on the lower gradient threshold and the advantage data corresponding to the j-th text unit; the j-th unit loss value is determined based on the comparison relationship among the j-th first candidate loss value, the j-th second candidate loss value, and the j-th third candidate loss value.

[0247] In some embodiments, unit weights respectively corresponding to at least two text units are obtained; after weighting and adding the advantage data respectively corresponding to the at least two text units by the unit weights, the loss value corresponding to the predicted translation content is obtained.

[0248] In some embodiments, frequency information respectively corresponding to at least two text units is obtained, and the frequency information is used to indicate the occurrence frequency of the text unit in the corresponding predicted translation content; the frequency information respectively corresponding to the at least two text units is normalized to the unit weights of the corresponding text units.

[0249] In some embodiments, a preset reward function is obtained; the at least two text units are respectively input into the reward function to obtain the reward signals respectively corresponding to the at least two text units.

[0250] In some embodiments, a sample translation content corresponding to the sample text to be translated is obtained, and the sample translation content includes at least two sample text units; based on the alignment relationship between the at least two sample text units in the sample translation content and the at least two text units in the predicted translation content, unit loss values respectively corresponding to the at least two text units are obtained; the advantage data respectively corresponding to the at least two text units are converted into loss weights; the unit loss values are weighted and summed by the loss weights to obtain the loss value corresponding to the predicted translation content.

[0251] In one example, taking the training of a lyric generation model applied to the songwriting scenario as an example, schematically, sample prompt information is obtained, and the sample prompt information is used to guide the lyric generation model to output predicted lyric content; the predicted lyric content corresponding to the sample prompt information is predicted through a natural language model, and the predicted lyric content includes at least two text units; advantage data corresponding to the at least two text units is generated, and the advantage data is used to indicate the value of the text unit relative to the natural language model when predicting the predicted lyric content; based on the advantage data corresponding to the at least two text units respectively, a loss value corresponding to the predicted lyric content is obtained; the natural language model is trained through the loss value to obtain the trained natural language model.

[0252] In some embodiments, reward signals corresponding to at least two text units are obtained, and the reward signals are used to indicate the influence of the text units on the quality of the generated predicted lyric content; based on the reward signals corresponding to the at least two text units respectively, advantage data corresponding to the at least two text units is generated.

[0253] In some embodiments, normalization processing is performed on the reward signals corresponding to at least two text units respectively to generate advantage data corresponding to the at least two text units respectively.

[0254] In some embodiments, the mean value and standard deviation of the reward signals between at least two text units are determined; based on the mean value and standard deviation of the reward signals, normalization processing is performed on the reward signals corresponding to at least two text units respectively to generate advantage data corresponding to the at least two text units respectively.

[0255] In some embodiments, the difference between the reward signal of the i-th text unit and the mean value of the reward signals is calculated to obtain the relative advantage data corresponding to the i-th text unit, where i is a positive integer; the quotient of the relative advantage data and the standard deviation of the reward signals is used as the advantage data of the i-th text unit.

[0256] In some embodiments, reward signals corresponding to at least two text units are obtained, and the reward signals are used to indicate the influence of the text units on the quality of the generated predicted lyric content; based on the advantage data corresponding to the at least two text units respectively and the reward signals corresponding to the at least two text units respectively, a loss value corresponding to the predicted lyric content is obtained.

[0257] In some embodiments, unit loss values corresponding to at least two text units are obtained based on the advantage data corresponding to the at least two text units respectively and the reward signals corresponding to the at least two text units respectively, where the j-th unit loss value is calculated based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit, and j is a positive integer; the unit loss values corresponding to the at least two text units are fused to obtain a loss value corresponding to the predicted lyric content.

[0258] In some embodiments, a first candidate loss value of the i-th is obtained based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit; an upper limit gradient threshold and a lower limit gradient threshold set in advance are obtained; a second candidate loss value of the j-th is obtained based on the upper limit gradient threshold and the advantage data corresponding to the j-th text unit; a third candidate loss value of the j-th is obtained based on the lower limit gradient threshold and the advantage data corresponding to the j-th text unit; a unit loss value of the j-th is determined based on the comparison relationship among the first candidate loss value, the second candidate loss value, and the third candidate loss value of the j-th.

[0259] In some embodiments, unit weights corresponding to at least two text units are obtained; after weighting and summing the advantage data corresponding to at least two text units through the unit weights, a loss value corresponding to the predicted lyric content is obtained.

[0260] In some embodiments, frequency information corresponding to at least two text units is obtained, and the frequency information is used to indicate the occurrence frequency of the text unit in the corresponding predicted lyric content; the frequency information corresponding to at least two text units is normalized to the unit weight of the corresponding text unit.

[0261] In some embodiments, a preset reward function is obtained; at least two text units are respectively input into the reward function to obtain reward signals corresponding to at least two text units.

[0262] In some embodiments, sample lyric content corresponding to the sample prompt information is obtained, and at least two sample text units are included in the sample lyric content; based on the alignment relationship between at least two sample text units in the sample lyric content and at least two text units in the predicted lyric content, unit loss values corresponding to at least two text units are obtained; the advantage data corresponding to at least two text units is converted into loss weights; the unit loss values are weighted and summed through the loss weights to obtain a loss value corresponding to the predicted lyric content.

[0263] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the present application.

[0264] Please refer to Figure 7 , which shows a block diagram of a training apparatus for a text prediction model provided by an embodiment of the present application. The apparatus includes:

[0265] An acquisition module 710, configured to acquire sample prompt information, where the sample prompt information is used to guide the first model to output a predicted response content, and the first model is a machine learning model to be trained;

[0266] A prediction module 720, configured to predict, by using the first model, a predicted response content corresponding to the sample prompt information, where the predicted response content includes at least two text units;

[0267] A loss generation module 730, configured to generate advantage data corresponding to the at least two text units respectively, where the advantage data is used to indicate an influence of the text units on a prediction process when the first model predicts the predicted response content;

[0268] The loss generation module 730 is further configured to obtain a loss value corresponding to the predicted response content based on the advantage data corresponding to the at least two text units respectively;

[0269] A training module 740, configured to train the first model by using the loss value to obtain a second model, where the second model is used to perform a text prediction task.

[0270] In some alternative embodiments, as Figure 8 shown, the loss generation module 730 further includes:

[0271] An acquisition unit 731, configured to acquire reward signals corresponding to the at least two text units respectively, where the reward signals are used to indicate an influence of the text units on a quality of the generated predicted response content;

[0272] A generation unit 732, configured to generate the advantage data corresponding to the at least two text units respectively based on the reward signals corresponding to the at least two text units respectively.

[0273] In some alternative embodiments, the generation unit 732 is further configured to perform a normalization process on the reward signals corresponding to the at least two text units respectively to generate the advantage data corresponding to the at least two text units respectively.

[0274] In some alternative embodiments, the generation unit 732 is further configured to determine a reward signal mean value and a reward signal standard deviation between the at least two text units;

[0275] The generation unit 732 is further configured to perform a normalization process on the reward signals corresponding to the at least two text units respectively based on the reward signal mean value and the reward signal standard deviation to generate the advantage data corresponding to the at least two text units respectively.

[0276] In some alternative embodiments, the generation unit 732 is further configured to subtract the reward signal mean value from the reward signal of the i-th text unit to obtain relative advantage data corresponding to the i-th text unit, where i is a positive integer;

[0277] The generating unit 732 is further configured to use the quotient of the relative advantage data and the standard deviation of the reward signal as the advantage data of the i-th text unit.

[0278] In some alternative embodiments, the obtaining unit 731 is further configured to obtain the reward signals respectively corresponding to the at least two text units, where the reward signal is used to indicate the quality impact of the text unit on the generated predicted response content.

[0279] The generating unit 732 is further configured to obtain the loss value corresponding to the predicted response content based on the advantage data respectively corresponding to the at least two text units and the reward signals respectively corresponding to the at least two text units.

[0280] In some alternative embodiments, the generating unit 732 is further configured to obtain the unit loss values respectively corresponding to the at least two text units based on the advantage data respectively corresponding to the at least two text units and the reward signals respectively corresponding to the at least two text units, where the j-th unit loss value is calculated based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit, and j is a positive integer.

[0281] The loss generating module 730 further includes:

[0282] A fusion unit 733, configured to fuse the unit loss values respectively corresponding to the at least two text units to obtain the loss value corresponding to the predicted response content.

[0283] In some alternative embodiments, the generating unit 732 is further configured to obtain the i-th first candidate loss value based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit.

[0284] The obtaining unit 731 is further configured to obtain a preset upper gradient threshold and a preset lower gradient threshold.

[0285] The generating unit 732 is further configured to obtain the j-th second candidate loss value based on the upper gradient threshold and the advantage data corresponding to the j-th text unit.

[0286] The generating unit 732 is further configured to obtain the j-th third candidate loss value based on the lower gradient threshold and the advantage data corresponding to the j-th text unit.

[0287] The generating unit 732 is further configured to determine the j-th unit loss value based on the comparison relationship among the j-th first candidate loss value, the j-th second candidate loss value, and the j-th third candidate loss value.

[0288] In some alternative embodiments, the obtaining unit 731 is further configured to obtain the unit weights respectively corresponding to the at least two text units;

[0289] The fusion unit 733 is further configured to, after weighting and adding the dominant data respectively corresponding to the at least two text units by the unit weights, obtain the loss value corresponding to the predicted response content.

[0290] In some alternative embodiments, the obtaining unit 731 is further configured to obtain the frequency information respectively corresponding to the at least two text units, where the frequency information is used to indicate the occurrence frequency of the text unit in the corresponding predicted response content;

[0291] The generating unit 732 is further configured to normalize the frequency information respectively corresponding to the at least two text units into the unit weights of the corresponding text units.

[0292] In some alternative embodiments, the obtaining unit 731 is further configured to obtain a preset reward function;

[0293] The generating unit 732 is further configured to input the at least two text units into the reward function respectively to obtain the reward signals respectively corresponding to the at least two text units.

[0294] In some alternative embodiments, the obtaining module 710 is further configured to obtain the sample response content corresponding to the sample prompt information, where the sample response content includes at least two sample text units;

[0295] The generating unit 732 is further configured to obtain the unit loss values respectively corresponding to the at least two text units based on the alignment relationship between the at least two sample text units in the sample response content and the at least two text units in the predicted response content;

[0296] The generating unit 732 is further configured to convert the dominant data respectively corresponding to the at least two text units into loss weights;

[0297] The fusion unit 733 is further configured to perform a weighted sum of the unit loss values by the loss weights to obtain the loss value corresponding to the predicted response content.

[0298] It should be noted that, when the device provided in the foregoing embodiments realizes its functions, only the division of the foregoing functional modules is used for illustration. In actual applications, the foregoing functions may be allocated to different functional modules according to needs, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the foregoing embodiments and the method embodiments belong to the same concept, and the specific implementation process thereof can be found in the method embodiments and will not be elaborated here.

[0299] Please refer to Figure 9 , which shows a structural block diagram of a computer device 900 provided in an embodiment of the present application. The computer device 900 may be Figure 1 the terminal in the computer system shown, and is used to implement the training method of the text prediction model provided in the above embodiment. Specifically:

[0300] Generally, the computer device 900 includes: a processor 901 and a memory 902.

[0301] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some alternative embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0302] The memory 902 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some alternative embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store a computer program, and the computer program is configured to be executed by one or more processors to implement the above training method of the text prediction model.

[0303] In some alternative embodiments, the computer device 900 may further optionally include other components 903: a peripheral device interface and at least one peripheral device. The processor 901, the memory 902, and the peripheral device interface may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit, a display screen, an audio circuit, and a power supply.

[0304] Those skilled in the art can understand that Figure 9 the structure shown in does not constitute a limitation on the computer device 900, and it may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component layout.

[0305] In an exemplary embodiment, there is also provided a computer-readable storage medium storing a computer program, which when executed by a processor, implements the above-mentioned training method of the text prediction model. Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disc, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0306] In an exemplary embodiment, there is also provided a computer program product including a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to execute the above-mentioned training method of the text prediction model.

[0307] It should be noted that when collecting and processing relevant data (such as medical images, etc.) in the actual application of this application, it should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing behaviors within the scope authorized by laws and regulations and the personal information subject.

[0308] It should be understood that the "plurality" mentioned herein refers to two or more. "And / or" describes the relationship between associated objects and indicates that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. In addition, the step numbers described herein only exemplarily show a possible execution sequence between steps. In some other embodiments, the above steps may not be executed in the order of the numbers. For example, two steps with different numbers can be executed simultaneously, or two steps with different numbers can be executed in the reverse order of the illustration. The embodiments of the present application do not limit this.

[0309] The above are only exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a text prediction model, characterized in that, The method includes: Obtaining sample prompt information, which is used to guide a first model to output a predicted response content, and the first model is a machine learning model to be trained; Predicting, through the first model, the predicted response content corresponding to the sample prompt information, where the predicted response content includes at least two text units; Generating advantage data corresponding to the at least two text units respectively, where the advantage data is used to indicate the influence of the text units on the prediction process when the first model predicts the predicted response content; Based on the advantage data corresponding to the at least two text units respectively, obtaining a loss value corresponding to the predicted response content; Training the first model through the loss value to obtain a second model, and the second model is used to perform a text prediction task.

2. The method according to claim 1, wherein The generating the advantage data corresponding to the at least two text units respectively includes: Obtaining reward signals corresponding to the at least two text units respectively, where the reward signals are used to indicate the influence of the text units on the quality of the generated predicted response content; Based on the reward signals corresponding to the at least two text units respectively, generating the advantage data corresponding to the at least two text units respectively.

3. The method according to claim 2, wherein The based on the reward signals corresponding to the at least two text units respectively, generating the advantage data corresponding to the at least two text units respectively includes: Performing a normalization process on the reward signals corresponding to the at least two text units respectively to generate the advantage data corresponding to the at least two text units respectively.

4. The method according to claim 3, wherein The performing a normalization process on the reward signals corresponding to the at least two text units respectively to generate the advantage data corresponding to the at least two text units respectively includes: Determining a reward signal mean value and a reward signal standard deviation among the at least two text units; Based on the reward signal mean value and the reward signal standard deviation, performing a normalization process on the reward signals corresponding to the at least two text units respectively to generate the advantage data corresponding to the at least two text units respectively.

5. The method according to claim 4, characterized in that, The based on the reward signal mean value and the reward signal standard deviation, performing a normalization process on the reward signals corresponding to the at least two text units respectively to generate the advantage data corresponding to the at least two text units respectively includes: Subtracting the reward signal of the i-th text unit from the reward signal mean value to obtain a relative advantage data corresponding to the i-th text unit, where i is a positive integer; Taking the quotient of the relative advantage data and the reward signal standard deviation as the advantage data of the i-th text unit.

6. The method according to any one of claims 1 to 5, characterized in that, Based on the advantage data corresponding to the at least two text units respectively, obtaining a loss value corresponding to the predicted response content includes: Obtaining reward signals corresponding to the at least two text units respectively, where the reward signals are used to indicate the influence of the text units on the quality of the generated predicted response content; Based on the advantage data corresponding to the at least two text units respectively and the reward signals corresponding to the at least two text units respectively, obtaining a loss value corresponding to the predicted response content.

7. The method according to claim 6, characterized in that, Obtaining the loss value corresponding to the predicted response content based on the advantage data respectively corresponding to the at least two text units and the reward signals respectively corresponding to the at least two text units includes: Based on the advantage data respectively corresponding to the at least two text units and the reward signals respectively corresponding to the at least two text units, obtaining the unit loss values respectively corresponding to the at least two text units, where the j-th unit loss value is calculated based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit, and j is a positive integer; Fusing the unit loss values respectively corresponding to the at least two text units to obtain the loss value corresponding to the predicted response content.

8. The method according to claim 7, characterized in that, The method further includes: Obtaining the i-th first candidate loss value based on the advantage data corresponding to the j-th text unit and the reward signal corresponding to the j-th text unit; Obtaining a preset upper gradient threshold and a lower gradient threshold; Obtaining the j-th second candidate loss value based on the upper gradient threshold and the advantage data corresponding to the j-th text unit; Obtaining the j-th third candidate loss value based on the lower gradient threshold and the advantage data corresponding to the j-th text unit; Determining the j-th unit loss value based on the comparison relationship among the j-th first candidate loss value, the j-th second candidate loss value, and the j-th third candidate loss value.

9. The method according to claim 7, wherein The fusing the unit loss values respectively corresponding to the at least two text units to obtain the loss value corresponding to the predicted response content includes: Obtaining the unit weights respectively corresponding to the at least two text units; After weighting and adding the advantage data respectively corresponding to the at least two text units by the unit weights, obtaining the loss value corresponding to the predicted response content.

10. The method according to claim 9, wherein The obtaining the unit weights respectively corresponding to the at least two text units includes: Obtaining the frequency information respectively corresponding to the at least two text units, where the frequency information is used to indicate the occurrence frequency of the text unit in the corresponding predicted response content; Normalizing the frequency information respectively corresponding to the at least two text units into the unit weights of the corresponding text units.

11. The method according to claim 6, wherein The obtaining the reward signals respectively corresponding to the at least two text units includes: Obtaining a preset reward function; Inputting the at least two text units into the reward function respectively to obtain the reward signals respectively corresponding to the at least two text units.

12. According to the method described in any one of claims 1 to 5, characterized in that, The obtaining the loss value corresponding to the predicted response content based on the advantage data respectively corresponding to the at least two text units includes: Obtaining the sample response content corresponding to the sample prompt information, where the sample response content includes at least two sample text units; Based on the alignment relationship between the at least two sample text units in the sample response content and the at least two text units in the predicted response content, obtaining the unit loss values respectively corresponding to the at least two text units; Converting the advantage data respectively corresponding to the at least two text units into loss weights; Weighted summing the unit loss values by the loss weights to obtain the loss value corresponding to the predicted response content.

13. A training device for a text prediction model, characterized in that, The device includes: An acquisition module, configured to acquire sample prompt information for guiding the first model to output a predicted response content, where the first model is a machine learning model to be trained; A prediction module, configured to predict, through the first model, a predicted response content corresponding to the sample prompt information, where the predicted response content includes at least two text units; A loss generation module, configured to generate advantage data corresponding to each of the at least two text units, where the advantage data is used to indicate the influence of the text unit on the prediction process when the first model predicts the predicted response content; The loss generation module is further configured to obtain a loss value corresponding to the predicted response content based on the advantage data corresponding to each of the at least two text units; A training module, configured to train the first model through the loss value to obtain a second model, where the second model is used to perform a text prediction task.

14. A terminal device, characterized in that, The terminal device includes a processor and a memory, and a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the training method of the text prediction model according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the training method of the text prediction model according to any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes a computer program, the computer program is stored in a computer-readable storage medium, and a processor reads and executes the computer program from the computer-readable storage medium to implement the training method of the text prediction model according to any one of claims 1 to 12.

Images