Method, electronic device and computer program product for building training data

Abstract

The embodiment of the invention relates to a method for constructing training data, electronic equipment and a computer program product. The method includes determining a plurality of clusters by clustering cues within a training data set. A plurality of sampling probabilities corresponding to the plurality of clusters are determined based on a plurality of cohesion degrees of the plurality of clusters, where the cohesion degrees indicate intra-cluster distances within the clusters. The method further includes determining a target cluster for sampling according to the plurality of sampling probabilities. The method further includes constructing target training data by sampling a target cue word from the target cluster. According to the embodiment of the invention, when the language model is finely adjusted, the cue words can be screened according to the clustering result of the cue words, so that the determined cue words have higher annotation value, and the comprehensiveness and diversification of the output result of the language model obtained by training are ensured.

Description

Technical Field

[0001] The present disclosure relates generally to the field of machine learning, and more particularly to methods, electronic devices, and computer program products for constructing training data. Background Art

[0002] With the development of artificial intelligence learning paradigm, language model (LM) has gradually become a hot topic in the field of artificial intelligence. Language model has learned rich language knowledge and world knowledge through pre-training on massive text data, so that it can achieve amazing results in various natural language processing (NLP) tasks.

[0003] In order to make the output of the language model more in line with human preferences, the language model can be fine-tuned by using human feedback for reinforcement learning, so that it can generate more accurate, logically coherent and readable output results. In the process of fine-tuning the language model, more efficient prompts need to be used to guide the language model to improve the training efficiency of the language model. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for constructing training data.

[0005] According to a first aspect of the present disclosure, a method for constructing training data is provided. The method includes determining a plurality of clusters by clustering prompt words in a training data set. Based on a plurality of cohesion degrees of the plurality of clusters, a plurality of sampling probabilities corresponding to the plurality of clusters are determined, wherein the cohesion degrees indicate the intra-cluster distances within the clusters. The method also includes determining a target cluster for sampling according to the plurality of sampling probabilities. The method also includes constructing target training data by sampling target prompt words from the target clusters. Thus, the prompt words in the training data set can be screened multiple times, so that the prompt words obtained after the screening have more annotation value. Thus, the annotation workload of the annotation personnel can be reduced, and the annotation cost and processing resources can be saved. In addition, sampling the target prompt words from each target cluster can avoid the randomness of prompt word sampling. In this way, it can be ensured that the output results of the language model are further improved to be more diversified and personalized, and more in line with human preferences, while saving annotation resources.

[0006] According to a second aspect of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a method for constructing training data, the method comprising: determining multiple clusters by clustering prompt words in a training data set. Based on multiple cohesion degrees of the multiple clusters, multiple sampling probabilities corresponding to the multiple clusters are determined, wherein the cohesion degrees indicate the intra-cluster distance within the cluster. The method also includes determining a target cluster for sampling based on the multiple sampling probabilities. The method also includes constructing target training data by sampling target prompt words from the target cluster.

[0007] According to a third aspect of the present disclosure, there is provided a computer program product, which is tangibly stored on a non-volatile computer-readable medium and comprises machine-executable instructions, which when executed cause a machine to perform the steps of the method in the first aspect of the present disclosure.

[0008] The purpose of this Summary is to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of the workflow for training a language model through reinforcement learning based on human feedback is shown;

[0011] Figure 2 A schematic diagram showing an example environment in which various embodiments according to the present disclosure may be implemented;

[0012] Figure 3 A schematic diagram showing reward model training according to multiple embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic flow chart of further screening target prompt words according to multiple embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic flow chart of a method for constructing training data according to multiple embodiments of the present disclosure is shown;

[0015] Figure 6A schematic diagram of a workflow for determining candidate output results using a beam search algorithm according to multiple embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0017] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0018] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0021] Language Model (LM) is an important technology in the field of natural language processing and text generation, with broad application prospects and development space. Among them, a language model is a model that uses deep learning technology to learn language rules and knowledge based on a large amount of text data, so as to generate natural and fluent text. Large language models have powerful expressive and generalization capabilities and can be applied to various natural language processing tasks, such as machine translation, text summarization, dialogue systems, question-answering systems, etc.

[0022] With the continuous development of algorithms, hardware, data, and application scenarios, users' performance requirements for language models are also increasing. In order to make the language model generate different outputs according to different users, applications, and application scenarios, the language model can be fine-tuned. Fine-tuning refers to further adjusting and training the language model using labeled data in a specific application scenario to make the language model more suitable for a specific application scenario.

[0023] Reinforcement learning with human feedback (RLHF) is a popular technique in recent years to improve the performance of language models. Among them, RLHF is a technique that combines reinforcement learning with human feedback, in which human preferences are used as reward signals to guide language models to generate high-quality language output. RLHF can use diverse feedback providers to help the model learn to generate text that better represents different viewpoints, making it more versatile and effective in various contexts. However, when training a language model, labelers label a large amount of unlabeled training data (prompt words), which consumes a lot of human resources. Moreover, due to the randomness of prompt word sampling, the language model may not be able to access a more comprehensive language pattern, which in turn causes the output of the language model to not meet the user's expected output.

[0024] Figure 1 FIG. 4 shows a schematic diagram of the workflow for training a language model using RLHF. Figure 1 As shown, the prompt word 102 and the response result 104 (annotated data) written by the annotator to the prompt word can be input into the language model 106 (for example, GPT-3.5) to pre-train the language model 106. Then, the language model 106 can generate multiple outputs 108 corresponding to the prompt word, such as output 108-1, output 108-2, and output 108-3. The annotator evaluates the generated multiple outputs 108 and sorts the multiple outputs 108 from the best output to the worst output to obtain a sorting result 110. Further, a reward model 112 is trained to predict the scores of multiple outputs of the language model. By training the outputs 108 and ranking scores of the language model 106, the reward model 112 can create a mathematical representation of human preferences. Finally, using the reinforcement learning method, the reward model 112 is used as a reward function to fine-tune the language model 106 so that the language model 106 can generate an output result with a higher score in the reward model 112.

[0025] Furthermore, before the prompt words and response results are input into the language model for fine-tuning, the embodiments of the present disclosure can filter the prompt words so that the user can specifically mark the filtered prompt words. In the embodiments of the present disclosure, the prompt words in the training data set can be clustered to determine multiple clusters. Then, the cohesion degree corresponding to each cluster is determined based on the intra-cluster distance of each cluster. The cohesion degree is used to represent the degree of similarity between the prompt words contained in the cluster. Further, based on the cohesion degree corresponding to each cluster, multiple sampling probabilities corresponding to each cluster are determined. Then, based on the multiple sampling probabilities, the target cluster for sampling is determined. Finally, the target prompt words are sampled from the target cluster to construct the training data.

[0026] Thus, the prompt words in the training data set can be screened so that the prompt words obtained after screening have more annotation value. Thus, the annotation workload of the annotators can be reduced, and the annotation cost and processing resources can be saved. In addition, sampling the target prompt words from each target cluster can avoid the randomness of prompt word sampling. In this way, it can be ensured that the output results of the language model are further improved to be more diversified and personalized, and more in line with human preferences, while saving annotation resources.

[0027] Figure 2 2 is a schematic diagram of an example environment 200 in which various embodiments of the present disclosure may be implemented. Figure 2 As shown, the example environment includes a computing device 201. In some embodiments, the computing device 201 can be implemented as various user terminals or service terminals with computing capabilities. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the computing device 100 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0028] like Figure 2As shown, the computing device 201 includes a workflow for training a language model using a training data set, wherein the workflow includes three stages. The first stage is the supervised fine-tuning (SFT) stage, in which the collected text data can be used to pre-train the language model in a self-supervised manner, and then the language model can be fine-tuned using manually annotated data. In box 202, a training data set can be obtained, for example, the training data set can be obtained directly from an open platform, or open text data can be collected from the Internet to construct a training data set. The training data set contains multiple prompt words, such as "How to use the ×× function", "Telling fairy tales to a 6-year-old child", etc.

[0029] In box 204, after obtaining the training data set, the prompt words in the training data set can be sampled to construct target training data for fine-tuning the language model. The prompt word refers to the text input to the language model, which can be used to prompt or guide the large model to give an expected output result. The type of prompt word can be question and answer, summary, translation, etc. In box 206, the prompt words in the training data set can be clustered to determine multiple clusters. For example, the prompt words can be clustered using a clustering algorithm, and each prompt word can be automatically classified into different groups to form multiple clusters. Each cluster can represent the theme of a class of prompt words.

[0030] In box 208, due to the different cohesion levels of different clusters, for example, the closeness between the prompt words and the centroid in the cluster is different. In order to make the training data used to train the language model more comprehensive and avoid the randomness of the prompt word sampling, the sampling probability corresponding to each cluster can be determined according to the cohesion level of each cluster. The lower the cohesion level, the lower the similarity between the prompt words contained in the cluster, and the theme of the cluster does not fully cover the prompt words in the cluster. Furthermore, the sampling probability of the cluster can be determined according to the cohesion level corresponding to the cluster. Among them, the sampling probability refers to the probability of constructing training data by sampling the prompt words in the cluster. For example, the sampling probability of the cluster with a lower cohesion level can be increased.

[0031] In box 210, the target cluster for sampling can be determined according to the sampling probability. For example, several clusters with higher sampling probability can be determined as target clusters. In box 212, after the target cluster is determined, the training data can be constructed by sampling the target prompt words from the target cluster. In this way, the user does not need to mark all the prompt words in the training data set, but only needs to mark the sampled prompt words. Thereby, the manual marking work can be reduced and a large amount of marking cost can be saved. In addition, since the target prompt words are sampled from the target cluster, the quality of data marking is higher and more valuable. In box 214, after sampling the target prompt words, the marking personnel are required to mark the target prompt words, such as writing the output results or response results corresponding to the target prompt words. In box 216, the target prompt words and the corresponding output are combined to construct the target training data. The language model is fine-tuned using the target training data.

[0032] Continue to refer Figure 2 , the second stage is the reward model training stage. In the reward model training stage, the evaluation results of the user on multiple output results of the language model can be obtained, and the reward model can be trained using the scores so that the reward model can judge the quality of the language model output results in the future. In reinforcement learning, the reward function can evaluate (score) the current output, so that the policy model produces higher quality output. Accordingly, in the process of training the language model using reinforcement learning, the language model is regarded as a policy model. Among them, the reward model can be another fine-tuned language model, or it can be a language model trained from scratch based on preference data.

[0033] In box 218, the target prompt words can be further screened according to the predicted output results corresponding to the target prompt words. In box 220, the screened target prompt words are input into the language model, and the output results corresponding to the screened target prompt words are output after being processed by the language model. In box 222, the annotator can evaluate the output results of the screened target prompt words and sort the output results according to preferences and application requirements. In box 224, after receiving the user's sorting, the reward model can be trained using the sorting. Figure 3 A schematic diagram of reward model training according to multiple embodiments of the present disclosure is shown.

[0034] like Figure 3As shown, the sampled target prompt word 302 can be output to the supervised fine-tuned language model (SFTModel) 106, and multiple outputs 304, such as A, B, and C, can be obtained after being processed by the SFT model 106. The labeler can evaluate the multiple outputs 304 and determine the ranking 306 between the multiple outputs 304. For example, the labeler likes A by 37%, B by 49%, and C by 14%. Finally, the ranking 306 and the multiple outputs 304 can be input into the reward model (Reward Model) 112 to train the reward model 112. In this way, the scoring results of the reward model obtained by training can be consistent with human values, so that the results output by the fine-tuned language model are more in line with the user's preferences.

[0035] In some embodiments, in order to further save manual annotation resources and enable the annotation personnel to focus their annotation efforts on more "valuable" target prompt words, the target prompt words can be further screened. If the target prompt word is not precise (the intention contained in the prompt word is unclear), the answer generated by the language model may be very broad, and the "response certainty" of the generated answer is not high. In order to make the language model output more accurate, more logically coherent and readable output, the annotation personnel can be requested to further annotate the target prompt word, for example, evaluate the output result corresponding to the target prompt word to determine the answer that best meets the user's needs. Accordingly, multiple target prompt words can be input into the language model to determine the predicted output results corresponding to the multiple target prompt words. If the predicted output results are large or the difference is large, the target prompt word needs to be further annotated. Figure 4 A schematic diagram of a process for further screening target prompt words according to multiple embodiments of the present disclosure is shown.

[0036] like Figure 4 As shown, in box 402, multiple target prompt words in the training data can be input into the language model in sequence. In box 404, the language model can output the predicted output results corresponding to the target prompt words. The number of predicted output results can be multiple. In box 406, the target prompt words can be further screened according to the predicted output results. For example, target prompt words with a large number of predicted output results or a large difference between predicted output results are selected. In box 408, after the target prompt words are screened, the target training data can be updated, and the annotation personnel can be requested to further annotate the screened target prompt words. For example, the output results corresponding to the screened target prompt words can be sorted to determine the sorting results.

[0037] Continue to refer Figure 2The third stage is the stage of fine-tuning the language model using a reinforcement learning algorithm. For example, a policy gradient reinforcement learning (Policy Gradient RL) algorithm or a proximal policy optimization (PPO) can be used to fine-tune some or all parameters of the language model. In box 226, prompt words can be sampled from the updated target training data and the sampled prompt words can be input into the language model. In box 228, the output generated by the language model for the prompt words can be obtained. In box 230, an evaluation score of the output is calculated using a reward model. In box 232, the language model is fine-tuned using a reinforcement learning algorithm using the evaluation score.

[0038] References Figures 1 to 4 The scheme of constructing training data and using the training data to fine-tune the language model in an embodiment of the present disclosure is described. Compared with the traditional method, the embodiment of the present disclosure can screen the prompt words in the training data set multiple times when constructing the training data, so that the prompt words obtained after screening have more annotation value. This can reduce the annotation workload of the annotation personnel and save annotation resources. In some embodiments, since the annotation data used in the process of fine-tuning the language model are indicator words of "incomplete instruction information" and "unclear output results", the language model obtained by fine-tuning is more comprehensive in contact with the language pattern, and the output answer is more in line with the user's preferences.

[0039] Figure 5 1 shows a schematic flow chart of a method 500 for constructing training data according to multiple embodiments of the present disclosure. The method 500 may be performed by, for example Figure 2 It is implemented by the computing device 200 shown. It should be understood that the method 500 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0040] like Figure 5 As shown, in box 502, the computing device can obtain a training data set and cluster the prompt words in the training data set to determine multiple clusters. The training data set can be generated by an application programming interface (API) prompt, or it can be collected from questions and answers in a public NLP data set. The training prompt word contains multiple prompt words. Among them, the prompt word refers to the input text used to trigger or guide the large model to generate content. The input text can be text in various languages, for example, English, Chinese, French, etc. Exemplarily, the prompt word can be "How to make scrambled eggs with tomatoes", "Introducing the physical scientist Newton", "Recommended weight loss recipes", etc.

[0041] Furthermore, after obtaining the training data set, the prompt words in the training data set are clustered to determine multiple clusters or multiple clusters. That is, clustering refers to dividing the training data set into different clusters or clusters. Exemplarily, there are many types of clustering methods, which can be mainly divided into partition-based clustering methods, density-based clustering methods, hierarchical clustering methods, etc. Among them, partition-based clustering methods can include K-means clustering algorithm, K-means++ clustering algorithm, etc. When clustering, similar prompt words can be divided together to form a cluster. In this way, the themes of the prompt words contained in a cluster are roughly the same.

[0042] In box 504, the cohesion degree corresponding to the cluster can be determined based on the intra-cluster distance of the cluster, and the sampling probability of the cluster can be determined using the cohesion degree. The intra-cluster distance is used to characterize the distance between each prompt word contained in the cluster and the centroid of the cluster. The smaller the intra-cluster distance is, the closer the prompt words in the cluster are, and the higher the cohesion degree corresponding to the cluster is. Correspondingly, the topic coverage of the cluster is higher, that is, the similarity of the prompt words in the cluster is higher (for example, the type similarity is higher). Further, after determining the cohesion degree corresponding to the cluster, the sampling probability of the cluster can be determined based on the cohesion degree. The sampling probability is used to characterize the probability or possibility of sampling the prompt word from the cluster.

[0043] In some embodiments, in order to ensure that the prompt words annotated by the annotation personnel are worthy of annotation, the sampling probability corresponding to the cluster with a lower cohesion degree can be increased, and the sampling probability corresponding to the cluster with a higher cohesion degree can be reduced. As mentioned above, the prompt words contained in the cluster with a lower cohesion degree have a low similarity, so it is necessary to sample multiple prompt words from the cluster to ensure the comprehensiveness and representativeness of the sampling. Accordingly, the sampling probability is negatively correlated with the cohesion degree and positively correlated with the distance within the cluster. Wherein, the negative correlation refers to that the correlation coefficient between the sampling probability and the cohesion degree is a negative number, and the sampling probability is low when the cohesion degree is high. The positive correlation refers to that the correlation coefficient between the sampling probability and the distance within the cluster is a negative number, and the sampling probability is low when the distance within the cluster is short.

[0044] In some embodiments, the distance between each prompt word and the centroid in the cluster is calculated respectively. The statistical value of each distance is used as the intra-cluster distance of the cluster. Among them, the statistical value can be the sum, average value, median value, square sum value, etc. of each distance. In some embodiments, the distance between the prompt word and the centroid can be Euclidean distance, Manhattan distance, cosine distance, etc. The mean value of all prompt words in the cluster is usually called the "centroid" of the cluster. In other embodiments, the degree of cohesion can also be determined according to the weighted value of the distance between each prompt word in the cluster and the distance between each prompt word and the centroid.

[0045] In block 506, the target cluster for sampling can be determined according to the sampling probabilities corresponding to the clusters. For example, the sampling probabilities corresponding to the clusters can be sorted, and the clusters with the top 50% sampling probabilities can be used as the target clusters. This can avoid the randomness of the prompt word sampling, and further ensure that the prompt words and language patterns exposed to the trained language model are more comprehensive.

[0046] In block 508, target training data is constructed by sampling multiple target prompt words from the target clusters. For example, multiple target prompt words may be sampled from multiple target clusters at one time, or the sampling step may be repeated to sample multiple target prompt words.

[0047] In this way, when constructing training data, the prompt words in the training data set can be sampled, so that the sampled prompt words have more annotation value. This can reduce the annotation workload of the annotation personnel and save annotation resources. In addition, the sampling target prompt words are collected from clusters with high sampling probability, thereby avoiding the randomness of prompt word sampling, so that the output of the fine-tuned language model is more comprehensive and diversified.

[0048] In some embodiments, the sampling probabilities of multiple target clusters can be counted to determine the sampling probability distribution. Afterwards, the sampling method of the target prompt word can be determined based on the sampling probability distribution. For example, if the sampling probability distribution conforms to the normal distribution, the target cluster can be sampled according to the sampling method corresponding to the normal distribution. In some embodiments, in order to ensure the uniformity of sampling, the target prompt word can be sampled from each target cluster using a uniform sampling method. Among them, the uniform sampling method can include roulette, function transformation method, etc.

[0049] In some embodiments, the number of target sampling words sampled from the target cluster can be determined based on the probability density function corresponding to the sampling probability distribution. For example, when the sampling probability of target cluster A is 70%, the sampling probability of target cluster B is 30%, and the total number of sampled target prompt words is 100, it can be set to sample 70 target prompt words from target cluster A and 30 target prompt words from target cluster B. Thus, the predetermined sampling number corresponding to target cluster A is 70, and the predetermined sampling number corresponding to target cluster B is B. It is understandable that after determining the predetermined sampling number, the target prompt words can be sampled from the target cluster at one time, or multiple target prompt words can be sampled from the target cluster in sequence. The probability of each prompt word in the target cluster being sampled is the same.

[0050] In some embodiments, before clustering the prompt words in the training data set, in order to improve the efficiency and stability of clustering, the prompt words can be encoded into embeddings in the form of vectors using a language model. The embedding of the prompt words contains the semantic information of the prompt words. The embeddings of each prompt word can be vectors of the same dimension, and operations such as addition and dot product can be performed. Therefore, the similarity between the embeddings corresponding to two prompt words can be determined by calculating the cosine distance, Euclidean distance, etc. Therefore, the prompt words can be clustered using the similarity between the embeddings corresponding to the prompt words.

[0051] In some embodiments, after sampling the prompt words in the training data set, the annotator can be requested to write a response result for the target prompt word. Thus, after receiving the response result written by the annotator, the response result and the prompt word can be combined to construct the target training data. Thus, annotation resources can be saved. The target training data is input into the language model to train the language model (for example, Figure 2 Based on this approach, the trained language model can give outputs that meet the user's preferences when faced with a variety of prompt words, thereby improving the user's usage experience.

[0052] In some embodiments, similar prompt words may exist in the training data set, for example, "delicious hot pot in Beijing" and "which hot pot in Beijing is delicious" are two equivalent prompt words. Therefore, before constructing the target training data, the training data set can be deduplicated to save processing resources. A hash algorithm can be used to deduplicate the training data set. When deduplicating, each word in the training data set can be generated with a hash value through a hash function, and the hash value can be in the form of a fixed-bit binary. Thus, the similarity of the two prompt words can be calculated based on the hash values ​​corresponding to the two prompt words, and one of the prompt words with higher similarity can be removed. The benefit of doing this is that a relatively "clean" training data set can be provided for subsequent training data construction tasks, thereby saving computing resources, and also ensuring the accuracy of subsequent target clustering to a certain extent.

[0053] In certain embodiments, in order to further save manual annotation resources, the annotation personnel will focus on the target prompt words with more "value" in the annotation, and the target prompt words can be further screened. Generally, if the prompt words are imprecise (for example, the intention contained in the prompt words is unclear), the answer generated by the language model may be very broad. That is, the "response certainty" corresponding to the prompt words is not high. In order to make the language model output more accurate, more logically coherent and readable output, the target prompt words can be further screened, and the annotation personnel are requested to evaluate the output results corresponding to the target prompt words after screening, to determine the answer that best meets the user's needs. Therefore, in certain embodiments, multiple target prompt words can be input into the language model, determine the predicted output results corresponding to multiple target prompt words, if the predicted output results are more or the difference is larger, then the target prompt words need to be further marked. In this way, the target prompt words can be screened again, and the annotation resources can be further saved.

[0054] In some embodiments, the target prompt word can be screened according to the number and similarity of the predicted output results. For example, if the number of predicted output results corresponding to a certain prompt word is large, or the similarity between the predicted output results is not high, the target prompt word can be marked again. The marking again is to input the target prompt word into the language model, obtain multiple candidate output results, and request the user to evaluate the multiple candidate output results, for example, asking the user whether the candidate output result is reasonable, accurate, meaningful, etc.

[0055] After determining the ranking, the ranking can be used to reward the model (e.g. Figure 2In this way, the output results corresponding to the filtered target prompt words are sorted according to human preferences, so that the trained language model can output answers that meet human preferences in the future.

[0056] In some embodiments, the essential task of the language model may be to predict what the next word is. That is, given a word sequence, the probability distribution of the next word is calculated. Accordingly, the output of the language model may be a text sequence. The text sequence may contain multiple texts, such as a first text, a second text, and a third text. The text may be in the form of words. Exemplarily, the text sequence is "I like cats", the first text is "I", the second text is "like", and the third text is "cats".

[0057] Generally, in the process of obtaining the decoder output of the language model, the candidate text with the highest probability is selected at each time step as the output of the current time step until the output ends. This may cause the output text sequence to be incoherent. Although the output at each time step is indeed the most likely, the overall probability is not necessarily the highest. Therefore, in some embodiments, the language model can use a beam search algorithm to determine multiple candidate output results corresponding to the target prompt word. The beam search algorithm will retain more candidate texts when determining the output at each time step. The number of candidate texts can be set by the user according to actual application requirements.

[0058] Figure 6 FIG. 2 shows a schematic diagram of a workflow for determining candidate output results using a beam search algorithm according to multiple embodiments of the present disclosure. Figure 6 As shown, the beam search includes three stages, corresponding to three time steps (at each time step, the candidate texts constituting the candidate output results are determined). In the first stage 602, two first candidate texts can be retained, such as "A" and "C". In the second stage 604, the two first candidate texts of the first stage 602 can be used as input to predict the output of the second stage 604. The second stage 604 can also retain two second candidate texts, such as "B" and "E". Similarly, the input of the third stage 606 is the two second candidate texts of the second stage 604. Finally, the two candidate output results obtained using the beam search algorithm are ABD and CED. In this way, the candidate output results corresponding to the target prompt word can be made more natural and smooth.

[0059] As mentioned above, the essence of the language model is to predict the probability distribution of the next word, where the next word can refer to a word, a term, or a text in the predicted output result. Therefore, the "response certainty" corresponding to the screened target prompt word can be determined based on the information entropy corresponding to the probability distribution of the next word. That is, if the information entropy corresponding to the probability distribution of the next word is large, the response certainty of the target prompt word is low. On the other hand, if the response certainty corresponding to the target prompt word is low, the number of output results corresponding to the target prompt word is large, or the difference between the output results is large. Thus, the target prompt word with low response certainty can be extracted, and the labeling personnel determine the order of the output results corresponding to the target prompt word, and use the ordering result to train the language model. Among them, the information entropy corresponding to the probability distribution of the next word can be determined according to the text information in the predicted output result.

[0060] In this way, the target prompt words can be further screened according to the response certainty corresponding to the target prompt words, so that the target prompt words with unclear output results can be further labeled. In this way, the trained language model can output answers that meet the user's preferences, further improving the overall performance of the language model.

[0061] Figure 7 FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 702, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 704 or loaded from a storage unit 716 to a random access memory (RAM) 706. In the RAM 706, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 702, the ROM 704, and the RAM 706 are connected to each other via a bus 708. An input / output (I / O) interface 710 is also connected to the bus 708. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a coprocessor.

[0062] A number of components in the device 700 are connected to the I / O interface 710, including: an input unit 712, such as a keyboard, a mouse, etc.; an output unit 714, such as various types of displays, speakers, etc.; a storage unit 716, such as a disk, an optical disk, etc.; and a communication unit 718, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 718 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0063] The various methods or processes described above may be performed by the CPU / GPU 702. For example, in some embodiments, the methods may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 716. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 704 and / or the communication unit 718. When the computer program is loaded into the RAM 706 and executed by the CPU / GPU 702, one or more steps or actions in the methods or processes described above may be performed.

[0064] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0065] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0066] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0067] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0068] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0069] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0070] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0071] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0072] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for constructing training data, comprising: By clustering the prompt words in the training data set, multiple clusters are determined; determining a plurality of sampling probabilities corresponding to the plurality of clusters based on a plurality of cohesion degrees of the plurality of clusters, wherein the cohesion degrees indicate intra-cluster distances within a cluster; Determining a target cluster for sampling according to the multiple sampling probabilities; as well as Target training data is constructed by sampling target prompt words from the target cluster.

2. The method according to claim 1, further comprising: For each cluster, determining the distance between each prompt word in the cluster and the centroid of the cluster; as well as The intra-cluster distance of the cluster is determined according to the distance. 3 . The method according to claim 1 , wherein the sampling probability is negatively correlated with the cohesion degree and positively correlated with the intra-cluster distance.

4. The method according to claim 1, wherein sampling target prompt words from the target cluster comprises: Determine the sampling probability distribution according to the sampling probability corresponding to the target cluster; as well as Based on the sampling probability distribution, the target prompt word is sampled from the target cluster.

5. The method according to claim 4, wherein based on the sampling probability distribution, sampling the target prompt word from the target cluster comprises: Based on the sampling probability distribution, determining a predetermined sampling quantity corresponding to the target cluster; The predetermined sampling number of target prompt words are sampled from the target cluster.

6. The method according to claim 4, wherein based on the sampling probability distribution, sampling the target prompt word from the target cluster comprises: Based on the sampling probability distribution, the target prompt word is sampled from each target cluster by using a uniform sampling method.

7. The method according to claim 1, wherein clustering the prompt words in the training data set comprises: Determining the embedding corresponding to the prompt word in the training data set; as well as The prompt words are clustered based on the similarity of the embeddings.

8. The method according to claim 1, further comprising: receiving a response result determined by the user for the target prompt word; as well as The language model is trained based on the response result and the target prompt word.

9. The method according to claim 1, before clustering the prompt words in the training data set, further comprising: Determine a text hash value corresponding to a prompt word in the training data set; as well as Based on the text hash value, the training data set is deduplicated.

10. The method according to claim 1, further comprising: Inputting the target prompt word in the target training data into the language model, wherein the predicted output result corresponding to the target prompt word is output after being processed by the language model; The target training data is updated by screening the target prompt word according to the predicted output result.

11. The method according to claim 10, further comprising: Based on the language model, determining a plurality of candidate output results corresponding to the filtered target prompt word; receiving a user's ranking of the plurality of candidate output results; as well as In response to the ranking, the language model is trained using the updated target training data.

12. The method according to claim 11, wherein the candidate output result is a text sequence, the text sequence includes a first text and a second text, and determining the plurality of candidate output results corresponding to the filtered target prompt word comprises: Determine a predetermined number of candidate first texts using a language model; Determine the predetermined candidate number of second texts corresponding to each first text respectively, the second text being located after the first text; and Based on the plurality of corresponding first texts and second texts, a plurality of candidate output results are determined.

13. The method according to claim 10, wherein screening the target prompt word according to the predicted output result comprises: Determining information entropy corresponding to text information in the predicted output result; Determining the response certainty corresponding to the target prompt word according to the information entropy; as well as Based on the response certainty, the target cue word is screened.

14. An electronic device comprising: at least one processor; as well as coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the electronic device to perform actions, the actions comprising: By clustering the prompt words in the training data set, multiple clusters are determined; determining a plurality of sampling probabilities corresponding to the plurality of clusters based on a plurality of cohesion degrees of the plurality of clusters, wherein the cohesion degrees indicate intra-cluster distances within a cluster; Determining a target cluster for sampling according to the multiple sampling probabilities; and Target training data is constructed by sampling target prompt words from the target cluster.

15. The apparatus according to claim 14, further comprising: For each cluster, determining the distance between each prompt word in the cluster and the centroid of the cluster; as well as The intra-cluster distance of the cluster is determined according to the distance.

16. The apparatus according to claim 14, wherein the sampling probability is negatively correlated with the cohesion degree and positively correlated with the intra-cluster distance.

17. The apparatus of claim 14, wherein sampling a target cue word from the target cluster comprises: Determine the sampling probability distribution according to the sampling probability corresponding to the target cluster; as well as Based on the sampling probability distribution, the target prompt word is sampled from the target cluster.

18. The device according to claim 17, wherein based on the sampling probability distribution, sampling the target prompt word from the target cluster comprises: Based on the sampling probability distribution, determining a predetermined sampling quantity corresponding to the target cluster; The predetermined sampling number of target prompt words are sampled from the target cluster.

19. The device according to claim 17, wherein based on the sampling probability distribution, sampling the target prompt word from the target cluster comprises: Based on the sampling probability distribution, the target prompt word is sampled from the target cluster by using a uniform sampling method.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions that when executed cause a machine to perform actions comprising: By clustering the prompt words in the training data set, multiple clusters are determined; determining a plurality of sampling probabilities corresponding to the plurality of clusters based on a plurality of cohesion degrees of the plurality of clusters, wherein the cohesion degrees indicate intra-cluster distances within a cluster; Determining a target cluster for sampling according to the multiple sampling probabilities; as well as Target training data is constructed by sampling target prompt words from the target cluster.

Images