A training method and device for a generative large language model and its value system recognition application
Through the combination of pre-training-fine-tuning mode and TF-IDF algorithm, the training process of large language models is optimized, and the problem that the generated text of the model is not in line with human values is solved, and more efficient professionalism and comprehensiveness is achieved, and practical application needs are met. .
Patent Information
- Application Number
- CN202411230146.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-04
AI Technical Summary
Existing large language models are difficult to generate text that conforms to human values during training, especially when facing complex problems, which are low in performance and cannot meet practical application needs.
Using pre-training-fine-tuning mode, the similarity between corpus text and reference text is calculated through the TF-IDF algorithm, the text is divided into three sample sets, and the generative large language model is supervised and unsupervised, and cross-verification is passed. Optimize model parameters and select depth or breadth generation large language model for answering according to the questions entered by the user.
It improves the professionalism and comprehensiveness of the large language model, can more accurately meet the needs of different types of users, and improves the performance of the model in handling problems.
Smart Images

Figure CN118733748B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of large language models, and in particular to a training method and device for a generative large language model and a value system recognition application. Background Art
[0002] Large Language Model (LLM) is a deep learning model trained on massive text data. It can understand and generate natural language text, and is applied in the field of natural language processing, such as text generation, text summarization, language translation and other tasks; applied in the field of artificial intelligence, it can help machines better understand human language and achieve more natural human-computer interaction; it can also be applied in other fields, such as intelligent customer service, intelligent writing, and intelligent recommendation. For example, in intelligent customer service, large language models can automatically answer users' questions and improve customer service efficiency; in intelligent writing, large language models can assist writers in generating high-quality text content; in intelligent recommendation, large language models can recommend content that better meets user needs based on users' historical behavior and preferences. However, as the training corpus expands, it inevitably contains harmful, biased content, or some factual errors. These negative passages may cause the pre-trained large model to generate text that does not meet expectations or violates human values in the process of responding to user instruction requests. In order to ensure that the text generated by the large model is consistent with human expectations and values (such as usefulness, honesty, and harmlessness), additional alignment and fine-tuning training is required so that the large model can correctly obey various instruction requests from human users and only generate response text that meets expectations.
[0003] In order to train a large language model for reading comprehension tasks, the initial model is usually trained based on open source reading comprehension data. The answers to the questions in the open source reading comprehension data are usually directly copied from the original text of the article to be processed, and the questions and answers are relatively simple. The trained large language model cannot give a good answer to some difficult questions, especially those that require summarization from the article, resulting in low performance of the model and failure to meet actual application needs. Summary of the invention
[0004] In response to the technical problems existing in the prior art, the present invention provides a training method, device and value system recognition application of a generative large language model, which uses a "pre-training-fine-tuning" model to generate a natural language processing model with different language comprehension capabilities suitable for specific downstream tasks, and obtains a broad generative large language model and a deep generative large language model, which are used to answer questions input by users in different scenarios, taking into account both professionalism and breadth.
[0005] In a first aspect, the present invention provides a method for training a generative large language model, the method comprising:
[0006] Obtain N corpus texts for training the model, preprocess each of the corpus texts, and perform word segmentation on the preprocessed corpus texts using a word segmentation tool to obtain N first word sequences corresponding to each of the corpus texts; wherein N is a positive integer greater than 0.
[0007] A reference text pre-input by a user is obtained, the reference text is pre-processed, and a word segmentation tool is used to perform word segmentation on the pre-processed reference text to obtain a second word sequence corresponding to the reference text.
[0008] The word weight corresponding to each word in all the first word sequences and the second word sequences is calculated based on the TF-IDF algorithm, and the cosine similarity between each first word sequence and the second word sequence is calculated based on the word weight, and the cosine similarity is used as the text similarity between each of the corpus texts and the reference text.
[0009] The M corpus texts whose text similarity is greater than a preset threshold are put into a first sample set, the M corpus texts whose text similarity is less than or equal to the preset threshold are put into a second sample set, and Q corpus texts whose text similarity is greater than the preset threshold and P corpus texts whose text similarity is less than or equal to the preset threshold are randomly selected and put into a third sample set; M, Q, and P are all positive integers greater than 0 and less than N, and the ratio of Q to P is a preset ratio value.
[0010] The steps of training the generative large language model include:
[0011] The pre-trained generative large language model is supervisedly trained using the first sample set to obtain a first generative large language model.
[0012] The first generative large language model is trained unsupervisedly using the second sample set to obtain a second generative large language model.
[0013] The parameters of the second generative large language model are optimized by cross-validation to obtain a third generative large language model.
[0014] When the preset ratio value is within a first preset ratio range, supervised fine-tuning training is performed on the third generative large language model using the third sample set to obtain a deep generative large language model.
[0015] When the preset ratio value is within a second preset ratio range, the third generative large language model is subjected to unsupervised fine-tuning training using the third sample set to obtain a wide generative large language model.
[0016] Receive a question input by a user, extract keywords from it, calculate the association index between the keywords and a preset keyword dictionary, and when the association index is greater than a preset threshold, select a deep generative large language model to answer the question input by the user; when the association index is less than or equal to the preset threshold, select a broad generative large language model to answer the question input by the user.
[0017] In a possible implementation of the present invention, the method provided by the present invention further includes:
[0018] The calculating the word weight corresponding to each word in the first word sequence and the second word sequence based on the TF-IDF algorithm specifically includes:
[0019] Define the words in each word sequence The word frequency TF is:
[0020]
[0021] in, Yes word In word sequence The number of times it appears in is a word sequence The sum of the number of occurrences of all words in .
[0022] Define the words in each word sequence The inverse document frequency IDF is:
[0023] Among them, |D| is the total number of word sequences, To contain words The number of word sequences.
[0024] Define the words in each word sequence The word weight is:
[0025]
[0026] In a possible implementation of the present invention, the method provided by the present invention further includes:
[0027] The step of calculating the cosine similarity between each first word sequence and the second word sequence based on the word weights specifically includes:
[0028] The cosine similarity between the first word sequence m and the second word sequence n is calculated according to the following formula:
[0029]
[0030] In a possible implementation of the present invention, the method provided by the embodiment of the present invention further includes:
[0031] The calculating of the correlation index between the keyword and the preset keyword dictionary specifically includes:
[0032] The calculation is performed using the following formula:
[0033] .
[0034] The vector X is a vector representation of a keyword set of a question input by a user, and the vector Y is a vector representation of a keyword set in the preset keyword dictionary.
[0035] In a second aspect, the present invention provides a value system identification method based on a generative large language model, wherein the generative large language model is obtained based on the above-mentioned training method of a generative large language model, and the N corpus texts used to train the model are specifically text materials related to the values of natural persons, including one or more of the following:
[0036] Psychological materials related to value understanding, theoretical content of basic value system, questionnaires related to value system, and interview materials related to value system.
[0037] In a third aspect, the present invention provides a training device for a generative large language model, characterized in that the device includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the above-mentioned training method for a generative large language model is implemented.
[0038] In the above technical solution, the technical effects and advantages provided by the present invention are:
[0039] The large language model training method provided in the embodiment of the present application first combines supervised training and unsupervised training to ensure that the large language model is basically professional and comprehensive, and then optimizes the large language model through cross-validation to further improve the performance of the large language model. Finally, by flexibly setting the sample set for training the large language model, the large language model is further trained from the two dimensions of depth and breadth. Different large language models can be selected according to the user input questions, meeting the needs of various types of users and improving the performance of the large language model in processing problems. The large language model training method provided in the embodiment of the present application provides a richer data foundation with better training effect for training a large language model with better performance, and better meets the needs of practical applications.
[0040] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0042] Figure 1 It is an exemplary system architecture of an embodiment of the present application.
[0043] Figure 2 The present invention is a flowchart of a method for training a generative large language model according to an exemplary embodiment. DETAILED DESCRIPTION
[0044] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.
[0045] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.
[0046] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0047] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions. Embodiment 1
[0048] Figure 1 An exemplary system architecture 100 of an embodiment of a generative large language model training method, a generative large language model-based value system identification method, and a supporting device of the present application is shown.
[0049] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0050] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 can be installed with various applications for realizing information communication between the two, such as value system identification applications, model training applications, etc.
[0051] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.
[0052] The server 105 can provide various services through various built-in applications. Since it takes more computing resources and stronger computing power to train the target generative large model, the generative large model training method provided in the subsequent embodiments of this application is generally performed by the server 105 with stronger computing power and more computing resources. Correspondingly, the generative large model training device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the various operations handed over to the server 105 through the model training applications installed thereon, and then output the same results as the server 105. Correspondingly, the generative large model training device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.
[0053] Of course, the server used to train the generative big model can be different from the server that calls the trained target generative big model for use. In particular, the generative big model trained by the server 105 can also be used to obtain a lightweight generative big model suitable for being placed in the terminal devices 101, 102, and 103 through model distillation, that is, it can be flexibly selected to use the lightweight generative big model in the terminal devices 101, 102, and 103 according to the recognition accuracy actually required, or to choose to use the more complex generative big model in the server 105.
[0054] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0055] Please refer to Figure 2 , Figure 2 The present invention is a flowchart of a method for training a generative large language model according to an exemplary embodiment, comprising the following steps:
[0056] Step 201: Obtain N corpus texts for training the model, preprocess each corpus text, and perform word segmentation on the preprocessed corpus text using a word segmentation tool to obtain N first word sequences corresponding to each corpus text; N is a positive integer greater than 0.
[0057] The corpus text used to train the model is usually text data related to a specific application scenario, which may include public domain materials, such as government websites, reports published by public research institutions, open access academic articles, etc.; open license materials, such as websites, forums and publications that use open license agreements such as Creative Commons; public data sets, such as Wikipedia, Baidu Encyclopedia, social media public posts, news articles, user comments, etc.; proprietary data sets, such as text data that need to be negotiated with the data owner for use.
[0058] Each corpus text is preprocessed to remove meaningless modal particles, stop words, etc. in the text data, and then the preprocessed corpus text is segmented using a word segmentation tool to segment the continuous text into individual words to form the first word sequence corresponding to each corpus text. In this field, commonly used word segmentation tools include ICTCLAS of the Chinese Academy of Sciences, LTP of Harbin Institute of Technology, and Chinese word segmentation tools such as Jieba Word Segmentation, which can all achieve word segmentation of preprocessed corpus texts, and users can select according to actual needs.
[0059] Step 202: Acquire a reference text pre-input by a user, pre-process the reference text, and perform word segmentation processing on the pre-processed reference text using a word segmentation tool to obtain a second word sequence corresponding to the reference text.
[0060] The reference text pre-entered by the user can be text data in a specific application scenario, such as the medical field, e-commerce field, financial field, etc., or text data related to a certain topic, such as travel, sports, music, movies, etc., or text data that the user is interested in, such as the theoretical content of the basic value system, etc. According to the above method of pre-processing and word segmentation of the corpus text, the reference text entered by the user is processed in the same way to obtain a second word sequence corresponding to the reference text.
[0061] Step 203: Calculate the word weight corresponding to each word in all the first word sequences and the second word sequences based on the TF-IDF algorithm, and calculate the cosine similarity between each first word sequence and the second word sequence based on the word weight, and use the cosine similarity as the text similarity between each corpus text and the reference text.
[0062] Term frequency-inverse document frequency (TF-IDF) is a statistical method used to assess the importance of a term to a document in a collection of documents or a corpus. The importance of a term increases with the number of times it appears in a document, but decreases inversely with the frequency of its occurrence in the corpus. Various forms of TF-IDF weighting are often used by search engines as a measure or rating of the relevance of a document to a user query.
[0063] Term frequency (TF) refers to how often a given word appears in a document. This number is normalized to prevent it from being biased towards long documents (the same word may have a higher word count in a long document than in a short document, regardless of whether the word is important or not).
[0064] For words in a word sequence , its word frequency TF can be expressed as:
[0065]
[0066] Is the word in the word sequence The number of times it appears in In the word sequence The sum of the number of occurrences of all words in .
[0067] Inverse document frequency (IDF) is a measure of the general importance of a word. The IDF of a particular word can be calculated by dividing the total number of documents by the number of documents containing the word and taking the logarithm of the quotient.
[0068] Therefore, we can define the words in each word sequence The inverse document frequency IDF is:
[0069]
[0070] Among them, |D| is the total number of word sequences, To contain words The number of word sequences;
[0071] It can be obtained that each word sequence Each word in The corresponding word weights are:
[0072] .
[0073] The calculation formula of cosine similarity is as follows:
[0074] .
[0075] In this way, the cosine similarity between any first word sequence and the second word sequence can be obtained, and the text similarity between any corresponding corpus text and the reference text can be obtained. In this way, corpus texts with high similarity to the reference text input by the user can be mined from massive corpus texts.
[0076] Step 204: M corpus texts with text similarity greater than a preset threshold are put into the first sample set, M corpus texts with text similarity less than or equal to the preset threshold are put into the second sample set, and Q corpus texts with text similarity greater than the preset threshold and P corpus texts with text similarity less than or equal to the preset threshold are randomly selected and put into the third sample set; M, Q, and P are all positive integers greater than 0 and less than N, and the ratio of Q to P is a preset ratio value.
[0077] After calculating the text similarity between all corpus texts and the reference text, corpus texts with higher similarity can be extracted from the corpus texts as the first sample set, and the corpus texts in this set are all text data that are highly relevant to the reference text. Corpus texts with lower similarity can be extracted from the corpus texts as the second sample set, and the corpus texts in this set are all text data that are basically irrelevant to the reference text. In addition, corpus texts with higher similarity and corpus texts with lower similarity can be extracted according to a preset ratio value and put into a third sample set at the same time. The preset ratio value can be adjusted according to the actual needs of the user. For example, when the value of Q is much larger than the value of P, the third sample set contains more text data that is highly relevant to the reference text and less text data that is basically irrelevant to the reference text. When the value of Q is much smaller than the value of P, the third sample set contains less text data that is highly relevant to the reference text and more text data that is basically irrelevant to the reference text, thereby improving the flexibility of model training.
[0078] Step 205: Use the first sample set to perform supervised training on the pre-trained generative large language model to obtain a first generative large language model.
[0079] The first generative large language model is trained unsupervisedly using the second sample set to obtain a second generative large language model.
[0080] The parameters of the second generative large language model are optimized by cross-validation to obtain the third generative large language model.
[0081] When the preset ratio value is within the first preset ratio range, supervised fine-tuning training is performed on the third generative large language model using the third sample set to obtain a deep generative large language model.
[0082] When the preset ratio value is within the second preset ratio range, the third generative large language model is unsupervisedly fine-tuned using the third sample set to obtain a wide generative large language model.
[0083] From the above steps, it can be seen that the corpus text in the first sample set has a high degree of similarity with the reference text, so it can be used as sample data for deep training of the generative large language model to improve the professionalism of the large language model, while the corpus text in the second sample set has a low degree of similarity with the reference text, so it can be used as sample data for breadth training of the generative large language model to improve the comprehensiveness of the large language model.
[0084] First, supervised training is performed on the pre-trained generative large language model using the first sample set to obtain a first generative large language model.
[0085] A pre-trained generative large language model refers to a general generative large language model obtained through training with basic training samples. It usually only generates a text sequence as a result based on the predicted probability of the language unit according to the knowledge contained in the text input and the large model parameters. The result is usually inaccurate. The purpose of supervised training is to enable the model to learn task-specific knowledge so that it can provide more accurate results in practical applications. For generative models, supervised training can also help the model generate text that better meets the requirements of specific fields or tasks. Since the first sample set has a high degree of similarity with the reference text, using the first sample set to perform supervised training on the pre-trained generative large language model can enable the generative large language model to learn the professional knowledge that the user is interested in, thereby improving the professionalism of the large language model.
[0086] Then, the first generative large language model is trained unsupervisedly using the second sample set to obtain a second generative large language model.
[0087] The purpose of unsupervised training is to enable the model to recognize patterns in input data without relying on external labels. During the unsupervised training phase, the model will learn from a large amount of unlabeled text data to generate coherent and natural text, and be able to understand and generate multiple languages and styles. Since the second sample set has a low similarity to the reference text, using the second sample set to perform unsupervised training on the first generative large language model can enable the generative large language model to learn a wider range of knowledge, thereby improving the comprehensiveness of the large language model.
[0088] Furthermore, cross-validation is used to optimize parameters of the second generative large language model to obtain a third generative large language model.
[0089] Cross-validation is a technique for evaluating the performance of machine learning models. It estimates the generalization ability of the model by dividing the dataset into several mutually exclusive subsets and using different subsets for training and testing. Since large language models are usually trained on very large datasets, standard cross-validation methods may not be applicable or inefficient. Therefore, the following strategies can be used to optimize the performance of large language models:
[0090] 1. Stratified Cross-Validation: If the dataset has an unbalanced class distribution, stratified cross-validation can ensure that there are representative samples from each class in each fold. This is particularly important for classification tasks.
[0091] 2. Time Series Cross-Validation: For time-dependent data, such as stock prices or social media data, the dataset needs to be split in chronological order to avoid future information leaking into the past.
[0092] 3. Continual Learning: Large language models can adopt a continuous learning strategy, that is, fine-tuning the model on the continuous influx of new data to keep it adaptable to new information.
[0093] 4. Transfer Learning: Perform transfer learning on a large language model, that is, use the model weights trained on a specific task as a starting point, and then fine-tune them on a new task.
[0094] 5. Model Ensembling: Combining the predictions of multiple models can improve the overall performance of the model. This can be achieved by training multiple models and averaging their outputs when predicting.
[0095] 6. Hyperparameter Optimization: Use methods such as grid search, random search, or Bayesian optimization to find the best hyperparameter settings.
[0096] 7. Early Stopping: During the training process, stop training when the performance of the validation set no longer improves to prevent overfitting.
[0097] In practical applications, multiple strategies can be combined according to actual needs to optimize the performance of large language models.
[0098] Supervised fine-tuning originates from the fine-tuning idea in deep learning. First, a large language model is pre-trained to obtain better parameter results on some general data sets. Then, the model is fine-tuned on a labeled data set for a specific task using cross-entropy loss, so that the parameters of the model are adjusted based on the initial parameters to minimize the loss function of the supervised task. Unsupervised fine-tuning is to further train the pre-trained model without labels in the training set, in an attempt to let the model learn the structure and pattern of the data by itself. In unsupervised fine-tuning, the model does not learn from manually annotated data, but through self-supervised tasks such as masked language modeling and prediction of the next word and sentence, to further understand and process the data and language patterns related to the specific task. This method can improve the performance of the model on a specific task without relying on a large amount of labeled data, especially for situations where labeled data is scarce or costly.
[0099] After obtaining the third generative large language model, the generative large language model can be further trained in depth or breadth according to the value range of the ratio of Q and P in the third sample set. The user presets the ratio range. When the preset ratio value is in the first preset ratio range, it indicates that the third sample set contains more text data that is highly relevant to the reference text and less text data that is basically irrelevant to the reference text. Therefore, the third generative large language model can be fine-tuned with supervision to further improve the professionalism of the generative large language model, thereby obtaining a deep generative large language model. When the preset ratio value is in the second preset ratio range, it indicates that the third sample set contains less text data that is highly relevant to the reference text and more text data that is basically irrelevant to the reference text. Therefore, the third generative large language model can be fine-tuned with supervision to further improve the comprehensiveness of the generative large language model, thereby obtaining a broad generative large language model. Exemplarily, the first preset ratio range can be 0.7-0.99, and the second preset ratio range can be 0.01-0.4. By setting the first preset ratio range and the second preset ratio range, the training method of the generative large language model can be flexibly selected to obtain a deep generative large language model that focuses on professionalism and a broad generative large language model that focuses on comprehensiveness.
[0100] Step 206: Receive a question input by a user, extract keywords from it, calculate a correlation index between the keywords and a preset keyword dictionary, and when the correlation index is greater than a preset threshold, select a deep generative large language model to answer the question input by the user; when the correlation index is less than or equal to the preset threshold, select a broad generative large language model to answer the question input by the user.
[0101] The preset keyword dictionary can be obtained based on the second word sequence. In this case, the keywords in the preset keyword dictionary are all professional knowledge that the user is interested in. It can also be a keyword dictionary related to a specific field, or a keyword dictionary pre-set by the user. This application does not limit the method of obtaining the preset keyword dictionary.
[0102] The correlation index between the keywords and the preset keyword dictionary can be calculated in the following way:
[0103] .
[0104] Among them, vector X is the vector representation of the keyword set of the user input question, and vector Y is the vector representation of the keyword set in the preset keyword dictionary.
[0105] When the correlation index between the keywords in the question input by the user and the preset keyword dictionary is high, it indicates that the user needs a more professional answer. For example, the question asked by the user is related to the medical field or the financial field. Therefore, a deep generative large language model can be used to answer the question input by the user, thereby ensuring the accuracy and professionalism of the answer; when the correlation index between the keywords in the question input by the user and the preset keyword dictionary is low, it indicates that the user does not need a more professional answer. For example, the question asked by the user is related to a certain movie or a travel-related question. Therefore, a breadth generative large language model can be used to answer the question input by the user, thereby providing the user with a more comprehensive answer with relatively broad knowledge.
[0106] This application combines supervised training and unsupervised training to train the generative model from two dimensions: depth and breadth. Based on the questions input by the user, the keywords in the questions are matched with a preset keyword dictionary to determine whether the questions involve specific application scenarios, so as to choose to use a deep generative large language model or a breadth generative large language model to answer the user's questions, thereby improving the training efficiency and accuracy of the generative large language model. Embodiment 2
[0107] A large language model is an ultra-large-scale language model trained with a wide range of text data. It has a text understanding capability far beyond that of traditional methods and a vast amount of knowledge reserves, providing basic support for text-based value understanding and recognition tasks. Large language models usually do not have their own values, but they need to understand and respect the values of users as much as possible and align with their values. Therefore, they need to follow the following principles:
[0108] 1. Respect users: Respect the rights and dignity of users and strive to provide unbiased information.
[0109] 2. Accuracy: Provide accurate, reliable and up-to-date information to help users make informed decisions.
[0110] 3. Objectivity: Be objective and, when possible, provide multiple viewpoints to help users gain a comprehensive understanding of the issue.
[0111] 4. Comply with legal and ethical standards: Comply with applicable legal and ethical standards and always follow these guidelines when interacting with users.
[0112] However, there is still a lack of methods to efficiently train value understanding and recognition tasks based on the characteristics of large language models through technologies such as cue word engineering, few-shot learning, and zero-shot learning.
[0113] To address this problem, an embodiment of the present invention further provides a value system identification method based on a generative large language model, wherein the generative large language model is obtained based on the training method of the above-mentioned generative large language model.
[0114] The N corpus texts used to train the model are specifically text materials related to natural person values, which can be psychological materials related to value understanding, theoretical content of basic value systems, questionnaires related to value systems, and interview materials related to value systems. Sample value understanding samples and value dimension query tables corresponding to "context corpus-value dimensions" are obtained from the World Value Survey, Rokeach Values Survey, The Valued Living Questionnaire, Portrait Value Questionnaire, and Personal Values Assessment, etc. This application does not limit this, so that the corpus texts related to the value system can be used to realize the training of the generative large language model, and a generative large language model that can identify the value system can be obtained.
[0115] By training the generative large language model with a data set containing specific values, the model can understand and internalize these values and follow these values when answering questions. In addition, in order to make the output generated by the model conform to the established values, reinforcement learning technology can be used to give rewards or penalties based on whether the output generated by the model conforms to the established values, thereby guiding the generative large language model to generate answers that are more in line with these values. When the model is deployed, filtering and review mechanisms can also be added to ensure that the output content conforms to specific values and standards. In addition, a user feedback mechanism can be introduced to allow users to provide feedback on the output of the model and adjust the model's behavior based on this feedback to better conform to the user's values.
[0116] When applying the human-computer interaction model to different human-computer interaction scenarios, relevant technical personnel configure a given rule set suitable for the specific human-computer interaction scenario, and input the given rule set and user instructions into the human-computer interaction model during the human-computer interaction. The human-computer interaction model can respond on the premise of satisfying the given rule set, so that the behavior of the human-computer interaction model is consistent with the given rule set, thereby improving the interaction quality of the human-computer interaction model. In addition, the model does not require additional training when switching the given rule set. Therefore, there is no need to separately annotate the dialogue data for different human-computer interaction scenarios and perform alignment training, which can greatly reduce the alignment cost of the human-computer interaction model. Embodiment 3
[0117] An embodiment of the present invention also provides a training device for a generative large language model, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the above-mentioned training method for the generative large language model when executed by the processor.
[0118] The present application also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.
[0119] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method of any of the aforementioned embodiments is implemented. The specific functions and technical effects that can be achieved are not repeated here.
[0120] The embodiment of the present application provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the electronic device in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.
[0121] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "including a ..." do not exclude the presence of other identical factors in the process, method, article or device including the elements.
[0122] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.
[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for training a generative large language model, characterized in that: include: Obtain N corpus texts for training the model, preprocess each of the corpus texts, and perform word segmentation processing on the preprocessed corpus texts using a word segmentation tool to obtain N first word sequences corresponding to each of the corpus texts; N is a positive integer greater than 0; Acquire a reference text pre-entered by a user, pre-process the reference text, and perform word segmentation processing on the pre-processed reference text using a word segmentation tool to obtain a second word sequence corresponding to the reference text; Calculate the word weight corresponding to each word in all the first word sequences and the second word sequences based on the word frequency-inverse document frequency TF-IDF algorithm, and calculate the cosine similarity between each first word sequence and the second word sequence based on the word weight, and use the cosine similarity as the text similarity between each of the corpus texts and the reference text; M of the corpus texts whose text similarity is greater than a preset threshold are put into a first sample set, M of the corpus texts whose text similarity is less than or equal to the preset threshold are put into a second sample set, and Q of the corpus texts whose text similarity is greater than the preset threshold and P of the corpus texts whose text similarity is less than or equal to the preset threshold are randomly selected and put into a third sample set; M, Q, and P are all positive integers greater than 0 and less than N, and the ratio of Q to P is a preset ratio value; The steps of training the generative large language model include: Using the first sample set to perform supervised training on the pre-trained generative large language model to obtain a first generative large language model; Using the second sample set to perform unsupervised training on the first generative large language model to obtain a second generative large language model; Optimizing the parameters of the second generative large language model by cross-validation to obtain a third generative large language model; When the preset ratio value is within a first preset ratio range, using the third sample set to perform supervised fine-tuning training on the third generative large language model to obtain a deep generative large language model; When the preset ratio value is within a second preset ratio range, performing unsupervised fine-tuning training on the third generative large language model using the third sample set to obtain a wide generative large language model; Receive a question input by a user, extract keywords from it, calculate the association index between the keywords and a preset keyword dictionary, and when the association index is greater than a preset threshold, select a deep generative large language model to answer the question input by the user; when the association index is less than or equal to the preset threshold, select a broad generative large language model to answer the question input by the user.
2. The method according to claim 1, characterized in that The calculating the word weight corresponding to each word in the first word sequence and the second word sequence based on the TF-IDF algorithm specifically includes: Define the words in each word sequence The word frequency TF is: ; in, Yes word In word sequence The number of times it appears in is a word sequence The sum of the number of occurrences of all words in ; Define the words in each word sequence The inverse document frequency IDF is: ; Among them, |D| is the total number of word sequences, To contain words The number of word sequences; Define the words in each word sequence The word weight is: 。 3. The method according to claim 2, characterized in that The step of calculating the cosine similarity between each first word sequence and the second word sequence based on the word weights specifically includes: The cosine similarity between the first word sequence m and the second word sequence n is calculated according to the following formula: 。 4. The method according to claim 3, characterized in that: The calculating of the correlation index between the keyword and the preset keyword dictionary specifically includes: The calculation is performed using the following formula: ; Among them, vector X is the vector representation of the keyword set of the user input question, and vector Y is the vector representation of the keyword set in the preset keyword dictionary.
5. A value system identification method based on a generative large language model, wherein the generative large language model is obtained based on a generative large language model training method according to any one of claims 1 to 4, characterized in that ; The N corpus texts used for training the model are specifically text materials related to the value of natural persons, including one or more of the following: Psychological materials related to value understanding, theoretical content of basic value system, questionnaires related to value system, and interview materials related to value system.
6. A training device for a generative large language model, characterized in that: The device includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, a training method for a generative large language model as described in any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
A method of enabling machine to compete more complex work
CN110163233A
User feedback data analysis method and device and computer equipment
CN115114916A
Enhanced retrieval user question and answer method and device, computer equipment and storage medium
CN118132732A
Artificial intelligence (AI)-based regulatory data processing system
US20200111023A1
Method and system for analyzing and establishing trust in systems that include artificical intelligence systems
US20240202405A1
