A language model corpus training method and system based on a classification task

Abstract

The application provides a language model corpus training method and system based on a classification task, which comprises the following steps: collecting sample corpora and performing data cleaning operations; selecting part of the sample corpora after data cleaning to perform manual labeling, and obtaining first labeled corpora; converting the manually labeled question semantic labels in the first labeled corpora into code expressions, and constructing a question semantic label-code association mapping table; converting the generation task of the language model into a classification task through the code expressions, fine-tuning the input first labeled corpora and the code expressions by using a voice model, determining a logprobs threshold value according to the return result of the fine-tuning, determining second labeled corpora through the logprobs threshold value, and saving the second labeled corpora in a corpus set to complete language model training; and training the trained language model based on the corpus set and the logprobs threshold value sample corpora, and obtaining target labeled corpora according to the association mapping table. Through the task conversion of the voice model, the cost of producing labeled corpora is reduced, and the production efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a language model corpus training method and system based on classification task. BACKGROUND

[0002] Intelligent question and answer customer service is an artificial intelligence system simulating human and aiming to form coherent and smooth dialogue / interaction with human, for example, can accurately judge the demand of the user according to the question raised by the user, which needs to be realized through intent recognition in neural network model. Usually, a considerable amount of labeled corpus needs to be invested in the process of model training, and the quality of the corpus can be guaranteed only by manual annotation. In the scenarios of e-commerce, industry, finance, bank and Internet hospital, there is often a dilemma of lack of labeled corpus, high cost and low efficiency of corpus annotation.

[0003] Therefore, there is an urgent need for a language model corpus training method based on classification task to solve the above problems of lack of labeled corpus, high cost and low efficiency of corpus annotation. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a language model corpus training method and system based on classification task.

[0005] The first aspect of the present application discloses a language model corpus training method based on classification task, comprising:

[0006] S1: collecting sample corpus and performing data cleaning operation;

[0007] S2: selecting part of the sample corpus after data cleaning for manual annotation to obtain first labeled corpus, wherein the first labeled corpus includes training set, validation set and test set.

[0008] S3: converting the manually annotated question semantic label in the first labeled corpus into code expression, and constructing a question semantic label-code association mapping table, wherein the question semantic label in the association mapping table corresponds to the code expression one by one.

[0009] S4: converting the generation task of the language model into classification task through the code expression, fine-tuning the input first labeled corpus and code expression by using the voice model, determining the logprobs threshold value according to the return result of fine-tuning, determining the second labeled corpus through the logprobs threshold value and saving it in the corpus set to complete the language model training;

[0010] S5: training the trained language model based on the corpus set and the logprobs threshold value, and translating the target labeled corpus according to the association mapping table of the prompt semantic label-code.

[0011] In an optional embodiment, the collecting sample corpus and performing data cleaning operation comprises:

[0012] S11: collecting sample corpus in business data according to business requirements and / or collecting business-related sample corpus through web crawler;

[0013] S12: scanning and removing invalid data in the sample corpus, the invalid data including duplicate data, website links, empty data, emoticons and language model training irrelevant data.

[0014] In an optional embodiment, the selecting part of the sample corpus after data cleaning for manual annotation to obtain the first labeled corpus:

[0015] S21: taking no less than 10% of the sample corpus data for manual annotation after manually analyzing the prompt semantics of the sample corpus, to obtain the first labeled corpus;

[0016] S22: taking 70%, 20% and 10% of the labeled data in the first labeled corpus as training set, validation set and test set, respectively.

[0017] In an optional embodiment, the converting the prompt semantic label manually annotated in the first labeled corpus into code expression comprises:

[0018] S31: converting the first labeled corpus into question text and prompt semantic expression based on prompt template, the prompt semantic expression being corresponding token number of code expression, and the first labeled corpus being saved in Json format after conversion.

[0019] In an optional embodiment, the determining logprobs threshold value according to the return result of fine-tuning training, determining second labeled corpus through logprobs threshold value and saving in corpus set to complete the language model training comprises:

[0020] S41: the language model input in the fine-tuning training is limited to corresponding token number and five return results, and the language model outputs five return values and logarithmic likelihood after iteration;

[0021] S42: determining the logprobs threshold value used by the speech model to constrain the return result according to the return value and logarithmic likelihood;

[0022] S43: Evaluate the return result of the language model input according to the logprobs threshold. If the return result is greater than the logprobs threshold, mark the first labeled corpus as correct and save it as the second labeled corpus in the corpus set. If the return result is less than the logprobs threshold, mark the first labeled corpus as incorrect and discard it. Continue iterating until the training of the language model is completed.

[0023] In an optional embodiment, training the sample corpus with the trained language model based on the corpus set and the logprobs threshold, and translating to obtain the target labeled corpus according to the association mapping table of the question semantic label-code includes:

[0024] S51: Call the Completion interface of the large model to train the language model, and input the sample corpus after setting the parameters to perform iteration;

[0025] S52: Compare the returned results of the sample corpus with the logprobs threshold. If the returned result is greater than the logprobs threshold, it is marked as correct. Then, the token number is translated into the question semantics through the association mapping table of question semantic label-code to generate the target labeled corpus. If the returned result is less than the logprobs threshold, it is marked as wrong and discarded.

[0026] In an optional embodiment, the preset limiting parameters include limiting the code to correspond to a single token number and limiting the return results to five.

[0027] A second aspect of this invention discloses a language model corpus training system based on a classification task, the system comprising:

[0028] The data acquisition module is used to collect sample corpora and perform data cleaning operations.

[0029] The preprocessing module is used to manually annotate a portion of the cleaned sample corpus to obtain the first annotated corpus, which includes a training set, a validation set, and a test set.

[0030] The expression conversion module is used to convert the manually annotated question semantic labels in the first annotated corpus into code expressions, and to construct an association mapping table of question semantic labels and codes, wherein the question semantic labels and code expressions in the association mapping table correspond one-to-one;

[0031] The model training module is configured to convert a generation task of a language model into a classification task through the code expression, fine-tune the first annotated corpus and the code expression, determine a logprobs threshold according to a return result of the fine-tuning, determine a second annotated corpus through the logprobs threshold, and save the second annotated corpus in a corpus set to complete the language model training.

[0032] The corpus generation module is configured to train the sample corpus based on the corpus set and the logprobs threshold, and translate the target annotated corpus according to the association mapping table of the question semantic label-code.

[0033] The third aspect of the present application discloses a language model corpus training device based on a classification task, comprising:

[0034] at least one processor, and

[0035] a memory in communication connection with the at least one processor; wherein

[0036] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the language model corpus training methods based on the classification task disclosed in the first aspect of the present application.

[0037] The fourth aspect of the present application discloses a computer readable storage medium, which stores computer executable instructions for enabling a computer to execute the language model corpus training method based on the classification task disclosed in any one of the first aspect of the present application.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] (1) The present application trains a language model through a small amount of manually annotated sample corpus, converts a production corpus task of the language model into a classification corpus task through code expression, realizes production of a corpus through the language model, and can meet the demand of feeding corpus in daily model training.

[0040] (2) The present application can effectively produce annotated corpus with correct annotations by judging the annotation of sample corpus through a trained language model, improve the production efficiency of corpus, and realize the production of target corpus after training the model with a small amount of manual annotation, thereby greatly reducing the time cost and production cost of the manual annotation part of the corpus. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor under the premise of the drawings.

[0042] Figure 1 Flow chart of the language model corpus training method based on a classification task of the present application;

[0043] Figure 2 Schematic diagram of the language model corpus training system based on a classification task of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0045] In the description of the present application, it should be noted that the orientations or positional relationships indicated by the terms “center”, “upper”, “lower”, “left”, “right”, “vertical”, “horizontal”, “inner”, “outer” and the like are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms “first”, “second”, “third”, “fourth” and the like are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0046] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms “mounting”, “connecting”, “connection” should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through an intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0047] Embodiment 1

[0048] Referring to Figure 1 The present application embodiment discloses a language model corpus training method based on a classification task, comprising:

[0049] S1: collect sample corpus and perform data cleaning operation;

[0050] In an optional embodiment, the collecting sample corpus and performing data cleaning operation comprises:

[0051] S11: collect sample corpus in business data according to business needs and / or collect business-related sample corpus through web crawler;

[0052] S12: scan and remove invalid data in the sample corpus, including duplicate data, website links, empty data, emoticons and language model training irrelevant data.

[0053] S2: select part of the sample corpus after data cleaning for manual annotation to obtain a first annotated corpus, the first annotated corpus including a training set, a validation set and a test set.

[0054] In an optional embodiment, the selecting part of the sample corpus after data cleaning for manual annotation to obtain a first annotated corpus:

[0055] S21: take no less than 10% of the sample corpus data for manual annotation after manually analyzing the semantics of the sample corpus, to obtain a first annotated corpus;

[0056] S22: take 70%, 20% and 10% of the annotated data in the first annotated corpus as the training set, the validation set and the test set, respectively.

[0057] It should be noted that sample corpus is the most basic and important link, data sources can be collected according to the business content of the corpus online business data or through network crawler to crawl related data, but the sample corpus contains a large amount of duplicate data, website links, empty data and invalid data such as emoticons, which need to be excluded from the training. After cleaning invalid data, the corpus needs to be manually annotated as annotated corpus that can be used for training model. The manual annotation process mainly annotates the problem represented in each corpus meaning, for example: the question is "Can I refund by adding WeChat?", and the matching annotated content is the uniform standard question expression "Buyer inquiry activity can refund".

[0058] S3: convert the question semantics label manually annotated in the first annotated corpus into code expression, and construct a question semantics label-code association mapping table, wherein the question semantics label and the code expression in the association mapping table correspond one-to-one.

[0059] It should be noted that the association mapping table of the question method semantic label-code is the process of representing the question method semantics by digital tokens. If this step is not processed, the generated language model will return multiple results when responding to the corpus, which will prevent us from selecting a standard answer as our labeled corpus. At the same time, the language model's response is random, such as the corpus "hello", which may be "buyer greeting" or "buyer greeting to you". If the label-code conversion is performed, the language model will only return a corresponding number, i.e. the converted code expression, which can significantly improve the robustness of language model training. The conversion process of the association mapping table is shown in the following table:

[0060] Corpus Label Code Hello Buyer greeting 1 No more complaints or I will complain Buyer expressing dissatisfaction / anger 2

[0061] In an optional embodiment, the conversion of the question method semantic label in the first labeled corpus into a code expression includes:

[0062] S31: converting the first labeled corpus into a question text and a question method semantic expression based on a prompt template, the question method semantic expression being a corresponding token number of the code expression, and the first labeled corpus being saved in a Json format after conversion.

[0063] S4: converting the language model generation task into a classification task through the code expression, fine-tuning the voice model on the input first labeled corpus and the code expression, determining a logprobs threshold value according to the return result of the fine-tuning, determining a second labeled corpus through the logprobs threshold value and saving it in a corpus set to complete the language model training;

[0064] In an optional embodiment, the conversion of the question method semantic label in the first labeled corpus into a code expression includes:

[0065] S41: the fine-tuning of the language model input limiting the code expression to a corresponding token number and limiting five return results, and the language model outputting five return values and logarithmic likelihoods after iteration;

[0066] S42: determining a logprobs threshold value for the voice model to constrain the return result according to the return value and the logarithmic likelihood;

[0067] S43: evaluate the return result of the language model input according to the logprobs threshold value, if the return result is greater than the logprobs threshold value, mark the input first annotated corpus as correct and save it in the corpus set as the second annotated corpus, if the return result is less than the logprobs threshold value, mark the input first annotated corpus as incorrect and discard it, continue iteration until the training of the language model is completed.

[0068] It should be noted that since the question is mapped to a token number, when calling the language model for fine-tuning, the input parameter is max_tokens=1 and logprobs=5, and there will be 5 relevant return values and logarithmic possibilities in the logprobs of the return result, where the greater the return result value, the more likely the question is correct. A large number of tests can be used to evaluate the threshold value. Greater than this threshold value means that it is likely to be correct. After multiple data tests and test result judgments, the model iteratively induces the logprobs threshold value of the correct result. For subsequent predictions, the result greater than the threshold value is marked as correct and will be saved in the corpus set, and the result less than the threshold value is discarded.

[0069] S5: train the sample corpus based on the corpus set and the logprobs threshold value of the trained language model, and translate the target annotated corpus according to the association mapping table of the question semantic label-code.

[0070] In an optional embodiment, the training of the sample corpus based on the corpus set and the logprobs threshold value of the trained language model includes:

[0071] S51: call the Completion interface of the large model of the trained language model, input the sample corpus after presetting the limited parameters for iteration;

[0072] S52: compare the return result of the sample corpus with the logprobs threshold value, if the return result is greater than the logprobs threshold value, mark it as correct, then translate the token number to the question semantic according to the association mapping table of the question semantic label-code to generate the target annotated corpus, if the return result is less than the logprobs threshold value, mark it as incorrect and discard it.

[0073] In an optional embodiment, the preset limited parameters include limiting the code expression to correspond to a token number and limiting five return results.

[0074] The application trains a language model through a small amount of manually annotated sample corpus, converts a production corpus task of the language model into a classification corpus task through code expression, realizes production of a corpus through the language model, and meets the demand of feeding corpus in daily model training. The annotation of the sample corpus is judged through the trained language model, the correctly annotated corpus can be effectively generated, the production efficiency of the corpus is improved, a small amount of manual annotation can be used for training the model to realize production of the target corpus, and the time cost and production cost of the manual annotation part of the corpus are greatly reduced.

[0075] As shown in Figure 2 The second aspect of the application discloses a language model corpus training system based on a classification task, which comprises:

[0076] The acquisition module is used for acquiring sample corpus and performing data cleaning operation;

[0077] The preprocessing module is used for selecting part of the sample corpus after data cleaning for manual annotation, obtaining first annotated corpus, and the first annotated corpus comprises a training set, a validation set and a test set;

[0078] The expression conversion module is used for converting the question semantic label manually annotated in the first annotated corpus into code expression, and constructing a question semantic label-code association mapping table, wherein the question semantic label in the association mapping table corresponds to the code expression one by one;

[0079] The model training module is used for converting the generation task of the language model into a classification task through the code expression, fine-tuning the input first annotated corpus and the code expression by using the voice model, determining the logprobs threshold value according to the return result of the fine-tuning, determining the second annotated corpus through the logprobs threshold value and saving it in the corpus set to complete the language model training;

[0080] The corpus generation module is used for training the sample corpus based on the corpus set and the logprobs threshold value through the trained language model, and obtaining the target annotated corpus according to the question semantic label-code association mapping table.

[0081] The third aspect of the application discloses a language model corpus training device based on a classification task, which comprises:

[0082] At least one processor, and

[0083] The memory in communication connection with the at least one processor; wherein

[0084] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the language model corpus training method based on a classification task as any one disclosed in the first aspect of the present application.

[0085] The computer device can be a terminal, and the computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement the language model corpus training method based on a classification task. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad or mouse, etc.

[0086] The fourth aspect of the present application discloses a computer readable storage medium, and the computer readable storage medium stores computer executable instructions. The computer executable instructions are configured to enable a computer to perform the language model corpus training method based on a classification task as any one disclosed in the first aspect of the present application.

[0087] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment of each training method of the language model corpus based on the classification task. Wherein, any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0088] Alternatively, the above-mentioned modules of the present application, if realized in the form of software function modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the embodiments of the present application. The above-mentioned storage medium includes mobile storage devices, RAM, ROM, magnetic or optical disks, and various media that can store program codes.

[0089] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a language model corpus based on a classification task, characterized in that, The method comprises: S1: collecting sample corpus and performing data cleaning operation; S2: selecting part of the sample corpus after data cleaning for manual annotation to obtain a first annotated corpus, the first annotated corpus comprising a training set, a validation set and a test set; S3: converting the manually annotated question semantic label in the first annotated corpus into code expression, comprising: S31: converting the first annotated corpus into question text and question semantic expression based on a prompt template, the question semantic expression being a corresponding token number of code expression, the first annotated corpus being saved in Json format after conversion; and constructing a question semantic label-code association mapping table, wherein the question semantic label and the code expression in the association mapping table correspond one-to-one; S4: converting the generation task of the language model into a classification task through the code expression, fine-tuning the language model on the input first annotated corpus and code expression, determining a logprobs threshold value according to the return result of fine-tuning, determining a second annotated corpus through the logprobs threshold value and saving it in a corpus set to complete the language model training; S5: training the sample corpus based on the corpus set and the logprobs threshold value using the trained language model, and obtaining a target annotated corpus according to the question semantic label-code association mapping table. 2.The method of claim 1, wherein, The collection of sample corpus and data cleaning operation comprises: S11: collecting sample corpus in business data according to business requirements and / or collecting business-related sample corpus through network crawler; S12: scanning and removing invalid data in the sample corpus, the invalid data including duplicate data, website links, null data, emoticons and language model training irrelevant data. 3.The method of claim 1, wherein, The selection of part of the sample corpus after data cleaning for manual annotation to obtain a first annotated corpus comprises: S21: taking no less than 10% of the sample corpus data for manual annotation after analyzing the question semantic of the sample corpus, to obtain a first annotated corpus; S22: taking 70%, 20% and 10% of the annotated data in the first annotated corpus as a training set, a validation set and a test set, respectively. 4.The method of claim 1, wherein, The determination of the logprobs threshold value according to the return result of fine-tuning, the determination of the second annotated corpus through the logprobs threshold value and the saving of the second annotated corpus in the corpus set to complete the language model training comprises: S41: the language model input in the fine-tuning is limited to a corresponding token number and five return results, and the language model outputs five return values and logarithmic likelihoods after iteration; S42: determining the logprobs threshold value used by the language model to constrain the return result according to the return value and the logarithmic likelihood. S43: evaluate the return result of the language model input according to the logprobs threshold value, if the return result is greater than the logprobs threshold value, mark the input first annotated corpus as correct and save it in the corpus set as the second annotated corpus, if the return result is less than the logprobs threshold value, mark the input first annotated corpus as error and discard, continue iteration until the training of the language model is completed. 5.The method of claim 1, wherein, The training of the trained language model based on the corpus set and the logprobs threshold value includes: S51: call the Completion interface of the trained language model, input the sample corpus after presetting the limited parameters for iteration; S52: compare the return result of the sample corpus with the logprobs threshold value, if the return result is greater than the logprobs threshold value, mark it as correct, and then translate the token number into the prompt semantic through the association mapping table of the prompt semantic label-code to generate the target annotated corpus, if the return result is less than the logprobs threshold value, mark it as error and discard. 6.The method of claim 5, wherein, The preset limited parameters include limiting the code expression to correspond to a token number and limiting five return results. 7.A language model corpus training system based on a classification task, characterized in that, The system comprises: a collection module for collecting sample corpus and performing data cleaning operation; a preprocessing module for selecting part of the sample corpus after data cleaning for manual annotation to obtain first annotated corpus, the first annotated corpus including training set, validation set and test set; an expression conversion module for converting the prompt semantic label manually annotated in the first annotated corpus into code expression, including: converting the expression of the first annotated corpus into question text and prompt semantic based on the prompt template, the expression of the prompt semantic being the corresponding token number of the code expression, and the first annotated corpus being saved in Json format after conversion; and constructing an association mapping table of prompt semantic label-code, the prompt semantic label in the association mapping table corresponding to the code expression one by one; a model training module for converting the generation task of the language model into a classification task through the code expression, fine-tuning the first annotated corpus and the code expression input by the language model, determining the logprobs threshold value according to the return result of fine-tuning, determining the second annotated corpus through the logprobs threshold value and saving it in the corpus set to complete the training of the language model; a corpus generation module for training the sample corpus based on the corpus set and the logprobs threshold value by the trained language model, and translating the target annotated corpus according to the association mapping table of the prompt semantic label-code.

8. A language model corpus training device based on a classification task, characterized by, comprises: at least one processor, and a memory connected in communication with the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the language model corpus training method based on a classification task according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to perform the language model corpus training method based on a classification task according to any one of claims 1 to 6.

Images