Retrieval question condition segmentation system based on pre-trained language model and random sample generation

Abstract

The technical solution of the present invention is to disclose a retrieval question conditional segmentation system based on a pre-trained language model and random sample generation, characterized in that it includes: a punctuation correction model; an error correction data pair generation unit. The present invention adopts a compromise strategy, selects a small-volume pre-trained text encoder model (such as Bert, Roberta, Ernie) as the base model, and indirectly solves the sub-conditional segmentation problem through punctuation annotation. At the same time, in the present invention, the punctuation marks in the original user question input are directly corrected to avoid the information loss caused by de-punctuation.

Description

Technical Field

[0001] The invention relates to a retrieval question condition segmentation system based on a pre-trained language model and random sample generation. Background Art

[0002] With the rapid development of artificial intelligence, the company's Internet and financial businesses are actively transforming and upgrading towards intelligence. By combining deep learning models with the business, it can not only provide users with faster and smarter services, but also save a lot of manpower and time costs. Take the intelligent stock picker as an example. With the iteration of the version, its function has been recognized by more and more users. Every day, tens of thousands of users use the stock picker to search for stocks that meet various conditions. After receiving the search query entered by the user, the stock picker's first task is to completely and accurately identify the sub-conditions contained therein for the purpose of subsequent process analysis. The difficulty here lies in how to accurately identify the boundaries of each sub-condition. In most cases, the search query entered by the user is a list of multiple search conditions explained in natural language. At the same time, most users do not use punctuation to appropriately segment the sub-conditions, which also makes it difficult for the system to properly handle the situation when encountering fuzzy sub-boundaries (such as "Don't choose the Science and Technology Innovation Board or Growth Enterprise Market for stocks with a stock price greater than 25 consecutive boards"). The simplest way to solve this problem is to configure multiple segmentation templates, and segment when the user's question matches the characteristics of a template. This method can quickly and specifically solve some bad cases, but it also has obvious shortcomings. The first is the weak generalization ability. Only user questions that match the template can be properly segmented. When the number of templates reaches a certain level, it will bring a strong leverage effect, which is difficult to balance well. In addition, too many templates will also make the code heavy and difficult to maintain. Relatively speaking, relying on deep learning models can not only reduce the resource consumption required to configure templates, but also bring better generalization capabilities to the stock selection system. At present, in the field of natural language processing, large language models have become the most effective tool for solving natural language understanding tasks. With the continuous expansion of parameter scale, the step-by-step reasoning ability and instruction-following ability of large language models have become stronger and stronger. In many tasks, they have achieved better results than humans. In this scenario, by designing a suitable prompt, the ability of the large language model can be used to segment the sub-conditions in the question. However, calling a large language model faces obvious performance issues: According to the scaling law, the larger the model is, the better the task is, but correspondingly, the longer it takes to call it once it is fully trained. Since online stock pickers have extremely high requirements for concurrency, although a large model can achieve the best results, it is not suitable for this scenario. This shows that building an intelligent sub-condition segmentation system based on a small-scale deep learning model to achieve rapid segmentation of search questions has important business value.

[0003] Before the advent of the big language model, the best text understanding model was the text encoder model built on the Transformer Encoder, such as Bert, Roberta, Ernie, etc. This type of model effectively extracts the information required for natural language understanding tasks from the text by stacking multiple layers of bidirectional encoders, and its parameter volume is about 1 / 100 of that of the big model, and the calling speed is fast. The existing solutions to the sentence segmentation problem based on the text encoder model mainly include two categories: one is to regard the task as a named entity recognition (NER) task, and each independent and complete sub-condition is an entity to be recognized; the other is to regard the task as a punctuation marking task, and indirectly achieve the segmentation of sub-conditions by de-punctuating the input question and then re-punctuating it. In project applications, both methods can achieve good results, but they also face some problems before going into production. First, in order to train the named entity recognition model, specific labels need to be created for each character, including: the beginning of the entity (Begin, B), the inside of the entity (Inside, I), the end of the entity (End, E), etc. However, for the collected user search questions, it takes a huge amount of time to create labels one by one, and it is difficult to accumulate a large number of samples. Secondly, in this scenario, the most critical problem to be solved is to identify the sub-conditional boundaries. For the second type of method, de-punctuation means ignoring the sentence segmentation information in the user's original input, which increases the difficulty of the problem. Summary of the invention

[0004] The technical problem to be solved by the present invention is: for the existing retrieval question condition segmentation system, the simple template configuration method is extremely inefficient and will bring a significant leverage effect. The optimal effect can be achieved by relying on the capabilities of a large language model, but it faces the problems of long time consumption and high cost.

[0005] In order to solve the above technical problems, the technical solution of the present invention is to provide a retrieval question condition segmentation system based on a pre-trained language model and random sample generation, which is characterized by comprising:

[0006] The punctuation correction model uses a small-scale pre-trained text encoder model as the base model;

[0007] The error correction data pair generation unit is used to generate a set Sp = {(s', s)}, where (s', s) is a pair of punctuation error correction data pairs, s is a manually annotated user search question, s∈S, S is a standard user search question set, and s' is a constructed incorrect user question. Specifically, the user search question with manually annotated punctuation is provided to the error correction data pair generation unit, and the error correction data pair generation unit generates the punctuation error correction data pairs required for the training model through a random sample generation algorithm, wherein the random sample generation algorithm includes the following steps:

[0008] Traverse each standard user question s in the standard user question set S and perform the following operations:

[0009] Step 1: Clean the current standard user question s;

[0010] Step 2: Add (s, s) to Sp, then s' = s;

[0011] Step 3, traverse each character c in s:

[0012] Case 1: If Where P is the standard punctuation set, then:

[0013] s'+=c, and at the same time randomly sample a punctuation point p from P with a preset probability, s'+=p;

[0014] Case 2: If c∈P, then:

[0015] Case 2.1: Ignore c with a preset probability of 2, and s' remains unchanged;

[0016] Case 2.2: With a preset probability of three, c is mistakenly replaced by p∈P (p≠c), at which time s'+=p;

[0017] Case 2.3: Keep c unchanged with the preset probability of four, then s'+=c, and randomly sample a punctuation point p from P with the preset probability of five, s'+=p;

[0018] Step 4. Add (s', s) to Sp.

[0019] Preferably, it also includes:

[0020] The data enhancement sample construction unit is used to generate new punctuation correction data pairs according to the new annotated question, add the generated punctuation correction data pairs to the punctuation correction data pair set Sp, and then iteratively train the trained punctuation correction model.

[0021] Preferably, after the data augmentation sample construction unit obtains the new labeled question:

[0022] The sub-conditions in the labeled question are segmented by the currently trained punctuation correction model, and the segmentation results are compared with the manual annotations. According to the comparison results, the sub-conditions in the labeled question are divided into two categories: the first category is the sub-conditions that the punctuation correction model output is consistent with the manual judgment, and the sub-condition set C1 is obtained; the second category is the sub-conditions that the punctuation correction model output is inconsistent with the manual judgment, and the sub-condition set C2 is obtained;

[0023] Traverse each sub-condition c2∈C2, randomly sample a certain number of sub-conditions from the sub-condition set C1 each time, splice the current sub-condition c2 with all the sub-conditions extracted from C1 in a random order, and use the spliced ​​samples as the constructed user retrieval annotation question, so that a new punctuation correction data pair can be generated by the above-mentioned correction data pair generation unit.

[0024] The present invention adopts a compromise strategy, selects a small-volume pre-trained text encoder model (such as Bert, Roberta, Ernie) as the base model, and indirectly solves the sub-conditional segmentation problem through punctuation annotation. At the same time, in the present invention, the punctuation marks in the original user question input are directly corrected to avoid the information loss caused by de-punctuation.

[0025] Specifically, compared with the prior art solutions, the present invention has the following beneficial effects:

[0026] 1. In order to solve the problem of high performance and low cost in large language model, the present invention adopts a small text encoder model as the base model of the system, which has low calling time and easy deployment. At the same time, compared with the simple template configuration method, the system of the present invention has good generalization performance.

[0027] 2. In response to the difficulty in collecting training data faced by training encoder models, the present invention designs a random sample generation algorithm for the conditional sentence segmentation task, which greatly reduces the time cost required to produce training samples.

[0028] 3. In order to solve the problem of information loss caused by the de-punctuation model, the present invention adopts a method of directly correcting the punctuation in the original question input by the user. The specific correction includes the "addition", "deletion" and "modification" operations on the punctuation. In actual application, the system can simultaneously complete the "addition", "deletion" and "modification" operations on the input question, so that in the final output result, the sub-conditions are completely and accurately separated by punctuation marks.

[0029] 4. For the subsequent optimization iterations of the model, the present invention specifically designs a data enhancement method based on difficult sample mining. By classifying, sampling, and randomly splicing sub-conditions to construct data enhancement samples, the model's ability to identify difficult sub-conditions is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the system principle diagram of the present invention;

[0031] Figure 2 Generate schematics for error correction pairs. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0033] The core of a retrieval question condition segmentation system based on a pre-trained language model and random sample generation disclosed in an embodiment of the present invention is a punctuation correction model. The punctuation correction data required for training the punctuation correction model is constructed based on annotated real user questions and a random sample generation algorithm. In the subsequent optimization and iteration process of the punctuation correction model, when a certain amount of new annotated data is accumulated, data enhancement can be performed by mining difficult sub-conditions, thereby accelerating the learning speed of the model to identify difficult sub-conditions on the basis of further expanding the amount of training samples.

[0034] Combination Figure 1 The embodiment of the present invention discloses a retrieval question condition segmentation system based on a pre-trained language model and random sample generation, which specifically includes:

[0035] The punctuation correction model uses a small-scale pre-trained text encoder model as a base model. In the embodiment of the present invention, the Ernie model is used.

[0036] The error correction data pair generation unit is used to generate a set of punctuation error correction data pairs for training the punctuation correction model. The user search questions with manual punctuation annotations are provided to the error correction data pair generation unit, and the error correction data pair generation unit generates the punctuation error correction data pairs required for the training model through a random sample generation algorithm.

[0037] The core idea of ​​the random sample generation algorithm is to simulate various punctuation usage situations that may occur in actual applications through random events at the character level, so as to generate a punctuation correction data pair set Sp = {(s', s)} for training the punctuation correction model, where (s', s) is a pair of punctuation correction data pairs, s is a standard user retrieval question after manual annotation, s∈S, S is a standard user question set, and s' is a constructed incorrect user question; the specific algorithm flow is as follows:

[0038] Combination Figure 2 , traverse each standard user question s in the standard user question set S and perform the following operations:

[0039] Step 1: Clean the current standard user question s, including half-width conversion and end punctuation removal;

[0040] Step 2: Add (s, s) to Sp, then s' = s;

[0041] Step 3: Initialize s' to an empty string and iterate over each character c in s:

[0042] Case 1: If Where P is the standard punctuation set, then:

[0043] s'+=c, and at the same time randomly sample a punctuation point p from P with probability 0.05, s'+=p;

[0044] Case 2: If c∈P, then:

[0045] Case 2.1: Ignore c with a probability of 0.4, and s' remains unchanged;

[0046] Case 2.2: c is mistakenly replaced by p∈P (p≠c) with a probability of 0.4, then s'+=p;

[0047] Case 2.3: Keep c unchanged with probability 0.2, then s'+=c, and randomly sample a punctuation point p from P with probability 0.1, s'+=p;

[0048] Step4: Add (s',s) to Sp.

[0049] In the present invention, the purpose of incorporating samples in Step 2 is to enable the model to recognize standard search questions, that is, when the sub-conditions in the input search question are accurately and completely separated by punctuation marks, no changes are required; Step 3 simulates the most common situation in actual applications of correcting some punctuation marks in the question, among which Case 2.1 simulates the situation of missing punctuation marks, Case 2.2 simulates the situation of incorrect use of punctuation marks, and Case 1 and Case 2.3 simulate the situation of correct and redundant use of punctuation marks. In the embodiment of the present invention, 0.02 is taken as the probability of the occurrence of samples of the redundant situation of multiple consecutive punctuation marks in the simulated question because the probability of this abnormal situation is the lowest. It is worth noting that the probabilities between different Cases here can be flexibly adjusted to achieve a better simulation of the actual situation. The characters in the sentence pairs in the error correction pair are one-to-one corresponding, and character-level labels can be easily produced by comparing character by character.

[0050] Since the use of punctuation marks in various combinations has been comprehensively considered during the construction of the error correction pairs, the trained punctuation correction model can simultaneously realize the addition, deletion, and modification of punctuation marks in the user-input questions, thereby indirectly realizing the segmentation of sub-conditions in the questions.

[0051] Data enhancement sample construction unit. In this unit, the present invention specifically designs a data enhancement method based on difficult sample mining. The core idea comes from the fact that with the continuous iteration of the model, the contribution of a small number of difficult samples to the model capability is often higher than the contribution of a large number of simple samples to improving the model capability.

[0052] The specific implementation process of the data enhancement sample construction unit is as follows:

[0053] When a batch of new annotated questions is received, the punctuation segmentation of the sub-conditions is first performed through the currently trained punctuation correction model. Then the segmentation results are compared with the manual annotations, and the sub-conditions in the annotated questions are divided into two categories according to the comparison results: the first category is the sub-conditions that the punctuation correction model and the manual judgment are consistent, and the sub-condition set C1 is obtained. This type of sub-condition indicates that the punctuation correction model can already fully and accurately identify it; the second category is the sub-conditions that the punctuation correction model output is inconsistent with the manual judgment, and the sub-condition set C2 is obtained. This type of sub-condition indicates that the current punctuation correction model cannot fully and accurately identify it, which is the difficult sample mined.

[0054] After completing the sub-condition classification, traverse each sub-condition c2∈C2, repeat N times, randomly sample 3-5 sub-conditions from C1 each time, splice the sub-condition c2 with multiple sub-conditions extracted from C1 in a random order, and use the spliced ​​samples as the constructed user retrieval annotation question.

[0055] The annotated questions constructed above are expanded to the user standard question set S, and then the corresponding punctuation correction data pairs are generated by the correction data pair generation unit. Finally, the newly generated punctuation correction data pairs are expanded to the training sample set, thereby realizing iterative training of the punctuation correction model.

[0056] In particular, for the sub-condition c2 that the current punctuation correction model cannot accurately identify, it may be a single condition and is classified into C2 because the current punctuation correction model incorrectly divides it into multiple conditions. It may also be a composite condition composed of multiple conditions and is classified into C2 because the current model fails to divide it properly. The above data augmentation method can be automatically implemented based on the trained model and the new annotated questions. In actual applications, it is found that this method can effectively accelerate the model to learn the features of difficult sub-conditions and achieve more effective model iteration.

Claims

1. A retrieval question condition segmentation system based on a pre-trained language model and random sample generation, characterized in that: include: The punctuation correction model uses a small-scale pre-trained text encoder model as the base model; The error correction data pair generation unit is used to generate a set Sp = {(s', s)}, where (s', s) is a pair of punctuation error correction data pairs, s is a manually annotated user search question, s∈S, S is a standard user search question set, and s' is a constructed incorrect user question. Specifically, the user search question with manually annotated punctuation is provided to the error correction data pair generation unit, and the error correction data pair generation unit generates the punctuation error correction data pairs required for the training model through a random sample generation algorithm, wherein the random sample generation algorithm includes the following steps; Traverse each standard user question s in the standard user question set S and perform the following operations: Step 1: Clean the current standard user question s; Step 2: Add (s, s) to Sp, then s' = s; Step 3, traverse each character c in s: Case 1: If Where P is the standard punctuation set, then: s'+=c, and at the same time randomly sample a punctuation point p from P with a preset probability, s'+=p; Case 2: If c∈P, then: Case 2.1: Ignore c with a preset probability of 2, and s' remains unchanged; Case 2.2: With a preset probability of three, c is mistakenly replaced by p∈P (p≠c), at which time s'+=p; Case 2.3: Keep c unchanged with the preset probability of four, then s'+=c, and randomly sample a punctuation point p from P with the preset probability of five, s'+=p; Step 4. Add (s', s) to Sp.

2. A retrieval question condition segmentation system based on a pre-trained language model and random sample generation as claimed in claim 1, characterized in that: Also includes: The data enhancement sample construction unit is used to generate new punctuation correction data pairs according to the new annotated question, add the generated punctuation correction data pairs to the punctuation correction data pair set Sp, and then iteratively train the trained punctuation correction model.

3. A search question condition segmentation system based on a pre-trained language model and random sample generation as claimed in claim 2, characterized in that: After the data augmentation sample construction unit obtains a new labeled question: The sub-conditions in the labeled question are segmented by the currently trained punctuation correction model, and the segmentation results are compared with the manual annotations. According to the comparison results, the sub-conditions in the labeled question are divided into two categories: the first category is the sub-conditions that are consistent with the punctuation correction model and the manual judgment, and the sub-condition set C1 is obtained; the second category is the sub-conditions that are inconsistent with the manual judgment of the punctuation correction model output, and the sub-condition set C2 is obtained; Traverse each sub-condition c2∈C2, randomly sample a certain number of sub-conditions from the sub-condition set C1 each time, splice the current sub-condition c2 with all the sub-conditions extracted from C1 in a random order, generate corresponding punctuation correction data pairs through the correction data pair generation unit for the spliced ​​samples, and finally expand the new punctuation correction data pairs to the training sample set, thereby realizing iterative training of the punctuation correction model.

Images