A Text Processing Method Based on Sampling Ratio

Abstract

The present invention relates to the technical field of electrical digital data processing, and particularly relates to a text processing method based on a sampling ratio. The method includes: obtaining the text a to be sampled of the i-th category i , a i The number of texts to be sampled included is q i , a i The q i texts to be sampled included have the same label; obtaining the sampling quantity s i of the i-th category; obtaining the sampling ratio r i , r i = s i / q i ; if r i < r0, then cluster a i according to the text similarity between the texts to be sampled included in a i to obtain a clustering result G i ; obtaining the number s i of texts to be sampled included in the smallest cluster in G i,min ; if s i,min ≥ f(s i / m), then traverse G i , and obtain f(s i,j / m) texts to be sampled from g i to form the initial sampling texts of the i-th category. The present invention improves the inference ability of the trained semantic analysis model.

Description

Technical Field

[0001] The invention relates to the technical field of electronic digital data processing, and in particular to a text processing method based on sampling ratio. Background Art

[0002] When sampling a text set corresponding to a certain label for training a semantic analysis model, the following problem may occur: the text set corresponding to the label includes a large number of texts, while the number of texts to be sampled from the text set corresponding to the label is small, that is, the sampling of the text set corresponding to the label is relatively small. In this case, if sampling is performed from the text set corresponding to the label by random sampling, the similarity between the sampled texts may be relatively large, affecting the reasoning ability of the semantic analysis model trained based on the sampled texts about the label.

[0003] The prior art has disclosed a method for clustering texts based on text similarity. For example, a Chinese patent application with publication number CN116401565A discloses a method for training a text clustering model and a text clustering method based thereon. The patent application obtains a text clustering model with stronger semantic analysis capability based on a pre-trained BERT model, and clustering of the texts to be clustered can be achieved based on the text clustering model. On the basis of the prior art method for clustering texts based on text similarity, how to avoid the situation where the sampled texts have high similarity when sampling a text set corresponding to a certain label, and thereby improve the reasoning ability of the trained semantic analysis model for the label, is a problem to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to provide a text processing method based on sampling ratio, which is used to optimize the method of sampling the text set corresponding to the label, thereby improving the reasoning ability of the trained semantic analysis model for the label.

[0005] According to the present invention, a text processing method based on sampling ratio comprises the following steps:

[0006] E100, get the i-th type of text to be sampled a i , a i The number of texts to be sampled is q i , a i Included q i The labels of the texts to be sampled are the same, but different i The labels of the texts to be sampled are different, and the value of i ranges from 1 to n, where n is the number of categories of the texts to be sampled.

[0007] E200, get the number of samples s of the i-th category i .

[0008] E300, obtain the sampling ratio r of the i-th category i , r i = s i / q i .

[0009] E400, if r i < r0, then enter E500; r0 is a preset sampling ratio threshold.

[0010] E500, according to the text similarity between the text to be sampled included in a i cluster a i to obtain a clustering result G i , G i includes m clusters, and m is the number of clusters obtained by clustering a i .

[0011] E600, obtain the number s of the text to be sampled included in the smallest cluster in G i . i,min .

[0012] E700, if s i,min ≥ f(s i / m), then enter E800; f() is a rounding function.

[0013] E800, traverse G i , and obtain f(s i,j / m) texts to be sampled from each g i to form the initial sampling text of the i-th category.

[0014] The present invention has at least the following beneficial effects:

[0015] For the text a to be sampled of the i-th category in the present invention i , first obtain its corresponding sampling ratio r i , r i is the ratio of the corresponding sampling quantity of a i to the number of texts included in a i . If the corresponding sampling ratio r i is less than the preset sampling ratio threshold r0, it is determined that the number of texts sampled from a i is relatively small. To solve the problem that the similarity of the sampled texts may be relatively high when randomly sampling a i , the present invention clusters a i according to the text similarity between the texts to be sampled included in a i . For the obtained m clusters, first obtain the number s of texts included in the cluster with the smallest number of texts (i.e., the smallest cluster) among these m clusters i,min , if si,min ≥f(s i / m), then from each g i,j Get f(s i / m) samples of texts to be sampled constitute the initial sampled text of the i-th category. i The m clusters represent a i The differences between the texts in the clusters are relatively large, so the present invention realizes the i Diversified sampling, compared to randomly sampling from the i-th category of text a i The text sampled by the present invention is more diversified, which improves the reasoning ability of the semantic analysis model trained based on the sampled text for the label. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a flowchart of a text processing method based on sampling ratio provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] Embodiment 1:

[0020] According to this embodiment, a text processing method based on sampling ratio is provided, such as Figure 1 As shown, including:

[0021] E100, get the i-th type of text to be sampled a i , a i The number of texts to be sampled is q i , a i Included q i The labels of the texts to be sampled are the same, but different i The labels of the texts to be sampled are different, and the value of i ranges from 1 to n, where n is the number of categories of the texts to be sampled.

[0022] In this embodiment, the text to be sampled is A, and A = (a1, a2, …, a i , …, a n ), where a i is the i-th type of text to be sampled.

[0023] E200. Obtain the sampling quantity s i of the i-th type.

[0024] In this embodiment, s i is an empirical value.

[0025] E300. Obtain the sampling ratio r i of the i-th type, where r i = s i / q i .

[0026] E400. If r i < r0, then proceed to E500; r0 is a preset sampling ratio threshold.

[0027] In this embodiment, r0 is an empirical value. Optionally, r0 is 5% or 10%.

[0028] In this embodiment, E400 further includes: If r i ≥ r0, then proceed to E410.

[0029] E410. If q i ≥ s i , then randomly select s i texts to be sampled from a i to form the target sampling text of the i-th type; otherwise, determine a i as the target sampling text of the i-th type.

[0030] E500. Cluster a i based on the text similarity between the texts to be sampled included in a i to obtain a clustering result G i , where G i includes m clusters, and m is the number of clusters obtained by clustering a i .

[0031] In this embodiment, G i = (g i,1 , g i,2 , …, g i,j , …, g i,m ), where g i,j is the j-th cluster obtained by clustering a i , and the value range of j is from 1 to m.

[0032] Those skilled in the art are aware that any method for obtaining text similarity in the prior art falls within the protection scope of the present invention.

[0033] Those skilled in the art know that any clustering method in the prior art falls within the protection scope of the present invention. Optionally, a density-based clustering method (such as DBSCAN) is used to cluster a i Clustering is performed, where a i Each of the texts to be sampled is taken as a point, and the distance between the two texts to be sampled is used to realize a i The distance between two to-be-sampled texts is negatively correlated with the text similarity between the two to-be-sampled texts. Therefore, a i The sampled texts with higher text similarity are clustered into one cluster, a i The sampled texts with lower text similarity are clustered into different clusters.

[0034] E600, Get G i The number of texts to be sampled in the smallest cluster s i,min .

[0035] In this embodiment, G i The smallest cluster in G i The cluster with the least amount of text is included.

[0036] E700, if s i,min ≥f(s i / m), then enter E800; f( ) is the rounding function.

[0037] Optionally, f( ) is a floor function or a round-to-nearest function.

[0038] E800, traverse G i , from each g i,j Get f(s i / m) texts to be sampled constitute the initial sampling text of the i-th category.

[0039] Optional, randomly selected from each g i,j Get f(s i / m) texts to be sampled constitute the initial sampling text of the i-th category.

[0040] In this embodiment, for the i-th type of sampled text a i , first obtain the corresponding sampling ratio r i , r i for a i The corresponding number of samples is i The ratio of the number of texts included, if its corresponding sampling ratio r i is less than the preset sampling ratio threshold r0, and the iThe number of sampled texts is relatively small. i Random sampling may lead to a high similarity of the sampled texts. In this embodiment, according to a i The text similarity between the texts to be sampled includes a i For the m clusters obtained, first obtain the number of texts s included in the cluster with the least number of texts (i.e. the smallest cluster) among the m clusters i,min , if s i,min ≥f(s i / m), then from each g i,j Get f(s i / m) samples of texts to be sampled constitute the initial sampled text of the i-th category. i The m clusters represent a i The differences between the texts in the m clusters are relatively large, so this embodiment implements the i Diversified sampling, compared to randomly sampling from the i-th category of text a i In the sampling method, the text sampled in this embodiment is more diverse, which improves the reasoning ability of the semantic analysis model trained based on the sampled text for the label.

[0041] As a specific implementation, after E800, the method further includes:

[0042] E900, if m×f(s i / m)=s i , then the initial sampled text of the i-th category is determined as the target sampled text of the i-th category; if m×f(s i / m)>s i , then randomly delete m×f(s i / m)-s i text, the initial sampled text of the i-th category after deletion is determined as the target sampled text of the i-th category, and the deleted m×f(s i / m)-s i The text belongs to a i m×f(s i / m)-s i clusters; if m×f(s i / m) i , then from a i Randomly get s from the remaining text i -m×f(s i / m) to-be-sampled texts are added to the initial sampled text of the i-th category, and the added initial sampled text of the i-th category is determined as the target sampled text of the i-th category; a i The remaining text is a i ​The text except the initial sampled text of the i-th category.

[0043] This embodiment implements the judgment of the number of texts in the initial sampled text of the i-th category based on E900. If m×f(s i / m)≠s i , then a small adjustment is made based on the initial sampled text of the i-th category, minimizing the i The difference in the number of texts sampled in different clusters due to a i The m clusters represent a i m kinds of text in a, so this embodiment realizes i The diverse sampling improves the subsequent trained semantic analysis model i The reasoning ability of the corresponding label.

[0044] As a specific implementation, E700 further includes: if s i,min <f(s i / m), enter E710.

[0045] E710, traverse G i , from g i,j Get s i,min The texts to be sampled constitute the first sampled text of the i-th category; g i,j For a i The jth cluster obtained by clustering, the value of j ranges from 1 to m.

[0046] Optional, randomly selected from each g i,j Get s i,min The texts to be sampled constitute the first sampled text of the i-th category.

[0047] E720, obtain the first remaining sampling quantity Δε1, Δε1=s i -m×s i,min .

[0048] E730, obtain the clustering result G after the first sampling 1 i , G 1 i Including m clusters, G 1 i The jth cluster included is g 1 i,j , g 1 i,j From g i,j Get s i,min The clusters are composed of the remaining texts after the texts to be sampled.

[0049] In this embodiment, G 1i =(g 1 i,1 ,g 1 i,2 ,…,g 1 i,j ,…,g 1 i,m ).

[0050] In this embodiment, if g i,j The amount of text included is greater than s i,min , then g 1 i,j is a non-empty cluster; if g i,j Includes an amount of text equal to s i,min , then g 1 i,j An empty cluster.

[0051] E740, Get G 1 i The number of non-empty clusters in δ.

[0052] E750, if Δε1=δ, then from G 1 i Obtain one to-be-sampled text from each non-empty cluster and append it to the first sampled text of the i-th category, and determine the first sampled text of the i-th category after the appending as the target sampled text of the i-th category.

[0053] Based on E710-E750, this embodiment is in G i The number of texts to be sampled in the smallest cluster s i,min Less than f(s i / m) in the case of i,j Get s i,min The texts to be sampled constitute the first sampled text of the i-th category, and then obtain the first remaining sample number Δε1 and the number of non-empty clusters δ. If δ=Δε1, then obtain 1 text to be sampled from each non-empty cluster and append it to the first sampled text of the i-th category. Therefore, this embodiment minimizes the number of samples from a i The difference in the number of texts sampled in different clusters due to a i The m clusters represent a i m kinds of text in a, so this embodiment realizes i The diverse sampling improves the subsequent trained semantic analysis model i The reasoning ability of the corresponding label.

[0054] In this embodiment, E750 further includes: if Δε1<δ, then from G 1 iObtain one to-be-sampled text from each target cluster of G and append it to the first sampled text of the i-th category, and determine the first sampled text of the i-th category after appending as the target sampled text of the i-th category; the target cluster is 1 i Δε1 non-empty clusters are randomly selected from the δ non-empty clusters.

[0055] In this embodiment, G i The number of texts to be sampled in the smallest cluster s i,min Less than f(s i / m) in the case of i,j Get s i,min The texts to be sampled constitute the first sampled text of the i-th category, and then obtain the first remaining sample quantity Δε1 and the number of non-empty clusters δ. If Δε1<δ, a small adjustment is made to the first sampled text of the i-th category to minimize the number of samples from a i The difference in the number of texts sampled in different clusters due to a i The m clusters represent a i m kinds of text in a, so this embodiment realizes i The diverse sampling improves the subsequent trained semantic analysis model i The reasoning ability of the corresponding label.

[0056] Preferably, E750 also includes: if Δε1>δ, entering E751.

[0057] E751, Get G 1 i The number of texts to be sampled in the smallest non-empty cluster s 1 i,min .

[0058] E752, if s 1 i,min ≥f(Δε1 / δ), then enter E753.

[0059] E753, from G 1 i Get f(Δε1 / δ) texts to be sampled from each non-empty cluster and append them to the first sampled text of the i-th category.

[0060] E754, if f(Δε1 / δ)×δ=Δε1, the first sample text of the i-th category after appending is determined as the target sample text of the i-th category.

[0061] Based on E751-E754, this embodiment is in G i The number of texts to be sampled in the smallest cluster s i,min Less than f(s iIn the case of ( / m), first obtain s from each g i,j to form the first sampled text of the i-th category for each s i,min texts to be sampled, and then obtain the first remaining sampling quantity Δε1 and the number δ of non-empty clusters. If δ < Δε1, obtain G 1 i the number s of texts to be sampled included in the smallest non-empty cluster in 1 i,min , if s 1 i,min ≥ f(Δε1 / δ), then obtain f(Δε1 / δ) texts to be sampled from each non-empty cluster and append them to the first sampled text of the i-th category. If f(Δε1 / δ)×δ = Δε1, then determine the appended first sampled text of the i-th category as the target sampled text of the i-th category; thus, in this embodiment, the difference in the number of texts sampled from different clusters of a i is minimized. Since the m clusters of a i represent m types of texts in a i , this embodiment realizes diversified sampling of a i and improves the inference ability of the subsequent trained semantic analysis model for the labels corresponding to a i .

[0062] In this embodiment, E754 further includes: if f(Δε1 / δ)×δ ≠ Δε1, obtain the target sampled text of the i-th category according to a method similar to E900, which will not be elaborated here.

[0063] In this embodiment, E754 further includes: if s 1 i,min < f(Δε1 / δ), obtain the target sampled text of the i-th category according to a method similar to E710 - E750, which will not be elaborated here.

[0064] Embodiment 2:

[0065] Considering that when sampling a text collection including different labels for training a semantic analysis model, if the number of texts corresponding to different labels is quite different, if the text collection is randomly sampled or sampled proportionally according to the number of texts corresponding to different labels, the proportion of texts corresponding to labels with a smaller number of corresponding texts in the sampled texts is relatively small, resulting in poor reasoning ability of the trained semantic analysis model for labels with a smaller number of corresponding texts; and if the number of texts corresponding to different labels is quite different, if a quantity threshold is directly set based on experience, when the number of texts corresponding to a certain label is greater than the quantity threshold, a small proportion of texts is sampled from the texts corresponding to the label; when the number of texts corresponding to a certain label is less than the quantity threshold, a large proportion of texts is sampled from the texts corresponding to the label; this sampling method is relatively rough, so that the quantity distribution of texts corresponding to different labels in the sampled text is quite different from the quantity distribution of texts corresponding to different labels in the original text collection; therefore, on the basis of the first embodiment, this embodiment further samples the number of samples s of the i-th category. i The acquisition process of s i =f((q i ) 0.5 / ((q i / (∑ n i=1 q i )) 1 / x )), x is the preset first value, x is a positive integer, and f( ) is the rounding function.

[0066] Based on the formula s of this embodiment i =f((q i ) 0.5 / ((q i / (∑ n i=1 q i )) 1 / x )) can extract as much text as possible from a text set including different labels in a class with a small number of texts. When the sampled text is subsequently used as training text for training a semantic analysis model, the reasoning ability of the trained semantic analysis model for labels corresponding to classes with a small number of texts can be improved. Moreover, based on the formula s of this embodiment, i =f((q i ) 0.5 / ((q i / (∑ n i=1 q i )) 1 / x)) can also achieve the purpose of sampling more classes that include a larger number of texts than classes that include a smaller number of texts, and can improve the consistency between the distribution of the number of texts corresponding to different labels in the sampled text and the distribution of the number of texts corresponding to different labels in the text to be sampled.

[0067] Optionally, x is an empirical value, for example, x=3.

[0068] Preferably, x=f( l g((max(Q)) 0.5 )) or x=f( l g((∑ n i=1 q i ) 0.5 )), Q is the sampling number sequence of A, Q=(q1,q2,…,q i ,…,q n ), max() is the maximum value function, l g is the logarithm to base 10. Small-scale experiments show that when x=f( l g((max(Q)) 0.5 )) or x=f( l g((∑ n i=1 q i ) 0.5 )), the quantity distribution of texts corresponding to different labels in the sampled text is basically consistent with the quantity distribution of texts corresponding to different labels in A.

[0069] Embodiment three:

[0070] Considering that when sampling a text set including different labels for semantic analysis model training, if the difference in the number of texts corresponding to different labels is small, that is, the number of texts corresponding to different labels is relatively balanced, the difference in sampling ratios of each category is relatively small, and the process of determining the preset sampling ratio threshold is relatively difficult; therefore, on the basis of the first embodiment, after E200 and before E300, the text processing method based on the sampling ratio further includes:

[0071] E201, obtain a text set A to be sampled, A includes n types of texts to be sampled, and the i-th type of text to be sampled included in A is a i .

[0072] In this embodiment, A={a1, a2, …, a i ,…,a n}.

[0073] E202, obtaining the sum w of the number of texts to be sampled included in the first u categories with the largest number of texts to be sampled in A, where u is a preset category quantity threshold.

[0074] Optionally, u is an empirical value or u=f(r'×n), r' is a preset label ratio threshold, and optionally, r' is 5%.

[0075] E203, if w / (∑ n i=1 q i )≥η, then go to E300; η is a preset ratio threshold; otherwise, go to E204.

[0076] Optional, η is an empirical value, optional, η is 50%.

[0077] E204, from a i Randomly obtain s i The texts to be sampled constitute the discriminant sample text B of the i-th category. i .

[0078] E205, pair B i Perform grouping and obtain the grouping result R i , R i ={R i,1 ,R i,2 ,…,R i,t ,…,R i,v}, R i,t For B i The t-th group is obtained by grouping, where the value of t ranges from 1 to v, and v is the preset number of groups.

[0079] Optional, for B i The included texts are randomly grouped so that each R i,t The number of included texts is greater than or equal to floor(s i / v) and less than or equal to ceil(s) i / v), floor() is used to round down, and ceil() is used to round up.

[0080] E206, according to R i The text similarity between the included groups is obtained by B i The repetition between Chinese texts is i .

[0081] Optionally, E206 includes:

[0082] E2061, Get the spliced ​​text CR i , CR i ={CR i,1 ,CR i,2 ,…,CR i,t ,…,CR i,v}, CR i,t To Ri,t The text obtained by concatenating the included texts.

[0083] Preferably, R i,t The included text is concatenated from beginning to end to get CR i,t .

[0084] E2062, Get CR i The text similarity of any two texts in , and append them to the preset similarity set SH, and get SH={sh1,sh2,…,sh d ,…,sh D}, sh d CR i The text similarity between the d-th pair of texts in , where d ranges from 1 to D, and D is CR i The corresponding number of text pairs, D=v×(v-1) / 2; SH is initialized to an empty set.

[0085] In this embodiment, from CR i Choose any two texts to form a pair of texts, CR i The amount of text included is v, so CR i The corresponding number of text pairs is v×(v-1) / 2.

[0086] Those skilled in the art are aware that any method for obtaining text similarity in the prior art falls within the protection scope of the present invention.

[0087] E2063, get sim i ,sim i =sum(SH) / D, sum( ) is the sum.

[0088] Based on E2061-E2063, this embodiment can obtain B i The repetition between Chinese texts is i ,sim i The larger the value, the higher the B i The higher the repetition between Chinese texts.

[0089] E207, if sim i ≤sim0, then B i Determine the target sample text of the i-th category; sim0 is the preset similarity threshold; otherwise, enter E500.

[0090] In this embodiment, sim0 is an empirical value, and optionally, sim0=0.5.

[0091] Based on the above E204-E207, in sim i >sim0, that is, from a i Randomly obtain si The discriminant sample text B of the i-th category is composed of the texts to be sampled i When the repetition rate of Chinese text is high, i The clustering method starts from a i Get s i The sampled texts constitute the target sampled text of the i-th category. i Randomly obtain s i The sampled texts constitute the target sampled text of the i-th category. i The clustering method to obtain the target sample text of the i-th category can significantly improve the i The diversity of the text sampled in the text analysis can significantly improve the subsequent semantic analysis model trained on a i The reasoning ability of the corresponding label; and for sim i ≤sim0, that is, from a i Randomly obtain s i The discriminant sample text B of the i-th category is composed of the texts to be sampled i If the repetition rate of Chinese text is low, directly replace B i Determining the target sample text as the i-th category is conducive to reducing the time for obtaining the target sample text of the i-th category and improving the efficiency of obtaining the target sample text of the i-th category.

[0092] Based on E202-E203, in w / (∑ n i=1 q i )≥η, that is, the number of texts corresponding to different labels included in A is quite different, that is, the number of texts corresponding to different labels is relatively unbalanced, execute E300-E400 to divide the n types of texts to be sampled included in A into 2 categories based on the preset sampling ratio threshold. The first category is the class with a sampling ratio less than the preset sampling ratio threshold (corresponding to the class with a large number of texts in A), and the other category is the class with a sampling ratio greater than or equal to the preset sampling ratio threshold (corresponding to the class with a small number of texts in A); the method of judging whether to perform cluster sampling based on the sampling ratio is more efficient, and for the class with a sampling ratio greater than or equal to the preset sampling ratio threshold, since the sampling ratio is relatively large, even if the class is directly randomly sampled, it will not have much impact on the inference effect of the label corresponding to the class by the later trained semantic analysis model. And in w / (∑ n i=1 q i )<η, that is, the difference in the number of texts corresponding to different labels included in A is small, that is, when the number of texts corresponding to different labels is relatively balanced, the difference in sampling ratios of each category is relatively small, and E204-E207 is executed without obtaining the preset sampling ratio threshold, but iThe repetition degree is judged. If the repetition degree is high, cluster sampling is performed on the class to improve the reasoning effect of the later trained semantic analysis model on the label corresponding to the class; the method of judging whether to perform cluster sampling based on the repetition degree in this embodiment can improve the reasoning effect of the later trained semantic analysis model on the labels corresponding to each class.

[0093] Embodiment 4:

[0094] Each text in the target sampled text obtained in the first embodiment has a corresponding labeled label, but the labeled label may be wrong, that is, the label annotated on the text does not match the text. In order to clean the target sampled text, the text processing method based on the sampling ratio of this embodiment also includes the following steps:

[0095] L100, get the text to be cleaned cle, cle=(cle1,cle2,…,cle ra ,…,cle to ),cle ra is the rath text to be cleaned, the value range of ra is 1 to to, and to is the number of texts to be cleaned.

[0096] In this embodiment, the target sample text is determined as tex, and to is the number of the target sample text.

[0097] L200, traverse cle, get cle ra Corresponding prompt word pro ra , pro ra Includes thought chain example text mod ra and the text to be inferred ans ra , mod ra =(mod ra,1 ,mod ra,2 ,…,mod ra,b ,…,mod ra,num ), mod ra,b For pro ra Includes the b-th thought chain example text, where b ranges from 1 to num, and num is cle ra The number of corresponding thought chain example texts; each mod ra,b Includes a corresponding target text tar ra,b , a corresponding label l ab ra,b and a corresponding judgment text tho ra,b , tho ra,b Including a pair l ab ra,b Is it tar? ra,b The label's judgment mark sigra,b and sig ra,b Corresponding reasoning text; ans ra Including cle ra andcle ra Tags l ab ra .

[0098] In this embodiment, if the value of num is small, that is, the number of thought chain example texts included in the prompt words input into the trained large language model is small, it is not conducive to the trained large language model to comprehensively learn the thinking of judging whether the label is correct, affecting the accuracy of the judgment result; if the value of num is large, that is, the number of thought chain example texts included in the prompt words input into the trained large language model is large, the trained large language model learns the thinking of judging whether the label is correct. It takes a long time, which is not conducive to quickly giving a judgment result; as an optional specific implementation method, num is an empirical value, optional, 2≤num≤10.

[0099] As a preferred embodiment, each l ab ra,b and l ab ra is a sub-tag under the same parent tag, thus, pro ra When used as a prompt word for a trained large language model, it is helpful for the trained large language model to learn the thinking that is more relevant to the text to be inferred, and further improve the accuracy of the output results of the trained large language model. For example, num=4, l ab ra For marriage and family disputes, l ab ra,1 For contract disputes, l ab ra,2 Disputes over compensation for property damage, l ab ra,3 For maintenance disputes, l ab ra,4 For marriage and family disputes, l ab ra , l ab ra,1 , l ab ra,2 , l ab ra,3 and l ab ra,4 They are all secondary tags. l ab ra , l ab ra,1 , l ab ra,2 , l abra,3 and l ab ra,4 The parent tags (i.e. first-level tags) of are all disputes. l ab ra , l ab ra,1 , l ab ra,2 , l ab ra,3 and l ab ra,4 All are sub-tags of the dispute tag.

[0100] As a specific implementation, tar ra,1 If there is a dispute between the two parties regarding the delivery date in the signed contract, l ab ra,1 For contract disputes, sig ra,1 To be correct, sig ra,b The corresponding reasoning text is: The two parties have signed a contract, and they have a dispute over the delivery date in the contract, which is a contractual dispute. Therefore, the label is correct.

[0101] L300, the pro ra Input the trained large language model to be judged by the trained large language model l ab ra Is it cle ra .

[0102] Those skilled in the art are aware that any large language model in the prior art falls within the protection scope of the present invention.

[0103] In this embodiment, ra It is a prompt word based on the thought chain. In this embodiment, pro ra As the input of the trained large language model, it is helpful for the trained large language model to learn the thinking of judging whether the annotated label is the label of the text, improve the accuracy of the output result of the trained large language model, and then improve the accuracy of cleaning.

[0104] L400, based on the trained large language model l ab ra Is it cle ra The result of the label judgment determines whether to cle ra Wash it off.

[0105] In this embodiment, for each text to be cleaned, the corresponding prompt word is first obtained, and the prompt word includes num thought chain example texts and the text to be inferred. The num thought chain example texts are used to enable the trained large language model to better learn the thinking of judging whether a label is a label of the text, so that the trained large language model can give a more accurate judgment result; the text to be inferred includes the text to be judged itself and the label corresponding to the text to be judged; thus, by inputting the prompt word corresponding to the text to be judged into the trained large language model, it is possible to automatically judge whether the label of the text is correct, thereby solving the problem of manual judgment of whether the label of the text is correct in the prior art, which occupies a lot of manual resources.

[0106] As a preferred specific implementation, L400 includes: if the trained large language model is l ab ra Is it cle ra If the judgment result of the tag is yes, enter L410.

[0107] L410, obtain the accuracy rate tpr of the trained large language model for the judgment result of yes.

[0108] In this embodiment, if the trained large language model l ab ra Is it cle ra The result of the label is yes, indicating that the trained large language model judges l ab ra for cle ra .

[0109] Optionally, the accuracy of the trained large language model for judging the result as yes is obtained based on the test text. When the accuracy tpr of the trained large language model for judging the result as yes is 90% and the trained large language model l ab ra Is it cle ra When the result of the label is yes, it means l ab ra for cle ra The probability of the label is 90%; when the trained large language model judges that the accuracy rate tpr is 78% and the trained large language model is l ab ra Is it cle ra When the result of the label is yes, it means l ab ra for cle ra The probability of the label is 78%.

[0110] L420, if tpr ≥ tf, then it is determined not to clean cle ra ; tf is a preset accuracy threshold.

[0111] In this embodiment, tf is an empirical value. Optionally, tf is 90% or 95%.

[0112] Based on L410 - L420, in this embodiment, when the accuracy of the trained large language model for the judgment result of "yes" is relatively high, the judgment result of the trained large language model is directly used as the standard. Thus, the text corresponding to the judgment result of "yes" output by the trained large language model in cle can be retained, achieving fast cleaning of these texts.

[0113] In this embodiment, L420 further includes: if tpr < tf, then go to L421.

[0114] L421, obtain the qua - time judgment result of the trained large language model on l ab ra whether it is the label of cle ra , where qua is a preset number of repeated inferences, qua is an odd number and qua ≥ 3.

[0115] L422, if there are ≥ (qua + 1) / 2 times of judgment results of "no" among the qua - time judgment results, then it is determined to clean cle ra ; otherwise, it is determined not to clean cle ra .

[0116] Based on L421 - L422, in this embodiment, when the accuracy of the trained large language model for the judgment result of "yes" is relatively low, the judgment result of the trained large language model is not directly used as the standard. Instead, the trained large language model is used to make multiple judgments on the prompt words corresponding to the text, and the judgment result with the most occurrences among the multiple judgments is used as the final judgment result. Thus, this embodiment can reduce the judgment deviation triggered by factors such as hallucinations in the trained large language model, thereby improving the accuracy of cleaning.

[0117] As a preferred specific implementation manner, L400 further includes: if the judgment result of the trained large language model on l ab ra whether it is the label of cle ra is "no", then go to L430.

[0118] L430, obtain the false positive rate fpr of the trained large language model for the judgment result of "no".

[0119] In this embodiment, if the trained large language model on l ab raWhether it is cle ra The judgment result of the label is no, indicating that the trained large language model judges l ab ra is not cle ra 's label.

[0120] Optionally, the accuracy rate of the trained large language model for the negative judgment result is obtained based on the test text. When the accuracy rate fpr of the trained large language model for the negative judgment result is 68% and the trained large language model for l ab ra Whether it is cle ra The judgment result of the label is no, indicating l ab ra is not cle ra The probability of the label is 68%; when the accuracy rate fpr of the trained large language model for the negative judgment result is 92% and the trained large language model for l ab ra Whether it is cle ra The judgment result of the label is no, indicating l ab ra is not cle ra The probability of the label is 92%.

[0121] L440, if fpr < tf, then enter L450; otherwise, clean cle ra up.

[0122] In this embodiment, tf is an empirical value. Optionally, tf is 90% or 95%.

[0123] L450, obtain the qua - th judgment result of the trained large language model for l ab ra Whether it is cle ra 's label, where qua is the preset number of repeated inferences, qua is an odd number and qua ≥ 3.

[0124] In this embodiment, qua is an empirical value. Optionally, qua is 3 or 5.

[0125] L460, if there are more than or equal to (qua + 1) / 2 negative judgment results among the qua - th judgment results, then judge to clean cle ra up; otherwise, judge not to clean cle ra up.

[0126] In this embodiment, if there are greater than or equal to (qua+1) / 2 negative judgment results among qua judgment results, it means that most of the judgment results among the qua judgment results are negative; if there are less than (qua+1) / 2 negative judgment results among the qua judgment results, it means that most of the judgment results among the qua judgment results are positive.

[0127] Based on L430-L460, when the accuracy of the trained large language model in judging the result of "no" is low, this embodiment does not directly take the judgment result of the trained large language model as the standard, but reuses the trained large language model to make multiple judgments on the prompt words corresponding to the text, and takes the judgment result with the largest number of occurrences in the multiple judgments as the final judgment result. Therefore, this embodiment can also recall the text from the text whose corresponding preliminary judgment result is "no", reduce the judgment deviation of the trained large language model triggered by the illusion light factor, and improve the accuracy of cleaning.

[0128] Embodiment five:

[0129] If the prompt words used to input into the trained large language model in the fourth embodiment are constructed manually, there is still a problem of occupying a lot of human resources. In order to further reduce the problem of occupying human resources, this embodiment, based on the fourth embodiment, also includes automatically obtaining pro ra process; specific, pro ra The acquisition process includes:

[0130] L210, will cle ra Input to the first trained model to obtain cle ra First-level label l ab 1 ra ; The trained first model is used to obtain the primary label of the input text, and the accuracy of the trained first model in obtaining the primary label of the input text is greater than or equal to tf, where tf is a preset accuracy threshold.

[0131] In this embodiment, the accuracy of the first-level label of the text inferred by the trained first model is relatively high. ra The output of the first model trained with ra First-level label l ab 1 ra .

[0132] In this embodiment, the trained first model is used to obtain the first-level label of the input text. The first-level label is a rough label. The first-level label also includes several second-level labels. The second-level label is a refined label. In this embodiment, the accuracy of the second-level label of the text inferred by the trained first model is low (i.e., less than tf). Therefore, this embodiment only uses the trained first model to obtain the first-level label of the text, and does not use the trained first model to obtain the second-level label of the text.

[0133] In this embodiment, the first model is a neural network model, and a supervised training method is used when training the first model. Optionally, the process of training the first model includes: obtaining a training text set, the training text set includes a plurality of training texts; obtaining a training label set, the training label set includes a primary label corresponding to each training text; training the first model according to the training text set and the training label set to obtain a trained first model.

[0134] L220, according to l ab 1 ra Match in the preset thinking chain sample text library bas to obtain l ab 1 ra Matching num thought chain example text mod ra ; bas=(bas1,bas2,…,bas ce ,…,bas he ),bas ce The sub-library of example texts of the thinking chain corresponding to the ceth first-level label corresponding to cle, bas ce =(bas ce,1 ,bas ce,2 ,…,bas ce,me ,…,bas ce,te ),bas ce,me is the example text of the thinking chain corresponding to the meth second-level label included in the ceth first-level label corresponding to cle, the value range of me is 1 to te, te is the number of second-level labels included in the ceth first-level label corresponding to cle, the value range of ce is 1 to he, he is the number of first-level labels corresponding to cle; num is the value of cle ra The number of corresponding thought chain example texts.

[0135] In this embodiment, bas ce,me Includes several thought chain sample texts, bas ce,me The corresponding labels in each thought chain example text are the meth second-level labels included in the ceth first-level labels of cle, but bas ce,meThe judgment identifiers corresponding to the included different thought chain example texts may be the same or different, bas ce,me The judgment identifier corresponding to the included thought chain example text is used to indicate whether the label corresponding to the thought chain example text is the label of the target text corresponding to the thought chain example text. For example, if bas ce,me the judgment identifier corresponding to the included thought chain example text is correct or yes, it indicates that the label corresponding to the thought chain example text is the label of the target text corresponding to the thought chain example text; if bas ce,me the judgment identifier corresponding to the included thought chain example text is wrong or no, it indicates that the label corresponding to the thought chain example text is not the label of the target text corresponding to the thought chain example text.

[0136] Optionally, L220 includes:

[0137] L221, if bas ce matches l ab 1 ra then enter L222.

[0138] In this embodiment, if bas ce the corresponding first-level label matches l ab 1 ra then it is judged that bas ce matches l ab 1 ra otherwise, it is judged that bas ce does not match l ab 1 ra

[0139] L222, if te = num, then obtain 1 thought chain example text from each bas ce,me to construct mod ra ; if te > num, then enter L223; if te < num, then enter L225.

[0140] L223, obtain the priority of each bas ce,me

[0141] In this embodiment, the priority of each bas ce,me is known. Optionally, the more the number of thought chain example texts included in bas ce,me the higher the priority of bas ce,me

[0142] L224, obtain 1 thought chain example text from each target sub-library to construct mod​​​ra , the target sub-library is bas ce The top num thought chain example text sub-libraries with the highest priority.

[0143] L225, from every bas ce,me Get 1 thought chain example text to build the initial text.

[0144] L226, obtain the text deviation quantity Δq, Δq=num-te;

[0145] L227, from bas ce The remaining thought chain example texts are obtained from Δq thought chain example texts and appended to the initial text, and the appended initial text is determined as mod ra bas ce The remaining thought chain example text is bas ce Example texts of thought chains other than the initial text.

[0146] Based on L221-L227, this embodiment is from l ab 1 ra Matching bas ce num thought chain example texts were obtained from the code and mod ra Automatic construction, eliminating the need to manually build mods ra Moreover, this embodiment takes as much as possible from l ab 1 ra Matching bas ce Get the thought chain sample text of different secondary tags in order to make the mod ra Includes example text of thought chains corresponding to various secondary tags, which is helpful for including mod later ra Prompt words included ra When input into the trained large language model, the large language model can more comprehensively learn the thinking of judging whether the secondary label is the secondary label of the text, so that the trained large language model can give a more accurate judgment result and improve the accuracy of cleaning.

[0147] L230, according to mod ra cle ra andcle ra Secondary tags l ab 2 ra Build cle ra Corresponding prompt word pro ra .

[0148] Based on L210-L230, this embodiment can automatically build prora , which eliminates the process of manually constructing prompt words, improves the efficiency of constructing prompt words, and reduces the manual resources occupied in the text cleaning process.

[0149] As a preferred specific implementation, the process of obtaining num includes:

[0150] L201, get the preset thinking chain sample text library bas l ab 1 ra Matching bas ce Number of thought chain example texts included pn ra .

[0151] L202, get the number of initial thought chain example texts cn ra ,cn ra =f(min(num max ,max(num min ,num avg ×pn ra / pn avg ))), num max The maximum number of preset thought chain sample texts, num max =tok max / (min(tok avg,1 ,tok avg,2 ,…,tok avg,ce ,…,tok avg,he )), tok max is the total number of characters in the preset thought chain sample text, tok avg,ce For bas ce The average number of characters in the example text of the thought chain included; num min The minimum number of preset thought chain sample texts, num min =tok max / (max(tok avg,1 ,tok avg,2 ,…,tok avg,ce ,…,tok avg,he )); num avg is the average number of preset thought chain sample texts, num avg =tok max / (avg(tok avg,1 ,tok avg,2 ,…,tok avg,ce ,…,tok avg,he )); pn avgis the preset threshold value of the number of thought chain example texts; max() is the maximum value, min() is the minimum value, avg() is the average value, and f() is the integer.

[0152] In this embodiment, the average number of example thinking chain texts included in each example thinking chain text sub-library in bas is determined as the preset number threshold of example thinking chain texts.

[0153] L203, if cn ra Greater than or equal to l ab 1 ra The number of secondary tags included, then cn ra Determined as num; if cn ra Less than l ab 1 ra The number of secondary tags included will be l ab 1 ra The number of secondary tags included is identified as num.

[0154] In this embodiment, if cn ra Less than l ab 1 ra The number of secondary tags included is prioritized from l ab 1 ra Matching bas ce Select or reconstruct a thought chain example text composition mod containing a small number of words ra To enable the mod ra The number of characters included does not exceed tok max That's it.

[0155] Based on L201-L203, this embodiment can achieve the purpose of setting different thinking chain example texts according to the text to be cleaned. Specifically, considering the character number limit corresponding to the prompt word, the maximum number, minimum number and average number of thinking chain example texts are used as reference factors. It can also set a larger number of thinking chain example texts when the thinking chain example texts corresponding to the first-level label of the text to be cleaned include a large number of thinking chain example texts, so as to improve the diversity of the set thinking chain example texts; considering the requirements of comprehensive learning and reasoning of the trained large language model, the number of set thinking chain example texts meets the requirement of being greater than or equal to the number of second-level labels included in the first-level label of the text to be cleaned, so that the trained large language model can learn comprehensive label judgment thinking, thereby improving the accuracy of judgment.

[0156] Embodiment six:

[0157] In the first embodiment, the label of the target sampled text may be wrong. In order to obtain a correct and more comprehensive label for each text in the target sampled text, the text processing method based on sampling ratio in this embodiment further includes:

[0158] F100, get the text tex of the label to be obtained, tex=(tex1,tex2,…,tex ac ,…,tex ae ),tex ac is the text of the acth label to be obtained, the value range of ac is 1 to ae, ae is the number of texts of the label to be obtained; each tex ac Corresponding to several original labels.

[0159] In this embodiment, the target sample text is determined as tex, and ae is the number of the target sample text.

[0160] In this embodiment, each tex ac The number of corresponding original labels is greater than or equal to 1, and the original label may be tex ac The corresponding correct label may also be tex ac The corresponding wrong label, and the proportion of correct labels in the original labels of all texts corresponding to tex is greater than the proportion of wrong labels in the original labels; as a specific implementation method, each tex ac The corresponding original labels are manually annotated labels.

[0161] F200, according to the text of the first label to be obtained in tex 1 Get the trained first label inference model mode1, mode1=(mode 1,1 ,mode 1,2 ,…,mode 1,av ,…,mode 1,ax ), mode 1,av According to tex 1 Get the avth trained first label inference model, where av ranges from 1 to ax, and ax is the value of tex 1 The number of trained first-label inference models obtained; ax ≥ 2.

[0162] In this embodiment, the text to be obtained in the tex tag is divided into two parts, one of which is tex 1 , the other part is tex 2 ; optional, tex 1 The number of texts to be labeled in is 1 / 2 of the number of texts to be labeled in tex. 2The number of texts with tags to be obtained in the Chinese text is also 1 / 2 of the number of texts with tags to be obtained in the tex text.

[0163] In this embodiment, each mode 1,av The training samples come from tex 1 , and the training samples corresponding to different modes 1,av are different; as a specific implementation, F200 includes:

[0164] F210, which divides tex 1 to obtain the divided text subsets zc1, zc1 = (zc 1,1 , zc 1,2 , …, zc 1,av , …, zc 1,ax ), where zc 1,av is the ax-th text subset obtained by dividing tex 1 .

[0165] Optionally, tex 1 is evenly divided into ax text subsets.

[0166] F220, initializes the model variable ha to 1.

[0167] F230, selects ax - 1 text subsets from zc1 except zc 1,ha as the ha-th training sample.

[0168] F240, uses the ha-th training sample and the corresponding original tags to train the ha-th first label inference model, and obtains the ha-th trained first label inference model mode 1,ha .

[0169] F250, if ha < ax, then ha = ha + 1, and repeats F230 - F240 until ha = ax.

[0170] Based on F210 - F250, when ha is 1, executing F230 - F240 can obtain the 1st trained first label inference model mode 1,1 ; when ha is 2, executing F230 - F240 can obtain the 2nd trained first label inference model mode 1,2 , and so on. When ha is ax, executing F230 - F240 can obtain the ax-th trained first label inference model mode 1,ax ; thus, ax trained first label inference models can be obtained after executing F210 - F250.

[0171] F300, obtains the text tex of the second tags to be obtained in tex according to mode12 The inference label qa of the text to be labeled 2 ,qa 2 =(qa 2 1,qa 2 2,…,qa 2 er ,…,qa 2 zn ), qa 2 er For tex 2 The inference label of the erth text to be labeled, er ranges from 1 to zn, zn is tex 2 The number of texts to be labeled, qa 2 er =(qa 2 er,1 ,qa 2 er,2 ,…,qa 2 er,av ,…,qa 2 er,ax ), qa 2 er,av According to mode 1,av Get tex 2 The inference label of the text to be labeled in erth; tex 2 Remove tex from tex 1 The text of the tag to be obtained outside the qa 2 er,av Includes several inference tags.

[0172] In this embodiment, tex 2 The text of the erth label to be obtained in is used as the input of the first trained first label inference model, and mode 1,1 Inferred tex 2 The inference label qa of the erth text to be labeled 2 er,1 ; 2 The text of the erth label to be obtained is used as the input of the second trained first label inference model, and mode 1,2 Inferred tex 2 The inference label qa of the erth text to be labeled 2 er,2 ; And so on, tex 2 The text of the erth label to be obtained is used as the input of the axth trained first label inference model, and mode can be obtained. 1,ax Inferred tex 2The inference label qa of the erth text to be labeled 2 er,ax .

[0173] F400, traverse qa 2 , for qa 2 er Filter and get the filtered inference label nqa 2 er ;nqa 2 er =(nqa 2 er,1 ,nqa 2 er,2 ,…,nqa 2 er,bv ,…,nqa 2 er,bx ), nqa 2 er,bv For qa 2 er The bvth inference label remaining after screening, the value range of bv is 1 to bx, bx is the value of qa 2 er The number of inference labels remaining after filtering.

[0174] In this embodiment, qa 2 er The purpose of screening is to 2 er The wrong labels are filtered out and the correct labels are retained.

[0175] As a first preferred specific implementation, F400 includes:

[0176] F410, get qa 2 Each inference label that appears in qa 2 The number of times it appears in .

[0177] F420, if qa 2 A certain inference label appears in qa 2 If the number of occurrences in is greater than or equal to the preset threshold, qa 2 The inference label that appears in is determined as the inference label retained after screening; otherwise, qa 2 The inference label that appears in is determined to be the inference label that is filtered out after filtering.

[0178] In this embodiment, qa 2 The inference tags whose occurrence times in the list are greater than or equal to the preset threshold are judged as tex 2 The correct label of the text to be labeled in er is qa2 The inference tags whose occurrence times are less than the preset threshold are judged as tex 2 The wrong label of the text of the erth label to be obtained in . Optionally, the preset number of times threshold is ax / 2.

[0179] Based on F410-F420, this embodiment realizes the 2 er Screening, since this embodiment will qa 2 The inference labels that appear more often in qa are judged as correct labels, while 2 The inference labels that appear more often in the above are consistent inference results of a large number of trained first label inference models. Therefore, the nqa obtained based on F410-F420 in this embodiment is 2 er The label is tex 2 The probability of the correct label of the erth text to be labeled is higher, which improves the accuracy of the label of the text finally obtained.

[0180] As a second preferred specific implementation, F400 includes:

[0181] F401, get qa 2 er The inference tag wqa appears in 2 er , wqa 2 er =(wqa 2 er,1 ,wqa 2 er,2 ,…,wqa 2 er,cv ,…,wqa 2 er,cx ), wqa 2 er,cv For QA 2 er The cvth inference label appears in , the value range of cv is 1 to cx, and cx is qa 2 er The number of inference labels that appear in .

[0182] In this embodiment, different wqa 2 er,cv corresponds to different inference labels; since different trained first-label inference models may infer the same or different inference labels, cx is less than or equal to qa 2 er Each qa in 2 er,av The sum of the number of inference labels included.

[0183] F402, traverse wqa 2 er , use the trained large language model to judge wqa 2 er,cv Is it tex 2 The label of the text of the erth label to be obtained.

[0184] Those skilled in the art know that any large language model in the prior art falls within the protection scope of the present invention.

[0185] Preferably, the cue words based on the thought chain enable the trained large language model to judge wqa 2 er,cv Is it tex 2 The label of the erth text to be labeled is included in the prompt word, and several example texts of the thought chain related to the content to be judged are included. Therefore, the trained large language model can be used to better learn the thinking of judging whether the label is correct and improve the accuracy of the judgment result.

[0186] F403, if yes, then wqa 2 er,cv Append to the preset erth tag sequence; otherwise, wqa is not appended. 2 er,cv Append to the preset erth label sequence; the preset erth label sequence is initialized to an empty sequence.

[0187] F404, determine the preset erth label sequence as nqa 2 er .

[0188] Based on F401-F404, this embodiment implements qa based on the trained large language model 2 er The screening of NQA is improved through the trained large language model 2 er The accuracy of the inference labels in .

[0189] F500, will nqa 2 er Determine as tex 2 The label of the erth text to be judged.

[0190] In this embodiment, each text in the text tex to be labeled corresponds to an original label, and a part of the text tex in tex is used. 1 A plurality of trained first label inference models can be obtained from the corresponding original labels, and another part of the text tex in tex can be obtained according to the plurality of trained first label inference models.2 inference labels; since the present embodiment uses multiple trained first label inference models to obtain inference labels for texts, different trained first label inference models may obtain different inference labels for texts, and therefore the inference labels for texts obtained by the present embodiment are relatively comprehensive; moreover, the present embodiment also screens the inference labels obtained according to multiple trained first label inference models to select the correct labels from the relatively comprehensive inference labels; thus, the present embodiment achieves the purpose of obtaining correct and relatively comprehensive labels for texts.

[0191] As an optional specific implementation, ax is an empirical value, for example, 3≤ax≤5.

[0192] As a preferred specific implementation, the method for obtaining ax includes:

[0193] F201, get tex 1 The number of texts to be included in the tags to be obtained 1 .

[0194] F202, get the initial number of divisions hua0, hua0=floor(shu 1 / shu0), shu0 is the preset text quantity threshold, and floor() is rounding down.

[0195] In this embodiment, shu0 is an empirical value, which corresponds to the minimum number of training samples required to train a first label inference model into a model with an accuracy greater than or equal to a preset accuracy threshold.

[0196] F203, obtain the first division quantity hua1, hua1=min(max(hua0,hua min ),hua max ), min() is the minimum value, max() is the maximum value, min is the preset minimum number of divisions, max The preset maximum number of divisions.

[0197] In this embodiment, hua min and hua max All are experience values.

[0198] F204, from tex 1 The text of the to-be-acquired label with a ratio of (hua1-1) / hua1 and the corresponding original label are used to train the test label inference model to obtain a trained test label inference model.

[0199] F205, if the accuracy of the trained test label inference model is greater than or equal to a preset accuracy threshold, hua1 is determined as ax.

[0200] In this embodiment, if the accuracy of the trained test label inference model is less than the preset accuracy threshold, then 1 The text of the to-be-acquired label with the proportion of hua1 / (hua1+1) and the corresponding original label are used to train the test label inference model to obtain a trained test label inference model. If the accuracy of the trained test label inference model is greater than or equal to the preset accuracy threshold, hua1+1 is determined as ax; otherwise, tex 1 The text of the to-be-acquired label with a proportion of (hua1+1) / (hua1+2) and the corresponding original label are used to train the test label inference model to obtain a trained test label inference model. If the accuracy of the trained test label inference model is greater than or equal to the preset accuracy threshold, hua1+2 is determined as ax; otherwise, tex 1 The text of the to-be-obtained label with a proportion of (hua1+2) / (hua1+3) and the corresponding original label are used to train the test label inference model to obtain a trained test label inference model; and so on, until the accuracy of the trained test label inference model is greater than or equal to the preset accuracy threshold, the corresponding number of divisions is determined as ax.

[0201] Based on the ax obtained from F201-F205, tex 1 Divide, you can make tex 1 The number of texts corresponding to the proportion of (hua1-1) / hua1 is relatively large, which improves the subsequent tex-based 1 The accuracy of the first label inference model trained on text with a ratio of (hua1-1) / hua1 is conducive to improving the accuracy of the final text label.

[0202] As an optional specific implementation, after F500, the method for obtaining the text label further includes:

[0203] F600, traverse tex 2 The original label corresponding to the text of the erth label to be obtained in tex 2 The original label corresponding to the text of the erth label to be obtained in nqa 2 er If it exists in 2 The original label corresponding to the text of the erth label to be obtained is correct; otherwise, judge tex 2 The original label corresponding to the text of the erth label to be obtained is wrong.

[0204] Therefore, this embodiment is based on tex 2 The label nqa corresponding to the text of the erth label to be obtained 2 er Implemented the tex 2 Determine the original label corresponding to the text of the erth label to be obtained.

[0205] In this embodiment, tex is obtained 1 The method of obtaining the label of each text to be labeled is the same as the above method of obtaining the text 2 The method of obtaining the label of each text to be labeled is similar, except that: 1 In the process of labeling each text to be labeled, according to the tex 2 Get the trained second label inference model mode2, which includes several 2 Obtain the trained second label inference model; obtain tex according to mode2 1 The inference label qa of the text to be labeled 1 ,qa 1 Includes all tex obtained by the trained second label inference model 1 The inference label of each text to be labeled; traverse qa 1 , for qa 1 er Filter and get the filtered inference label nqa 1 er ; will nqa 1 er Determine as tex 1 The label of the erth text to be judged in . Get tex 1 The specific process of obtaining the label of each text to be labeled is the same as the above-mentioned process of obtaining the text label of the tex 2 The specific process of labeling each text to be labeled is similar and will not be repeated here.

[0206] Embodiment seven:

[0207] In order to obtain a named entity recognition model capable of performing entity annotation of multiple preset types on the target sampled text, the text processing method based on sampling ratio of this embodiment further includes the following steps:

[0208] D100, obtain the training text set data of the trained named entity recognition model, data={data1,data2,…,data se ,…,data st}, data seis the se-th training text set of the trained named entity recognition model, data se The invention comprises a plurality of training texts, wherein the value range of se is 1 to st, and st is the number of training text sets of the trained named entity recognition model; the trained named entity recognition model is a model obtained by training using data, and the trained named entity recognition model is used to recognize entities of a preset type ty in the input text and mark the corresponding labels; ty=(ty1, ty2, ..., ty mt ,…,ty ht ),ty mt It is the mtth preset type, the value range of mt is 1 to ht, and ht is the number of preset types.

[0209] In this embodiment, the trained named entity recognition model is obtained by training using the st training text sets included in the data. The purpose of training the named entity recognition model using the st training text sets included in the data is to enable the trained named entity recognition model to have the function of recognizing ht preset types of entities.

[0210] In this embodiment, when the named entity recognition model is trained using the st training text sets included in data, at least one entity of a preset type is annotated when each training text set is annotated, and entities other than ht entities of the preset types are not annotated, and each entity of the ht preset types has been annotated by at least one training text set. Thus, the trained named entity recognition model has and only has the function of recognizing entities of ht preset types.

[0211] D200, traverse data, if using data se When training the named entity recognition model, se The entity of the mtth preset type is labeled, then the data se The proportion of positive samples of the mt-th preset type of entity in data is determined as data se The mtth weight qz se,mt ; Otherwise, set qz se,mt is 0.

[0212] In this embodiment, when the named entity recognition model is trained using the st training text sets included in data, not all entities of ht preset types are annotated when each training text set is annotated. For example, st=2, ht=4, when the named entity recognition model is trained using the first training text set and the second training text set, only entities of the first preset type and the second preset type are annotated when the first training text set is annotated, and only entities of the second preset type, the third preset type, and the fourth preset type are annotated when the second training text set is annotated.

[0213] In this embodiment, data se The proportion of positive samples of the mt-th preset type of entity in data is se The number of entities marked as entities of the mtth preset type in data se The ratio of the number of entities in se When training the named entity recognition model, se The entity of the mtth preset type is labeled, then the data se The value of the positive sample ratio of the mt-th preset type of entity in is assigned to qz se,mt ; If you use data se When training the named entity recognition model, there is no data se To label the entity of the mtth preset type, then qz se,mt Set to 0.

[0214] D300, obtain the positive sample proportion cb of ht preset types of entities in the test text set, cb=(cb1,cb2,…,cb mt ,…,cb ht ), cb mt is the proportion of positive samples of the mt-th preset type of entity in the test text set.

[0215] In this embodiment, the positive sample ratio of the mt-th preset type of entity in the test text set is the ratio of the number of entities in the test text set that are marked as entities of the mt-th preset type to the number of entities in the test text set.

[0216] In this embodiment, the proportion of positive samples of the mt-th preset type of entity in the test text set can represent the proportion of positive samples of the mt-th preset type of entity in the text set to be subsequently recognized by the trained named entity recognition (for example, the target sampling text). Optionally, the test text set is a text extracted from the text set to be subsequently recognized by the trained named entity recognition, and the ht preset types of entities in the test text set are annotated by manual annotation methods, and the positive sample proportions of the ht preset types of entities are obtained.

[0217] D400, obtain the positive sample ratio zb of ht preset types of entities in the newly added text set, zb=(zb1,zb2,…,zb mt ,…,zb ht ), zb mt It is the proportion of positive samples of the mt-th preset type of entity in the newly added text set.

[0218] In this embodiment, the newly added text set is a text obtained in order to enable the trained text set to have the function of identifying entities of the newly added preset type, and the positive sample ratio of the mt-th preset type entity in the newly added text set is the ratio of the number of entities marked as entities of the mt-th preset type in the newly added text set to the number of entities in the newly added text set. As an optional specific implementation, D400 includes:

[0219] D410, obtaining a preset proportion of text from the newly added text set to form a first extracted text set cq.

[0220] Optionally, a random extraction method is used to obtain a preset proportion of texts from the newly added text set to form a first extracted text set cq, where the preset proportion is an empirical value.

[0221] D420, using the trained named entity recognition model to perform ht preset types of entity recognition on each text in cq, and obtain labeling results of ht preset types of entities corresponding to each text in cq.

[0222] In this embodiment, each text in cq is used as the input of the trained named entity recognition model to obtain the corresponding labeling results of ht preset types of entities.

[0223] D430, according to the annotation result of each text in cq corresponding to the entity of the mt-th preset type, it is determined as zb mt .

[0224] In this embodiment, zb mt =∑ fs fe=1 en fe,mt / (∑ fs fe=1 q fe ),zh fe,mt is the number of entities annotated as entities of the mtth preset type in the feth text in cq, qh fe It is the number of entities included in the fe-th text in cq. The value range of fe is 1 to fs, and fs is the number of texts included in cq.

[0225] D500, according to qz se,mt、cb and zb determine the optimal amount of text to be obtained from the newly added text set for training the trained named entity recognition model.

[0226] The trained named entity recognition model in this embodiment is obtained by training based on the training text set data, which has the function of identifying ht preset types of entities; the newly added text set is the text obtained in this embodiment in order to enable the trained text set to have the function of identifying the newly added preset type of entities. This embodiment determines the optimal number of texts to be obtained from the newly added text set for training the trained named entity recognition model based on the positive sample ratio of ht preset types of entities in the test text set, the positive sample ratio zb of ht preset types of entities in the newly added text set, and the weight of each training text set corresponding to each preset type in data, wherein the newly added text set has the optimal number of texts to be obtained from the newly added text set for training the trained named entity recognition model. The proportion of positive samples of ht preset types of entities determines the extent of the impact of using the newly added text set to train the trained named entity recognition model on the recognition accuracy of the original ht preset types of entities. The proportion of positive samples of ht preset types of entities in the test text set and the weight of each training text set corresponding to each preset type in the data reflect the recognition accuracy of the trained named entity recognition model for the original ht preset types of entities. Therefore, this embodiment can achieve the purpose of taking into account the recognition accuracy of the re-trained named entity recognition model for the original ht preset types of entities and the recognition accuracy of the newly added preset types of entities.

[0227] As a preferred embodiment, D500 includes:

[0228] D510, get the loss dif of the newly added text, dif=(∑ ht mt=1 |cb mt -((∑ st se=1 qz se,mt ×mu se )+zb mt ×zq) / (su se + zq)|)+ke / zq,mu se For data se The amount of text included, su se is the corresponding qz in data se,mt The sum of the number of texts included in the training text set that is not 0; ke is a preset adjustment coefficient, ke>0, and zq is the number of texts to be obtained from the newly added text set for training the trained named entity recognition model.

[0229] In this embodiment, ke is an empirical value, optional, 0 <ke<1。

[0230] D520, determining the value of zq when dif takes the minimum value as the optimal number of texts obtained from the newly added text set for training the trained named entity recognition model.

[0231] Based on the above preferred specific implementation manner, the loss of newly added texts corresponding to the optimal number of texts obtained is minimal. Obtaining texts from the newly added text set according to the optimal number of texts as training samples for the trained named entity recognition model can make the proportion of positive samples of entities of each preset type used for the named entity recognition model closer to the proportion of positive samples of entities of the corresponding preset type in the test sample, so that the named entity recognition model trained with the texts of the optimal number of texts in the newly added text set has a higher recognition accuracy rate for entities of the original preset type; obtaining texts from the newly added text set according to the optimal number of texts as training samples for the trained named entity recognition model can also make the number of texts obtained from the newly added text set larger, so that the named entity recognition model trained with the texts of the optimal number of texts in the newly added text set has a higher recognition accuracy rate for entities of the newly added preset type; therefore, the optimal number of texts obtained based on this preferred specific implementation manner can achieve the purpose of taking into account the recognition accuracy rate of the named entity recognition model for both entities of the original preset type and entities of the newly added preset type.

[0232] As an optional specific implementation, after D500, when the computer readable instructions are executed by the processor, the following steps are further implemented:

[0233] D600, fk texts are obtained from the newly added text set to form a text set to be annotated, where fk is the optimal number of texts.

[0234] D700, obtaining the annotation result res corresponding to the text set to be annotated, res including the annotation result ty corresponding to the text set to be annotated and the annotation result of the newly added preset type corresponding to the text set to be annotated; the newly added preset type and any ty mt Are different.

[0235] In this embodiment, the annotation result of ty corresponding to the to-be-annotated text set includes each ty corresponding to the to-be-annotated text set. mt The labeling results.

[0236] D800: train the trained named entity recognition model using the to-be-annotated text set and res.

[0237] Based on D600-D800, the obtained trained named entity recognition model not only has the function of identifying entities of newly added preset types, but also enables the trained named entity recognition model to reduce or improve the recognition accuracy of entities of the original preset types, thereby improving the recognition function of the trained named entity recognition model as a whole.

[0238] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A text processing method based on sampling ratio, characterized in that: The following steps are involved: E100, get the i-th type of text to be sampled a i , a i The number of texts to be sampled is q i , a i Included q i The labels of the texts to be sampled are the same, but different i The labels of the texts to be sampled are different, and the value range of i is 1 to n, where n is the number of categories of the texts to be sampled; E200, get the number of samples s of the i-th category i ; E300, get the sampling ratio r of the i-th category i , r i =s i / q i ; E400, if r i <r0, then enter E500; r0 is a preset sampling ratio threshold; E500, according to a i The text similarity between the texts to be sampled includes a i Perform clustering and obtain the clustering result G i , G i Includes m clusters, m is the number of a i The number of clusters obtained by clustering; E600, Get G i The number of texts to be sampled in the smallest cluster s i,min ; E700, if s i,min ≥f(s i / m), then enter E800; f( ) is the rounding function; E800, traverse G i , from each g i,j Get f(s i / m) texts to be sampled constitute the initial sampled text of the i-th category; g i,j For a i The jth cluster obtained by clustering, the value of j ranges from 1 to m; After E200 and before E300, the method further includes: E201, obtain a text set A to be sampled, A includes n types of texts to be sampled, and the i-th type of text to be sampled included in A is a i ; E202, obtaining the sum w of the number of texts to be sampled included in the first u categories with the largest number of texts to be sampled in A, where u is a preset category number threshold; E203, if w / (∑ n i=1 q i )≥η, then enter E300; η is the preset ratio threshold.

2. The text processing method based on sampling ratio according to claim 1, characterized in that: E700 also includes: If s i,min <f(s i / m), then enter E710; E710, traverse G i , from g i,j Get i,min The texts to be sampled constitute the first sampled text of the i-th category; E720, obtain the first remaining sampling quantity Δε1, Δε1=s i -m×s i,min ; E730, obtain the clustering result G after the first sampling 1 i , G 1 i Including m clusters, G 1 i The jth cluster included is g 1 i,j , g 1 i,j From g i,j Get i,min The clusters are composed of the remaining texts after the texts to be sampled; E740, Get G 1 i The number of non-empty clusters in δ; E750, if Δε1=δ, then from G 1 i Obtain one to-be-sampled text from each non-empty cluster and append it to the first sampled text of the i-th category, and determine the first sampled text of the i-th category after the appending as the target sampled text of the i-th category.

3. The text processing method based on sampling ratio according to claim 1, characterized in that: After E800, the method further includes: E900, if m×f(s i / m)=s i , then the initial sampled text of the i-th category is determined as the target sampled text of the i-th category; if m×f(s i / m)>s i , then randomly delete m×f(s i / m)-s i text, the initial sampled text of the i-th category after deletion is determined as the target sampled text of the i-th category, and the deleted m×f(s i / m)-s i The text belongs to a i m×f(s i / m)-s i clusters; if m×f(s i / m) i , then from a i Randomly get s from the remaining text i -m×f(s i / m) to-be-sampled texts are added to the initial sampled text of the i-th category, and the added initial sampled text of the i-th category is determined as the target sampled text of the i-th category; a i The remaining text is a i The text except the initial sampled text of the i-th category.​ 4. The text processing method based on sampling ratio according to claim 2, characterized in that: E750 also includes: If Δε1<δ, then from G 1 i Obtain one to-be-sampled text from each target cluster of G and append it to the first sampled text of the i-th category, and determine the first sampled text of the i-th category after appending as the target sampled text of the i-th category; the target cluster is 1 i Δε1 non-empty clusters are randomly selected from the δ non-empty clusters.

5. The text processing method based on sampling ratio according to claim 2, characterized in that: E750 also includes: if Δε1>δ, enter E751; E751, Get G 1 i The number of texts to be sampled in the smallest non-empty cluster s 1 i,min ; E752, if s 1 i,min ≥f(Δε1 / δ), then enter E753; E753, from G 1 i Obtain f(Δε1 / δ) texts to be sampled from each non-empty cluster and append them to the first sampled text of the i-th category; E754, if f(Δε1 / δ)×δ=Δε1, the first sample text of the i-th category after appending is determined as the target sample text of the i-th category.

6. The text processing method based on sampling ratio according to claim 1, characterized in that: E400 also includes: If r i ≥r0, then enter E410; E410, if q i ≥s i , then from a i Randomly obtain s i The sampled texts constitute the target sampled text of the i-th category; otherwise, a i Determine the target sample text for the i-th category.

Images