Text data processing method and device, equipment and storage medium

Abstract

The present disclosure provides a text data processing method, device and equipment and storage medium. The present disclosure obtains a plurality of perplexity values corresponding to a plurality of sample texts respectively by performing inference on each of the plurality of sample texts through a plurality of pre-trained models. Further, a confidence interval is determined according to the plurality of perplexity values corresponding to the plurality of sample texts respectively, so as to determine an abnormal sample in the plurality of sample texts according to the confidence interval. After clustering processing of the plurality of sample texts is performed to obtain a plurality of clustering clusters, a clustering cluster containing an abnormal sample in the plurality of clustering clusters is removed, that is, all sample texts contained in the clustering cluster are removed. These sample texts are usually low-quality or irrelevant text data. The remaining sample texts constitute a high-quality data set, thereby improving the accuracy of subsequent processing or calculation processes that rely on the data set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a text data processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the continuous development of computer technology, in order to improve the accuracy of text data analysis, the text data can be preprocessed, such as cleaning the text data to improve the quality of the text data.

[0003] However, the prior art cannot fully eliminate low-quality text data when cleaning the text data, resulting in low-quality text data after cleaning, and thus the subsequent processing process or calculation process relying on the cleaned text data is not accurate enough. SUMMARY

[0004] According to an aspect of the present application, a text data processing method is provided, the method comprising:

[0005] obtaining a plurality of sample texts;

[0006] for each sample text in the plurality of sample texts, performing inference on the sample text by a plurality of pre-trained models to obtain a plurality of perplexity values corresponding to the sample text, the plurality of models and the plurality of perplexity values corresponding one-to-one;

[0007] determining a confidence interval according to the plurality of perplexity values corresponding to the plurality of sample texts respectively;

[0008] performing clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters;

[0009] removing a clustering cluster containing an abnormal sample from the plurality of clustering clusters, the abnormal sample being a sample text outside the confidence interval in the plurality of sample texts.

[0010] According to another aspect of the present application, a text data processing device is provided, the device comprising:

[0011] an obtaining module configured to obtain a plurality of sample texts;

[0012] an inference module configured to, for each sample text in the plurality of sample texts, perform inference on the sample text by a plurality of pre-trained models to obtain a plurality of perplexity values corresponding to the sample text, the plurality of models and the plurality of perplexity values corresponding one-to-one;

[0013] a determination module configured to determine a confidence interval according to the plurality of perplexity values corresponding to the plurality of sample texts respectively;

[0014] a clustering module, configured to perform clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters;

[0015] a removing module, configured to remove a clustering cluster containing an abnormal sample from the plurality of clustering clusters, the abnormal sample being a sample text outside the confidence interval in the plurality of sample texts.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0017] a processor; and

[0018] a memory storing a program,

[0019] wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to the first aspect.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause the computer to perform the method according to the first aspect.

[0021] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the first aspect.

[0022] One or more technical solutions provided in the embodiments of the present application can sufficiently eliminate low-quality sample texts, so that the remaining sample texts constitute a high-quality data set, thereby improving the accuracy of subsequent processing or calculation processes that rely on the data set. BRIEF DESCRIPTION OF DRAWINGS

[0023] In the following description of exemplary embodiments in conjunction with the accompanying drawings, more details, features and advantages of the present disclosure are disclosed, in which:

[0024] Figure 1 a flowchart of a text data processing method provided by the embodiments of the present disclosure;

[0025] Figure 2 a schematic diagram of an application scenario to which the text data processing method provided by the embodiments of the present disclosure is applicable;

[0026] Figure 3 a flowchart of a text data processing method provided by the embodiments of the present disclosure;

[0027] Figure 4 a schematic diagram of an application scenario to which the text data processing method provided by the embodiments of the present disclosure is applicable;

[0028] Figure 5 a flowchart of a text data processing method provided by the embodiments of the present disclosure;

[0029] Figure 6 A flow chart of a text data processing method provided by an embodiment of the present disclosure is shown in FIG. 8.

[0030] Figure 7 A structural schematic diagram of a text data processing apparatus provided by an embodiment of the present disclosure is shown in FIG. 9.

[0031] Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown in FIG. 10. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail by referring to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather the embodiments are provided so as to more completely and thoroughly understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that the various steps in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term “comprising” and variations thereof as used herein are open-ended, that is, “including but not limited to”. The term “based on” is “based, at least in part, on”. The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”. Related definitions are given throughout the description. It is noted that the concepts of “first”, “second”, etc. mentioned in the present disclosure are only used to distinguish different apparatuses, modules or units, and are not intended to limit the order or interdependence of the functions performed by these apparatuses, modules or units.

[0035] It is noted that the modification of “one”, “multiple” mentioned in the present disclosure is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as “one or more”.

[0036] The names of the messages or information exchanged between the multiple apparatuses in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0037] Generally, in order to improve the accuracy of text data analysis, the text data can be pre-processed, for example, by cleaning the text data to improve the quality of the text data. However, the prior art cannot sufficiently eliminate low-quality text data when cleaning the text data, resulting in low-quality text data after cleaning, which leads to inaccurate subsequent processing or calculation processes that rely on the cleaned text data. To solve this problem, the present disclosure provides a text data processing method, which will be described in detail below in conjunction with specific embodiments.

[0038] Figure 1 A flowchart of a text data processing method provided by the present disclosure is shown. The method can be performed by a text data processing device, which can be implemented in software and / or hardware, and can be configured in an electronic device such as a server, a server cluster, or a terminal. The terminal can be a desktop computer, a tablet computer, a mobile phone, etc. The server cluster can be a group of servers working together to provide the same service, which appears to the client as a single server. The server cluster can use multiple computers for parallel computing to achieve higher computing speed, or use multiple computers for backup, so that the entire system can still function normally even if one machine fails. That is, in the present disclosure, the text data processing process can be performed by a server, a server cluster, or a terminal.

[0039] For example, the text data processing method described in the present embodiment is performed by a server 21 as shown in Figure 2 The terminal 22 can communicate with the server 21. Specifically, the terminal 22 can provide the server 21 with information required for text data processing, such as sample text. It can be understood that the server 21 is not limited to obtaining sample text from the terminal 22. For example, in other embodiments, the server 21 can obtain sample text from other servers or other terminals. Alternatively, the server 21 can have sample text pre-stored locally. In addition, the present disclosure does not limit the specific form of sample text. For example, the sample text can be Chinese text, English text, etc. The sample text can be at least one sentence, at least one paragraph, or at least one article.

[0040] The text data processing method will be described in detail below. As shown in Figure 2 The specific steps of the method are as follows: Figure 1

[0041] S101, obtaining a plurality of sample texts.

[0042] ​Specifically, the plurality of sample texts obtained by the server 21 can be obtained from the same device. For example, the server 21 obtains the plurality of sample texts from the terminal 22, or the server 21 obtains the plurality of sample texts from other servers or other terminals. Alternatively, the server 21 can obtain the plurality of sample texts locally.

[0043] In other embodiments, the plurality of sample texts obtained by the server 21 can be obtained from a plurality of different devices. It can be understood that when the plurality of sample texts obtained by the server 21 are obtained from a plurality of different devices, the plurality of different devices are not limited by the embodiments of the present disclosure. For example, some sample texts are obtained from the terminal 22, some sample texts are obtained from other servers or other terminals, and some sample texts are obtained locally from the server 21.

[0044] Specifically, the plurality of sample texts obtained by the server 21 can be sample text 1, sample text 2, …, sample text m as shown in FIG. 1. Figure 3

[0045] S102, for each sample text in the plurality of sample texts, a plurality of pre-trained models are used to infer the sample text respectively, and a plurality of perplexity values corresponding to the sample text are obtained, the plurality of models and the plurality of perplexity values correspond one by one.

[0046] Specifically, the server 21 can store a plurality of pre-trained models locally, such as a plurality of pre-trained large language models (Large Language Model, LLM). The plurality of pre-trained large language models can be large language model 1, large language model 2, …, large language model N as shown in FIG. 1. The server 21 can input each of the sample text 1, sample text 2, …, sample text m into the plurality of large language models, so that the plurality of large language models infer the sample text respectively, and obtain a plurality of perplexity values corresponding to the sample text. For example, when the server 21 inputs the sample text 1 into the plurality of large language models, each large language model in the plurality of large language models independently infers the sample text 1, that is, different large language models infer the sample text 1 without affecting each other. For example, the inference result output by the large language model 1 after inferring the sample text 1 is perplexity 1 as shown in FIG. 1, the inference result output by the large language model 2 after inferring the sample text 1 is perplexity 2 as shown in FIG. 1, and so on. The inference result output by the large language model N after inferring the sample text 1 is perplexity N as shown in FIG. 1. Figure 3 Figure 3 Figure 3 Figure 3 ​​​​The perplexity N is shown. Therefore, the perplexity 1, the perplexity 2, …, the perplexity N are a plurality of perplexity values corresponding to the sample text 1. Similarly, after the server 21 inputs the sample text 2 into the plurality of large language models, a plurality of perplexity values corresponding to the sample text 2 can also be obtained through reasoning. That is, through reasoning, a plurality of perplexity values corresponding to each sample text can be obtained, for example, N perplexity values can be obtained for each sample text. It can be understood that the plurality of perplexity values corresponding to different sample texts can be different, for example, the plurality of perplexity values corresponding to the sample text 1 and the plurality of perplexity values corresponding to the sample text 2 can be different. Or it can be understood that for the same large language model, the perplexity values obtained after the large language model reasons different sample texts can be different, for example, the perplexity value obtained after the large language model 1 reasons the sample text 1 and the perplexity value obtained after the large language model 1 reasons the sample text 2 can be different.

[0047] In addition, in other embodiments, the plurality of large language models pre-trained as described above can also be stored on other servers or terminals. For example, the large language model 1, the large language model 2, …, the large language model N as described above are deployed in the server 41 as shown, at this time, the server 21 can send the sample text 1, the sample text 2, …, the sample text m to the server 41, so that the server 41 reasons each of the sample text 1, the sample text 2, …, the sample text m through the large language model 1, the large language model 2, …, the large language model respectively, thereby obtaining a plurality of perplexity values corresponding to each sample text respectively, the specific process can refer to the process of reasoning each sample text through the plurality of large language models by the server 21 as described above. After reasoning, the server 41 can feed back the plurality of perplexity values corresponding to each sample text respectively to the server 21. That is, in the embodiment of the present disclosure, the server 21 can calculate the plurality of perplexity values corresponding to each sample text respectively locally, or can calculate the plurality of perplexity values corresponding to each sample text respectively through other devices such as the server 41. Figure 4 The server 41 as shown, at this time, the server 21 can send the sample text 1, the sample text 2, …, the sample text m to the server 41, so that the server 41 reasons each of the sample text 1, the sample text 2, …, the sample text m through the large language model 1, the large language model 2, …, the large language model respectively, thereby obtaining a plurality of perplexity values corresponding to each sample text respectively, the specific process can refer to the process of reasoning each sample text through the plurality of large language models by the server 21 as described above. After reasoning, the server 41 can feed back the plurality of perplexity values corresponding to each sample text respectively to the server 21. That is, in the embodiment of the present disclosure, the server 21 can calculate the plurality of perplexity values corresponding to each sample text respectively locally, or can calculate the plurality of perplexity values corresponding to each sample text respectively through other devices such as the server 41.

[0048] S103, determining a confidence interval according to the plurality of perplexity values corresponding to the plurality of sample texts respectively.

[0049] Since the perplexity values corresponding to different sample texts in the sample text 1, the sample text 2, …, and the sample text m can be different, the perplexity values corresponding to the m sample texts have certain randomness. The embodiments of the present disclosure can use statistical principles to statistically analyze the perplexity values corresponding to the m sample texts. For example, the normal distribution is a basic theoretical distribution of statistics, and the normal distribution is the most common distribution in nature. Therefore, when the perplexity values corresponding to the m sample texts conform to the normal distribution, the confidence interval conforming to the normal distribution can be calculated according to the properties of the normal distribution. Through the confidence interval, the abnormal samples in the sample text 1, the sample text 2, …, and the sample text m can be determined, for example, the sample text located outside the confidence interval is recorded as an abnormal sample, and the abnormal sample can be one or more.

[0050] S104, performing clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters.

[0051] For example, the embodiments of the present disclosure can use a density clustering algorithm to perform clustering processing on the sample text 1, the sample text 2, …, and the sample text m, thereby obtaining a plurality of clustering clusters, each clustering cluster including at least one sample text.

[0052] S105, removing the clustering cluster including the abnormal sample in the plurality of clustering clusters, the abnormal sample being the sample text located outside the confidence interval in the plurality of sample texts.

[0053] For example, when any clustering cluster in the plurality of clustering clusters includes an abnormal sample, the clustering cluster can be removed, that is, all sample texts included in the clustering cluster are removed, and these sample texts are usually low-quality or irrelevant text data to the theme.

[0054] In addition, when the abnormal samples are multiple, the multiple abnormal samples can be concentrated in one clustering cluster or distributed in multiple different clustering clusters. When the multiple abnormal samples are concentrated in one clustering cluster, all sample texts included in the clustering cluster can be removed. If the multiple abnormal samples are distributed in multiple different clustering clusters, the multiple different clustering clusters need to be removed, that is, all sample texts included in the multiple different clustering clusters are removed.

[0055] The embodiments of the present disclosure obtain multiple perplexity values corresponding to the multiple sample texts respectively by respectively performing inference on each of the multiple sample texts by using the multiple pre-trained models. Further, the confidence interval is determined according to the multiple perplexity values corresponding to the multiple sample texts respectively, so as to determine the abnormal sample in the multiple sample texts according to the confidence interval. After clustering processing of the multiple sample texts is performed to obtain multiple clustering clusters, the clustering cluster containing the abnormal sample is removed, that is, all sample texts contained in the clustering cluster are removed. These sample texts are usually low-quality or irrelevant to the theme. It can be understood that all sample texts contained in the clustering cluster are not all abnormal samples, for example, some sample texts in the clustering cluster are abnormal samples, and some sample texts are not abnormal samples. However, since these sample texts can be clustered into the same cluster, it indicates that these sample texts are relatively similar. Therefore, by removing all sample texts contained in the clustering cluster, low-quality sample texts can be sufficiently removed, so that the remaining sample texts constitute a high-quality data set, thereby improving the accuracy of a subsequent processing process or a calculation process depending on the data set.

[0056] On the basis of the above-mentioned embodiments, the sample text corresponds to multiple text units, and the perplexity value output by the model for the sample text is calculated according to parameters of the model and a log-likelihood probability of each text unit in the multiple text units, and the log-likelihood probability of the text unit is determined based on other text units before the text unit.

[0057] For example, in the present embodiment, for any sample text as described above, the sample text can be mapped into multiple text units, and each text unit is denoted as a Token. Token generally refers to a basic unit in text, which can be a word, a character, or any other divisible element in text. It is assumed that the sample text 1 as described above corresponds to t Tokens, and the perplexity value output by the large language model 1 for the sample text 1 is denoted as ppl, which is calculated according to parameters of the large language model 1 and a log-likelihood probability of each Token in the t Tokens, and the log-likelihood probability of each Token is determined based on other Tokens before the Token. Specifically, the calculation formula of ppl can be expressed as the following formula (1):

[0058]

[0059] wherein x i represents the i th Token in the t Tokens, x <i represents all Tokens before the i th Token, and x <idoes not include the i-th Token itself, that is, x <i represents the first i-1 Tokens in the t Tokens. θ represents the parameters of the large language model, for example, when the large language model is the large language model 1, θ represents the parameters of the large language model 1, and it can be understood that the parameters of different large language models are different. p θ represents the probability. log p θ (x i |x <i ) represents the logarithmic likelihood probability of the i-th Token based on all Tokens before the i-th Token.

[0060] It can be understood that although the above content takes the perplexity value output by the large language model 1 for the sample text 1 as an example to introduce the application process of the above formula (1), the formula (1) is not limited to this. Specifically, the above formula (1) can be applied to the perplexity value output by any large language model for any sample text, that is, the perplexity value output by any large language model for any sample text can be represented by the above formula (1). In this embodiment, the perplexity value output by any large language model for any sample text can be used to quantify the uncertainty of the large language model for the sample text. Since the parameters of different large language models are different, as Figure 3 shown, the N perplexity values obtained after the N large language models respectively infer the same sample text can be different.

[0061] Optionally, according to the plurality of perplexity values corresponding to the plurality of sample texts, a confidence interval is determined, including the following steps as Figure 5 shown:

[0062] S501, for each sample text in the plurality of sample texts, integrating the plurality of perplexity values corresponding to the sample text into a perplexity vector, and calculating the average value of the elements contained in the perplexity vector.

[0063] For example, for each sample text in the m sample texts as Figure 3 shown, the N perplexity values corresponding to the sample text are integrated into a perplexity vector, and this process corresponds to the perplexity vector generation as Figure 3 shown. For example, the perplexity vector corresponding to the i-th sample text in the m sample texts is denoted as P i , P i can be represented by the following formula (2):

[0064] P i = [p1, p2,..., p N ] (2)

[0065] wherein, pi represents the perplexity value output by the large language model 1 after reasoning on the i-th sample text, p2 represents the perplexity value output by the large language model 2 after reasoning on the i-th sample text, and so on, and pN represents the perplexity value output by the large language model N after reasoning on the i-th sample text. N wherein, pi represents the perplexity value output by the large language model 1 after reasoning on the i-th sample text, p2 represents the perplexity value output by the large language model 2 after reasoning on the i-th sample text, and so on, and pN represents the perplexity value output by the large language model N after reasoning on the i-th sample text.

[0066] It can be understood that for each sample text in the m sample texts, there can be an N-dimensional perplexity vector similar to the formula (2).

[0067] In addition, for each sample text in the m sample texts, the average value of the elements contained in the perplexity vector corresponding to the sample text can be further calculated. For example, the perplexity vector corresponding to each sample text is an N-dimensional vector, that is, the perplexity vector corresponding to each sample text contains N elements. Since the m sample texts correspond to m perplexity vectors, for each perplexity vector in the m perplexity vectors, the average value of the N elements in the perplexity vector is calculated, and m average values are obtained, that is, m sample texts correspond to m average values, and each sample text corresponds to an average value.

[0068] S502, calculating a confidence interval conforming to a normal distribution according to the average values corresponding to the plurality of sample texts.

[0069] For example, according to the m average values corresponding to the m sample texts, a confidence interval conforming to a normal distribution is calculated, which can be a 3sigma (i.e. three standard deviations) confidence interval, which can be expressed as the following formula (3):

[0070] (mu-3delta, mu+3delta) (3)

[0071] wherein, mu represents the expected value of the normal distribution. Delta represents the standard deviation of the normal distribution.

[0072] Optionally, calculating a confidence interval conforming to a normal distribution according to the average values corresponding to the plurality of sample texts comprises: calculating the expected value and the standard deviation of the normal distribution according to the average values corresponding to the plurality of sample texts; and calculating the confidence interval conforming to the normal distribution according to the expected value and the standard deviation of the normal distribution.

[0073] For example, after calculating the m average values corresponding to the m sample texts, the expected value mu and the standard deviation delta of the normal distribution are calculated according to the m average values. Assuming that the m average values are x1, x2,..., xm. The calculation formula of the expected value mu can be expressed as the following formula (4) or formula (5):

[0074] For example, after calculating the m average values corresponding to the m sample texts, the expected value mu and the standard deviation delta of the normal distribution are calculated according to the m average values. Assuming that the m average values are x1, x2,..., xm. The calculation formula of the expected value mu can be expressed as the following formula (4) or formula (5):

[0075] μ=α1*x1+α2*x2+…+αm*xm (5)

[0076] In formula (5), α1, α2, ..., αm represent weighting coefficients.

[0077] The standard deviation δ of the normal distribution is the variance δ of the normal distribution. 2 The arithmetic square root, the variance δ of the normal distribution 2 It can be expressed as follows (6):

[0078]

[0079] After calculating the expected value μ and standard deviation δ of the normal distribution according to the formula (4) or formula (5) and formula (6) as described above, the confidence interval that conforms to the normal distribution can be determined according to the formula (3) above.

[0080] Furthermore, based on this confidence interval, outliers among sample text 1, sample text 2, ..., sample text m can be identified. For example, sample text 1 corresponds to the average x1, sample text 2 corresponds to the average x2, and so on, with sample text m corresponding to the average xm. For each of the m averages, x1, x2, ..., xm, it is determined whether the average is outside the confidence interval described above. If the average is outside the confidence interval, the sample text corresponding to that average is marked as an outlier. Outlier texts can serve as initial core points in the density clustering algorithm. It is understood that there may be more than one outlier text; therefore, when there is only one outlier text, it is used as the initial core point in the density clustering algorithm. If there are multiple outlier texts, each of these outlier texts is used as an initial core point in the density clustering algorithm, i.e., the number of initial core points is the same as the number of outlier texts. Specifically, the process of calculating the confidence interval and setting the initial core points for density clustering corresponds to the following... Figure 3 The perplexity statistics are shown.

[0081] In this embodiment, a density-based clustering algorithm is used to cluster sample text 1, sample text 2, ..., sample text m to obtain multiple clusters. Several feasible implementation methods are described below.

[0082] In one possible implementation, clustering the plurality of sample texts to obtain multiple clusters includes: calculating the distance between any two sample texts based on the perplexity vectors corresponding to any two sample texts; and clustering the plurality of sample texts based on the distance between any two sample texts to obtain multiple clusters.

[0083] For example, according to the perplexity vectors corresponding to any two of the sample texts 1, 2, …, m, the distance between the two sample texts is calculated, which can be in the form of Euclidean distance. For example, the two sample texts selected from the sample texts 1, 2, …, m are the sample texts 1 and 2, the perplexity vector corresponding to the sample text 1 is denoted as [x(1), x(2), …, x(N)], the perplexity vector corresponding to the sample text 2 is denoted as [y(1), y(2), …, y(N)], and the Euclidean distance between the sample texts 1 and 2 is denoted as d(x, y), which can be expressed as formula (7) as follows:

[0084]

[0085] where x(i) is a real number representing the i-th coordinate in the perplexity vector [x(1), x(2), …, x(N)]. y(i) is a real number representing the i-th coordinate in the perplexity vector [y(1), y(2), …, y(N)]. Further, using the density clustering algorithm, the distances between any two of the sample texts 1, 2, …, m are used to cluster the sample texts 1, 2, …, m, and a plurality of clustering clusters are obtained. Further, the clustering cluster containing the abnormal samples is removed, so that the remaining sample texts form a high-quality data set. Specifically, the process of clustering to remove the clustering cluster containing the abnormal samples to obtain the high-quality data set can be data filtering as shown in Figure 3 .

[0086] In another possible implementation, the plurality of sample texts are text data belonging to the same field; and the clustering processing of the plurality of sample texts to obtain a plurality of clustering clusters comprises: adjusting a hyperparameter required for clustering processing according to the standard deviation; and clustering processing the plurality of sample texts according to the hyperparameter and the distance between any two of the plurality of sample texts to obtain a plurality of clustering clusters.

[0087] For example Figure 3The sample text 1, the sample text 2, …, and the sample text m shown are text data belonging to the same field. In this case, after calculating the standard deviation δ corresponding to the field according to formula (3) as described above, the super parameter required for the clustering process can be adjusted according to the standard deviation δ, which can be the density clustering initial core point as described above. Further, according to the adjusted super parameter and the distance between any two of the sample text 1, the sample text 2, …, and the sample text m, the sample text 1, the sample text 2, …, and the sample text m are clustered to obtain a plurality of clustering clusters. The distance between any two sample texts can be calculated according to formula (7) as described above.

[0088] The embodiments of the present disclosure adjust the super parameter required for the clustering process by the standard deviation δ corresponding to the field to which the plurality of sample texts belong, so that the adjusted super parameter can affect the clustering process of the plurality of sample texts, to ensure the accuracy of the clustering result, reduce the dependence on manual intervention, and improve the universality and flexibility of the sample text cleaning process.

[0089] It can be understood that the above embodiments are illustratively described by taking the plurality of sample texts belonging to the same field as an example. The sample texts in multiple fields will be introduced in combination with specific embodiments. For example, Figure 6 As shown, in the data collection phase, i.e., the sample text acquisition phase, the server 21 can acquire a plurality of data sources, such as data source 1, data source 2, …, and data source M, wherein each data source corresponds to a field, and each data source includes a plurality of sample texts, i.e., the plurality of sample texts in the same data source are text data belonging to the same field. In addition, the data source 1, the data source 2, …, and the data source M can constitute a field data set. In this case, each data source is independent of each other, i.e., each data source is processed by the N large language models independently. For example, for each sample text in the data source 1, the sample text is input into the N large language models to obtain N perplexity values corresponding to the sample text. Further, for each sample text in the data source 2, the sample text is input into the N large language models to obtain N perplexity values corresponding to the sample text. And so on. Figure 6 As shown, when calculating the confidence interval, each data source corresponds to a confidence interval, i.e., one confidence interval is calculated according to the plurality of sample texts in each data source, and the calculation process of the confidence interval can refer to the related content described in the above embodiments. In addition, the confidence intervals corresponding to different data sources can be different. When setting the density clustering initial core point, since the confidence intervals corresponding to different data sources can be different, the abnormal samples determined from different data sources can be different, and therefore the initial core points in the density clustering algorithm for different data sources are different.

[0090] In addition, since the specific value of the standard deviation δ calculated according to text data of different fields, such as different data sources, is different, the super parameter adjusted according to the corresponding δ of different fields is also different, and the super parameter will affect the clustering process, so that the clustering clusters obtained after the text data in different fields are clustered are also different. Thus, the accuracy of the clustering results of different data sources is ensured.

[0091] As shown in Figure 6 During the clustering process, different data sources are clustered respectively, and for each data source, the clustering cluster containing the abnormal sample in the plurality of clustering clusters corresponding to the data source is removed to obtain a high-quality data set corresponding to each data source.

[0092] The embodiments of the present disclosure can significantly improve the quality of the data set by automatically identifying and filtering abnormal samples, and provide a reliable data foundation for natural language processing tasks in specific fields. In addition, the method described in the embodiments of the present disclosure has the advantages of simple operation, high processing efficiency, strong scalability, etc., and is particularly suitable for processing large-scale data sets, providing strong support for the application and development of natural language processing technology.

[0093] Figure 7 The structure diagram of the text data processing device provided by the embodiments of the present disclosure. The text data processing device provided by the embodiments of the present disclosure can execute the processing flow provided by the method embodiments, as shown in Figure 7 The text data processing device 70 includes an acquisition module 71, an inference module 72, a determination module 73, a clustering module 74, and a removal module 75.

[0094] The acquisition module 71 is configured to acquire a plurality of sample texts.

[0095] The inference module 72 is configured to infer each sample text in the plurality of sample texts by a plurality of pre-trained models to obtain a plurality of perplexity values corresponding to the sample text, the plurality of models and the plurality of perplexity values being one-to-one corresponding.

[0096] The determination module 73 is configured to determine a confidence interval according to the plurality of perplexity values corresponding to the plurality of sample texts.

[0097] The clustering module 74 is configured to perform clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters.

[0098] The removal module 75 is configured to remove a clustering cluster containing an abnormal sample in the plurality of clustering clusters, the abnormal sample being a sample text outside the confidence interval in the plurality of sample texts.

[0099] Optionally, the sample text corresponds to a plurality of text units, and the perplexity value output by the model for the sample text is calculated according to parameters of the model and a log-likelihood probability of each text unit in the plurality of text units, the log-likelihood probability of the text unit being determined based on other text units before the text unit.

[0100] Optionally, when the determination module 73 determines the confidence interval according to the plurality of perplexity values respectively corresponding to the plurality of sample texts, the determination module 73 is specifically configured to: for each sample text in the plurality of sample texts, integrate the plurality of perplexity values corresponding to the sample text into a perplexity vector, and calculate an average value of elements contained in the perplexity vector; and calculate the confidence interval conforming to a normal distribution according to the average values respectively corresponding to the plurality of sample texts.

[0101] Optionally, when the clustering module 74 performs clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters, the clustering module 74 is specifically configured to: calculate distances between any two sample texts in the plurality of sample texts according to perplexity vectors respectively corresponding to the any two sample texts; and perform clustering processing on the plurality of sample texts according to the distances between the any two sample texts in the plurality of sample texts to obtain the plurality of clustering clusters.

[0102] Optionally, when the determination module 73 calculates the confidence interval conforming to the normal distribution according to the average values respectively corresponding to the plurality of sample texts, the determination module 73 is specifically configured to: calculate an expected value and a standard deviation of the normal distribution according to the average values respectively corresponding to the plurality of sample texts; and calculate the confidence interval conforming to the normal distribution according to the expected value and the standard deviation of the normal distribution.

[0103] Optionally, the plurality of sample texts are text data belonging to the same field; and when the clustering module 74 performs clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters, the clustering module 74 is specifically configured to: adjust a hyperparameter required for the clustering processing according to the standard deviation; and perform clustering processing on the plurality of sample texts according to the hyperparameter and the distances between any two sample texts in the plurality of sample texts to obtain the plurality of clustering clusters.

[0104] Figure 7 The text data processing apparatus of the illustrated embodiment can be used to execute the technical solutions of the method embodiments described above, and has similar implementation principles and technical effects, which will not be described herein again.

[0105] The example embodiments of the present disclosure also provide an electronic device, including: at least one processor; and a memory connected in communication with the at least one processor. The memory stores a computer program capable of being executed by the at least one processor, and the computer program, when executed by the at least one processor, is configured to cause the electronic device to perform the method according to the embodiments of the present disclosure.

[0106] The exemplary embodiments of this disclosure further provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, causes the computer to perform the method according to the embodiments of this disclosure.

[0107] The exemplary embodiments of this disclosure further provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, causes the computer to perform the method according to the embodiments of this disclosure.

[0108] Reference Figure 8 will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent a wide variety of digital electronic computer devices such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent a variety of mobile devices such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components, their connections, and their functions, as described herein, are meant to be examples only, and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] As shown in Figure 8 , the electronic device 80 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded into a random access memory (RAM) 803 from a storage unit 808. In the RAM 803, various programs and data required for the operation of the device 80 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0110] A plurality of components in the electronic device 80 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device that can input information to the electronic device 80, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 807 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 804 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 809 allows the electronic device 80 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0111] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above. For example, in some embodiments, the methods described in the above method embodiments can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 80 via the ROM 802 and / or the communication unit 809. In some embodiments, the computing unit 801 can be configured to perform the methods described in the above method embodiments by any other appropriate means, such as by means of firmware.

[0112] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow charts and / or block diagrams to be implemented. The program code can be implemented in a wholly in machine language, in part in machine language, in part in assembly language, and / or in part in a high-level language such as C, C++, C#, Java, Python, etc. The program code can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or server, or entirely on the remote machine or server.

[0113] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0114] As used in this disclosure, the terms "machine-readable medium" and "computer- readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or data to a programmable processor.

[0115] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0116] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0117] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A method for processing text data, the method comprising: obtaining a plurality of sample texts; for each of the plurality of sample texts, performing inference on the sample text by a plurality of pre-trained models to obtain a plurality of perplexity values corresponding to the sample text, the plurality of models and the plurality of perplexity values corresponding one-to-one; determining a confidence interval according to the plurality of perplexity values corresponding to the plurality of sample texts respectively; performing clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters; removing a clustering cluster containing an abnormal sample from the plurality of clustering clusters, the abnormal sample being a sample text located outside the confidence interval among the plurality of sample texts; wherein the determining of the confidence interval according to the plurality of perplexity values corresponding to the plurality of sample texts respectively comprises: for each of the plurality of sample texts, integrating the plurality of perplexity values corresponding to the sample text into a perplexity vector, and calculating an average value of elements contained in the perplexity vector; calculating a confidence interval conforming to a normal distribution according to the average values corresponding to the plurality of sample texts respectively; wherein the performing of the clustering processing on the plurality of sample texts to obtain the plurality of clustering clusters comprises: calculating a distance between any two sample texts in the plurality of sample texts according to perplexity vectors corresponding to the any two sample texts respectively; performing clustering processing on the plurality of sample texts according to the distances between any two sample texts in the plurality of sample texts to obtain the plurality of clustering clusters.

2. The method of claim 1, wherein, The sample text corresponds to a plurality of text units, and the perplexity value output by the model for the sample text is calculated according to parameters of the model and a log-likelihood probability of each text unit in the plurality of text units, and the log-likelihood probability of the text unit is determined based on other text units before the text unit.

3. The method of claim 1, wherein, The calculating of the confidence interval conforming to the normal distribution according to the average values corresponding to the plurality of sample texts respectively comprises: calculating an expected value and a standard deviation of a normal distribution according to the average values corresponding to the plurality of sample texts respectively; calculating the confidence interval conforming to the normal distribution according to the expected value and the standard deviation of the normal distribution.

4. The method of claim 3, wherein, The plurality of sample texts are text data belonging to the same field; The performing of the clustering processing on the plurality of sample texts to obtain the plurality of clustering clusters comprises: adjusting a hyperparameter required for clustering processing according to the standard deviation; performing clustering processing on the plurality of sample texts according to the hyperparameter and the distances between any two sample texts in the plurality of sample texts to obtain the plurality of clustering clusters. 5.An apparatus for processing text data, the apparatus comprising: an obtaining module configured to obtain a plurality of sample texts; an inference module configured to, for each of the plurality of sample texts, perform inference on the sample text by a plurality of pre-trained models to obtain a plurality of perplexity values corresponding to the sample text, the plurality of models and the plurality of perplexity values corresponding one-to-one; a determining module configured to determine a confidence interval according to the plurality of perplexity values corresponding to the plurality of sample texts respectively; The clustering module is configured to perform clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters. The removing module is configured to remove a clustering cluster containing an abnormal sample from the plurality of clustering clusters, the abnormal sample being a sample text outside the confidence interval in the plurality of sample texts. The method further includes determining a confidence interval according to a plurality of perplexity values corresponding to the plurality of sample texts, wherein the confidence interval is determined according to the plurality of perplexity values corresponding to the plurality of sample texts, including: For each sample text in the plurality of sample texts, integrating the plurality of perplexity values corresponding to the sample text into a perplexity vector, and calculating an average value of elements contained in the perplexity vector; According to the average values corresponding to the plurality of sample texts, calculating a confidence interval conforming to a normal distribution; The method further includes performing clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters, wherein the clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters includes: According to perplexity vectors corresponding to any two sample texts in the plurality of sample texts, calculating distances between the any two sample texts; According to the distances between any two sample texts in the plurality of sample texts, performing clustering processing on the plurality of sample texts to obtain a plurality of clustering clusters.

6. An electronic device comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-4.

7. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

8. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-4.

Images