Abnormal detection method, device, equipment and medium for text classification and annotation samples

Abstract

The embodiment of the present invention discloses a method, device, equipment and medium for detecting anomalies of text classification and annotation samples. The method includes: obtaining a text classification and annotation data set to be denoised; calculating the semantic similarity between two text classification and annotation data, clustering the text classification and annotation data set to obtain at least one cluster cluster; in each cluster cluster, performing secondary clustering on each text classification and annotation data with the same classification label to obtain classification subclusters corresponding to each cluster cluster; identifying abnormal classification subclusters according to the proportion of the number of text classification and annotation data in the classification subclusters in the cluster cluster to which they belong. The embodiment of the present invention solves the problem of heavy workload of staff due to low recognition accuracy of the sample denoising model and no explanatory description of sample denoising, realizes the ability to effectively detect anomalies of text classification and annotation samples, improves the accuracy of sample denoising, and reduces the labor cost of sample denoising.

Description

Technical Field

[0001] Embodiments of the present invention relate to computer data processing technology, and in particular to a method, device, equipment and medium for detecting anomalies in text classification and annotation samples. Background Art

[0002] Text classification is a common processing task in the field of machine learning, and its application scenarios include news classification, sentiment analysis, intent recognition, and so on. In the actual text classification task processing process, developers first need to complete a certain amount of labeled samples according to the classification labels required by the scenario, and then complete the text classification service construction through model training. In this process, the quality of the labeled samples is closely related to the accuracy of the text service prediction. The performance of the model constructed by the sample set with high labeling quality is better, otherwise it will lead to poor classification results. There are many reasons for the source of noise samples that affect the quality of labeling, including inconsistent standards within the labeling team and subjective judgment errors of the labelers, so sample denoising is an important part of the text classification application development process.

[0003] The inventors found in the process of invention that the defects of the prior art are: the noise samples may select normal samples for elimination and retain wrong samples from conflicting samples by judging whether the neural network has converged, which further reduces the data quality. The low recognition accuracy of noise samples is easy to introduce a lot of manual work, and it will also cause normal samples to be incorrectly filtered out and noise samples to be retained. Summary of the invention

[0004] The embodiments of the present invention provide a method, apparatus, device and medium for detecting anomalies of text classification and annotation samples, so as to effectively detect anomalies of text classification and annotation samples and reduce the labor cost of sample denoising.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting anomalies of text classification and annotation samples, which includes:

[0006] Obtain a text classification annotation data set to be denoised, wherein each text classification annotation data includes a classification label;

[0007] Calculating the semantic similarity between two text classification and annotation data, and performing clustering processing on the text classification and annotation data sets according to the semantic similarity calculation results to obtain at least one cluster cluster;

[0008] In each cluster, the text classification annotation data with the same classification label are clustered again to obtain classification sub-clusters corresponding to each cluster;

[0009] Identify abnormal classification subclusters based on the proportion of text classification annotation data in the classification subclusters in the corresponding clustering clusters.

[0010] In a second aspect, an embodiment of the present invention further provides an anomaly detection device for text classification and annotation samples, the anomaly detection device for text classification and annotation samples comprising:

[0011] A text classification annotation data set acquisition module is used to acquire a text classification annotation data set to be denoised, wherein each text classification annotation data includes a classification label;

[0012] A cluster determination module, used to calculate the semantic similarity between two text classification and annotation data, and perform clustering processing on the text classification and annotation data set according to the semantic similarity calculation result to obtain at least one cluster;

[0013] A classification sub-cluster determination module is used to perform secondary clustering on each text classification annotation data with the same classification label in each cluster cluster to obtain classification sub-clusters corresponding to each cluster cluster;

[0014] The abnormal classification sub-cluster identification module is used to identify abnormal classification sub-clusters according to the proportion of the text classification annotation data in the classification sub-cluster in the corresponding cluster.

[0015] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the anomaly detection method for text classification and annotation samples as described in any embodiment of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting anomalies of text classification and annotation samples as described in any embodiment of the present invention is implemented.

[0017] The technical solution provided by the embodiment of the present invention obtains a text classification annotation data set to be denoised; calculates the semantic similarity between two text classification annotation data, performs clustering processing on the text classification annotation data set, and obtains at least one cluster cluster; in each cluster cluster, performs secondary clustering on each text classification annotation data with the same classification label to obtain classification subclusters corresponding to each cluster cluster; and identifies abnormal classification subclusters according to the proportion of the number of text classification annotation data in the classification subclusters in the corresponding cluster clusters. The embodiment of the present invention solves the problem of heavy workload of staff due to low recognition accuracy of the sample denoising model and no explanatory description of sample denoising, realizes the ability to effectively detect anomalies in text classification annotation samples, improves the accuracy of sample denoising, and reduces the labor cost of sample denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a method for detecting anomalies in text classification and annotation samples provided in Embodiment 1 of the present invention;

[0019] Figure 2 A flowchart of another method for detecting anomalies of text classification and annotation samples provided in Embodiment 2 of the present invention;

[0020] Figure 3 It is a structural schematic diagram of an anomaly detection device for text classification and annotation samples provided in Embodiment 3 of the present invention;

[0021] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0023] Embodiment 1

[0024] Figure 1 The present invention provides a flowchart of a method for detecting anomalies of text classification and annotation samples in accordance with the first embodiment of the present invention. The present embodiment is applicable to the case where sample denoising is performed on text classification and annotation samples. The method of the present embodiment can be performed by an apparatus for detecting anomalies of text classification and annotation samples, which can be implemented by software and / or hardware, and can be configured in a server or terminal device.

[0025] Accordingly, the method specifically comprises the following steps:

[0026] S110: Obtain a text classification and annotation dataset to be denoised.

[0027] Among them, each text classification annotation data includes a classification label.

[0028] The text classification annotation data set may be a data set including a plurality of text classification annotation data. The text classification annotation data may be different annotation data according to different classification labels, and the text classification annotation data may be divided into different categories according to the classification labels.

[0029] For example, assume that there are 100 text classification annotation data in the text classification annotation dataset, among which there are 20 text classification annotation data in classification label A, 50 text classification annotation data in classification label B, 25 text classification annotation data in classification label C, and 5 text classification annotation data in classification label D.

[0030] S120, calculating the semantic similarity between two sets of text classification and annotation data, and performing clustering processing on the text classification and annotation data sets according to the semantic similarity calculation results to obtain at least one cluster.

[0031] Among them, semantic similarity can be the similarity between two text classification annotation data. The cosine distance of the two text classification annotation data can be calculated according to the trained semantic similarity model, and then the semantic similarity between the two text classification annotation data can be obtained. Clustering processing can be to cluster and merge adjacent similar classification areas using morphological operators, that is, to cluster similar text classification annotation data. Cluster clusters can be a set of samples generated by clustering, and samples in the same cluster are similar to each other and different from samples in other clusters. Specifically, after clustering similar text classification annotation data, one or more cluster clusters can be obtained.

[0032] Optionally, calculating the semantic similarity between pairwise text classification and annotation data includes: inputting pairwise text classification and annotation data into a pre-trained semantic similarity model respectively, and obtaining the semantic similarity between pairwise text classification and annotation data.

[0033] The semantic similarity model may be a model that calculates the semantic similarity between two input text classification and annotation data.

[0034] Continuing with the previous example, assume that there are 100 text classification annotation data in the text classification annotation data set. The corresponding semantic similarity between the 100 text classification annotation data is calculated. Then, the text classification annotation data set can be clustered according to the semantic similarity calculation results. Assume that 2 clusters can be obtained. 20 text classification annotation data in classification label A and 5 text classification annotation data in classification label D can be clustered into one category, and cluster 1 is obtained. 50 text classification annotation data in classification label B and 25 text classification annotation data in classification label C can be clustered into one category, and cluster 2 is obtained.

[0035] The advantage of this setting is that the semantic similarity can be obtained by inputting the pairwise text classification annotation data into the pre-trained semantic similarity model, which makes it more convenient to calculate the semantic similarity between the two, and the calculation of semantic similarity through the semantic similarity model is more reasonable and accurate.

[0036] S130 . In each cluster, perform secondary clustering on the text classification annotation data with the same classification label to obtain classification subclusters corresponding to each cluster.

[0037] Among them, the secondary clustering can be to perform clustering processing again in the obtained cluster cluster to obtain more similar classification sub-clusters. Classification sub-clusters can be sub-clusters existing in the cluster cluster, and a cluster cluster can contain one or more classification sub-clusters.

[0038] Continuing with the previous example, in the obtained clusters 1 and 2, the text classification annotated data with the same classification label are clustered again. Two classification subclusters can be obtained in cluster 1, and there are 20 text classification annotated data in the classification label A as classification subcluster 1, and there are 5 text classification annotated data in the classification label D as classification subcluster 2. Two classification subclusters can be obtained in cluster 2, and there are 50 text classification annotated data in the classification label B as classification subcluster 3, and there are 25 text classification annotated data in the classification label C as classification subcluster 4.

[0039] S140, identifying abnormal classification subclusters according to the proportion of the text classification annotation data in the classification subclusters in the corresponding clustering clusters.

[0040] The abnormal classification sub-cluster may be a classification sub-cluster in the clustering cluster that does not meet the weighted filtering threshold.

[0041] Optionally, abnormal classification subclusters are identified based on the proportion of the text classification annotation data in the classification subcluster in the corresponding cluster, including: counting the number of text classification annotation data in the current classification subcluster, and calculating the weight value of the proportion of the number of each classification annotation data in the corresponding cluster; judging whether each proportion weight value is greater than a preset proportion weight filtering threshold, if not, identifying the text classification annotation data in the current classification subcluster as an abnormal classification subcluster.

[0042] The weight ratio value may be the weight value of the current classification subcluster in the cluster to which it belongs. The weight ratio filtering threshold may be a pre-set weight ratio filtering threshold. If the weight ratio value of the current classification subcluster is less than or equal to the weight ratio filtering threshold, the current classification subcluster is an abnormal classification subcluster. If the weight ratio value of the current classification subcluster is greater than the weight ratio filtering threshold, the current classification subcluster is a normal classification subcluster.

[0043] Continuing with the previous example, assume that the weighted filter threshold is 30%. In cluster 1, it can be calculated that the weighted value of classification sub-cluster 1 is 80%. Since 80% is greater than 30%, classification sub-cluster 1 is a normal classification sub-cluster. The weighted value of classification sub-cluster 2 is 20%. Since 20% is less than 30%, classification sub-cluster 2 is an abnormal classification sub-cluster.

[0044] Furthermore, in cluster 2, it can be calculated that the weight value of classification subcluster 3 is 66.67%. Since 66.67% is greater than 30%, classification subcluster 3 is a normal classification subcluster. The weight value of classification subcluster 4 is 33.33%. Since 33.33% is greater than 30%, classification subcluster 4 is a normal classification subcluster.

[0045] The advantage of this setting is that by calculating the weight value of the proportion of the number of each classification annotation data in the current classification subcluster to the number of the cluster to which it belongs, and comparing it with the preset weight filter threshold, it can be determined whether the current classification subcluster belongs to a normal classification subcluster or an abnormal classification subcluster. In this way, the weight value can be calculated scientifically and accurately, and the current classification subcluster can be effectively judged, which improves the effectiveness and reliability of the judgment of abnormal classification subcluster.

[0046] The technical solution provided by the embodiment of the present invention obtains a text classification annotation data set to be denoised; calculates the semantic similarity between two text classification annotation data, performs clustering processing on the text classification annotation data set, and obtains at least one cluster cluster; in each cluster cluster, performs secondary clustering on each text classification annotation data with the same classification label to obtain classification subclusters corresponding to each cluster cluster; and identifies abnormal classification subclusters according to the proportion of the number of text classification annotation data in the classification subclusters in the corresponding cluster clusters. The embodiment of the present invention solves the problem of heavy workload of staff due to low recognition accuracy of the sample denoising model and no explanatory description of sample denoising, realizes the ability to effectively detect anomalies in text classification annotation samples, improves the accuracy of sample denoising, and reduces the labor cost of sample denoising.

[0047] Optionally, before inputting the pairwise text classification annotation data into the pre-trained semantic similarity model respectively, it also includes: inputting the two acquired sample classification annotation data into the parameter sharing layer respectively to obtain multiple word vectors corresponding to the two sample classification annotation data respectively; inputting each of the word vectors corresponding to the first sample classification annotation data into the pooling layer to obtain a first sample classification annotation data vector, and inputting each of the word vectors corresponding to the second sample classification annotation data into the pooling layer to obtain a second sample classification annotation data vector; calculating the absolute value of the difference between the first sample classification annotation data vector and the second sample classification annotation data vector to obtain a sample classification annotation data difference vector; concatenating the first sample classification annotation data vector, the second sample classification annotation data vector and the sample classification annotation data difference vector to obtain a sample classification annotation data concatenation vector; inputting the sample classification annotation data concatenation vector into the semantic classifier for training, and obtaining a semantic similarity model after the training is completed.

[0048] The sample classification annotation data may be sample data obtained from a sample classification annotation data set. The parameter sharing layer may be a sharing layer capable of processing the received sample classification annotation data. Specifically, the parameter sharing layer may be a Bert layer in a sentence-Bert semantic similarity model, which may represent the received sample classification annotation data in the form of a word vector. The word vector may be a vector obtained by vectorizing each word in the sample classification annotation data.

[0049] Among them, the first sample classification annotation data may be one of the sample classification annotation data. The pooling layer may be a method of partitioning and sampling the data, downsampling a large matrix into a small matrix, reducing the amount of calculation, and preventing overfitting. Specifically, it refers to averaging all the word vectors corresponding to each sample classification annotation data. The first sample classification annotation data vector may be a vector obtained by averaging all the word vectors corresponding to the first sample classification annotation data. The second sample classification annotation data may be another sample data of the two sample classification annotation data. The second sample classification annotation data vector may be a vector obtained by averaging all the word vectors corresponding to the second sample classification annotation data. The sample classification annotation data difference vector may be a difference vector obtained by calculating the difference between the two sample classification annotation data vectors and performing absolute value processing on the obtained difference. The sample classification annotation data splicing vector may be a vector obtained by splicing two or more vectors. The semantic classifier may be a processing layer capable of performing semantic classification on the input sample classification annotation data splicing vector.

[0050] For example, assume that two sample classification annotation data are obtained, namely, the first sample classification annotation data and the second sample classification annotation data. The first sample classification annotation data and the second sample classification annotation data are respectively input into the parameter sharing layer. The word vector obtained by the first sample classification annotation data is {m 1 , m 2 , m 3 ,…,m p}; The word vector obtained by the second sample classification labeling data is {n 1 , n 2 , n 3 ,…,n q The word vectors corresponding to the first sample classification label data are input into the pooling layer, and the first sample classification label data vector is obtained as Input each word vector corresponding to the second sample classification label data into the pooling layer, and obtain the second sample classification label data vector as Therefore, the sample classification labeling data difference vector can be obtained as |uv|.

[0051] Furthermore, vectors u, v and |uv| are concatenated to obtain a sample classification annotation data concatenation vector w = {u, v, uv}. Specifically, assuming that vector u and vector v are three-dimensional vectors, the sample classification annotation data concatenation vector w is a nine-dimensional vector. Accordingly, the sample classification annotation data concatenation vector w is input into the semantic classifier for training, and a semantic similarity model is obtained after the training is completed.

[0052] The advantage of this setting is that the semantic similarity model is trained through sample classification and annotation data, so that the trained semantic similarity model can more accurately output the semantic similarity between two text classification and annotation data, thereby being able to more accurately cluster each text classification and annotation data.

[0053] Optionally, after identifying abnormal classification subclusters according to the proportion of the text classification annotation data in the classification subclusters in the corresponding clustering cluster, it also includes: adding explanation labels to each text classification annotation data in the identified abnormal classification subclusters, and feeding back to the user.

[0054] The explanation label may be a label that can explain the weight value of the proportion of the abnormal classification sub-cluster.

[0055] Continuing with the previous example, assume that the weighted filter threshold is 30%. In cluster 1, it can be calculated that the weighted value of classification subcluster 1 is 80%. Since 80% is greater than 30%, classification subcluster 1 is a normal classification subcluster. The weighted value of classification subcluster 2 is 20%. Since 20% is less than 30%, classification subcluster 2 is an abnormal classification subcluster. Classification subcluster 1 corresponds to label A, and classification subcluster 2 corresponds to label D. Since classification subcluster 1 is a normal classification subcluster, there is no need to add an explanatory label and feedback to the user. The explanatory label that can be added is: In cluster 1, the weighted value of label D is 20%, and the weighted value of label D is lower than the threshold.

[0056] Furthermore, in cluster 2, it can be calculated that the weight value of classification subcluster 3 is 66.67%. Since 66.67% is greater than 30%, classification subcluster 3 is a normal classification subcluster. The weight value of classification subcluster 4 is 33.33%. Since 33.33% is greater than 30%, classification subcluster 4 is a normal classification subcluster. Classification subcluster 3 corresponds to label B, and classification subcluster 4 corresponds to label C. Since classification subcluster 3 and classification subcluster 4 are both normal classification subclusters, there is no need to add explanation labels and feedback to the user.

[0057] The advantage of this setting is that by adding explanation labels to the text classification annotation data in the identified abnormal classification sub-cluster and feeding it back to the user, it can make the work of the staff more convenient, reduce the workload of the staff, improve the efficiency of the staff, and increase the readability of the labels corresponding to the abnormal classification sub-cluster.

[0058] Embodiment 2

[0059] Figure 2 A flowchart of another method for detecting anomalies of text classification and annotation samples provided in Embodiment 2 of the present invention. This embodiment is refined based on the above embodiments. In this embodiment, the text classification and annotation dataset is clustered according to the result of semantic similarity calculation to obtain at least one cluster for further refinement.

[0060] Accordingly, the method specifically comprises the following steps:

[0061] S210: Obtain a text classification and annotation dataset to be denoised.

[0062] S220, calculating the semantic similarity between two text classification annotation data.

[0063] S230: construct a semantic similarity matrix according to the semantic similarity calculation result.

[0064] A matrix element in the semantic similarity matrix is ​​the semantic similarity between two text classification annotation data.

[0065] The semantic similarity matrix may be a similarity matrix obtained by filling in the semantic similarities between each pair of text classification annotation data.

[0066] S240: Obtain an unprocessed target data from the text classification and annotation data set, and mark the target data as being in a processed state.

[0067] The target data may be unprocessed text classification annotation data selected from the text classification annotation data set as the target processing data.

[0068] S250 , starting from the target data, querying the semantic similarity matrix, and traversing all density-connected data with the target data one by one.

[0069] Among them, all density-connected data may be density-connected data associated with target data, which can reflect the closeness between the target data and other data, so as to determine whether clustering can be performed.

[0070] S260: The target data and each of the density-connected data are combined into a cluster, and the processed status of each of the density-connected data is marked.

[0071] S270, determining whether there is unprocessed target data in all text classification and annotation data, if so, returning to execute S240, if not, executing S280;

[0072] S280: In each cluster, perform secondary clustering on the text classification annotation data with the same classification label to obtain classification subclusters corresponding to each cluster.

[0073] S290, identifying abnormal classification subclusters according to the proportion of the text classification annotation data in the classification subclusters in the corresponding clustering clusters.

[0074] Exemplarily, the text classification annotation data set to be denoised can be obtained first. Assuming that it contains 100 text classification annotation data, the semantic similarity between the two text classification annotation data is calculated respectively, so that the semantic similarity matrix can be obtained. Assuming that there are 90 unprocessed text classification annotation data among the 100 text classification annotation data, one of the 90 unprocessed text classification annotation data is selected as the target data 1, and it is marked as processed. Taking the target data 1 as the starting point, the semantic similarity matrix is ​​queried, and all the density-connected data with the target data are traversed one by one. Assuming that there are 20 density-connected data. The target data and the 20 density-connected data are combined into a cluster 1, and the processed state of the 20 density-connected data is marked respectively. This is the remaining 69 unprocessed target data. Therefore, one target data is selected from the 69 unprocessed target data, and the remaining clusters can be obtained by the same logic. Assume that after the processing is completed, 3 clusters are obtained.

[0075] Furthermore, in each of the three clusters, the text classification annotation data with the same classification label are clustered again to obtain classification subclusters corresponding to each cluster. According to the proportion of the text classification annotation data in the classification subcluster in the cluster to which it belongs, the abnormal classification subclusters are identified.

[0076] Optionally, after clustering the text classification and annotation data set according to the semantic similarity calculation result to obtain at least one cluster, the method further includes: identifying isolated text classification and annotation data that does not belong to any cluster as abnormal annotation data.

[0077] The isolated text classification annotated data may be data that does not belong to any cluster after clustering, and the abnormal annotated data may be data that is in an abnormal state.

[0078] For example, assuming that the text classification annotation data set to be denoised contains 100 text classification annotation data, after clustering processing, 3 clusters and 5 isolated text classification annotation data can be obtained, which are then identified as abnormal annotation data.

[0079] An optional embodiment of this embodiment, assuming 100 text classification annotation data, can be set in each cluster to be determined as a valid cluster only if there are at least 10 text classification annotation data. Assuming that after clustering the 100 text classification annotation data, 4 clusters can be obtained, cluster 1 contains 20 text classification annotation data, cluster 2 contains 5 text classification annotation data, cluster 3 contains 50 text classification annotation data, and cluster 4 contains 25 text classification annotation data. Although the 5 text classification annotation data in cluster 2 can be clustered, because the threshold condition of the valid cluster is not met, the 5 text classification annotation data of cluster 2 are identified as abnormal annotation data.

[0080] The technical solution provided by the embodiment of the present invention obtains a text classification and annotation data set to be denoised; calculates the semantic similarity between two text classification and annotation data, and constructs a semantic similarity matrix according to the semantic similarity calculation result; obtains an unprocessed target data in the text classification and annotation data set, and marks the target data as processed; takes the target data as the starting point, queries the semantic similarity matrix, and successively traverses all density-connected data with the target data; forms a cluster with the target data and each density-connected data, and marks the processed state of each density-connected data; returns to execute the operation of obtaining an unprocessed target data in the text classification and annotation data set until the processing of all text classification and annotation data is completed; in each cluster cluster, performs secondary clustering on each text classification and annotation data with the same classification label, and obtains classification subclusters corresponding to each cluster cluster; identifies abnormal classification subclusters according to the proportion of the number of text classification and annotation data in the classification subcluster in the cluster cluster to which it belongs. The embodiment of the present invention realizes the ability to effectively cluster text classification and annotation samples, thereby improving the accuracy of sample denoising.

[0081] Embodiment 3

[0082] Figure 3 This is a schematic diagram of the structure of an anomaly detection device for text classification and annotation samples provided in the third embodiment of the present invention. The anomaly detection device for text classification and annotation samples provided in this embodiment can be implemented by software and / or hardware and can be configured in a terminal device or a server. It is used to implement an anomaly detection method for text classification and annotation samples in the embodiment of the present invention. Figure 3As shown, the device may specifically include: a text classification annotation data set acquisition module 310, a cluster determination module 320, a classification sub-cluster determination module 330 and an abnormal classification sub-cluster identification module 340.

[0083] The text classification annotation data set acquisition module 310 is used to acquire the text classification annotation data set to be denoised, wherein each text classification annotation data includes a classification label;

[0084] The cluster determination module 320 is used to calculate the semantic similarity between the two text classification and annotation data sets, and perform clustering processing on the text classification and annotation data sets according to the semantic similarity calculation results to obtain at least one cluster;

[0085] The classification sub-cluster determination module 330 is used to perform secondary clustering on each text classification annotation data with the same classification label in each cluster cluster to obtain classification sub-clusters corresponding to each cluster cluster;

[0086] The abnormal classification sub-cluster identification module 340 is used to identify abnormal classification sub-clusters according to the proportion of the text classification annotation data in the classification sub-cluster in the corresponding cluster.

[0087] The technical solution provided by the embodiment of the present invention obtains a text classification annotation data set to be denoised; calculates the semantic similarity between two text classification annotation data, performs clustering processing on the text classification annotation data set, and obtains at least one cluster cluster; in each cluster cluster, performs secondary clustering on each text classification annotation data with the same classification label to obtain classification subclusters corresponding to each cluster cluster; and identifies abnormal classification subclusters according to the proportion of the number of text classification annotation data in the classification subclusters in the corresponding cluster clusters. The embodiment of the present invention solves the problem of heavy workload of staff due to low recognition accuracy of the sample denoising model and no explanatory description of sample denoising, realizes the ability to effectively detect anomalies in text classification annotation samples, improves the accuracy of sample denoising, and reduces the labor cost of sample denoising.

[0088] On the basis of the above embodiments, the cluster determination module 320 may be specifically used to: input the pairwise text classification annotation data into a pre-trained semantic similarity model, respectively, to obtain the semantic similarity between the pairwise text classification annotation data.

[0089] On the basis of the above embodiments, it also includes a semantic similarity model training module 320, which can be specifically used for: before inputting the pairwise text classification annotation data into the pre-trained semantic similarity model respectively, inputting the two obtained sample classification annotation data into the parameter sharing layer respectively to obtain a plurality of word vectors corresponding to the two sample classification annotation data respectively; inputting the word vectors corresponding to the first sample classification annotation data into the pooling layer to obtain the first sample classification annotation data vector, and inputting the word vectors corresponding to the second sample classification annotation data into the pooling layer to obtain the second sample classification annotation data vector; calculating the absolute value of the difference between the first sample classification annotation data vector and the second sample classification annotation data vector to obtain the sample classification annotation data difference vector; concatenating the first sample classification annotation data vector, the second sample classification annotation data vector and the sample classification annotation data difference vector to obtain the sample classification annotation data concatenation vector; inputting the sample classification annotation data concatenation vector into the semantic classifier for training, and obtaining the semantic similarity model after the training is completed.

[0090] On the basis of the above embodiments, the cluster determination module 320 can be specifically used to: construct a semantic similarity matrix according to the semantic similarity calculation result, wherein a matrix element in the semantic similarity matrix is ​​the semantic similarity between two text classification annotation data; obtain an unprocessed target data in the text classification annotation data set, and mark the target data as a processed state; take the target data as the starting point, query the semantic similarity matrix, and traverse all density-connected data with the target data one by one; form the target data and each density-connected data into a cluster together, and mark the processed state of each density-connected data; return to execute the operation of obtaining an unprocessed target data in the text classification annotation data set until the processing of all text classification annotation data is completed.

[0091] On the basis of the above embodiments, it further includes an abnormal annotated data determination module, which can be specifically used to: identify isolated text classification annotated data that does not belong to any cluster as abnormal annotated data.

[0092] On the basis of the above embodiments, the abnormal classification subcluster identification module 340 can be specifically used to: count the number of each classification annotation data of the text in the current classification subcluster, and calculate the weight value of the proportion of the number of each classification annotation data to the number in the corresponding cluster; determine whether each proportion weight value is greater than a preset proportion weight filtering threshold, if not, identify the each classification annotation data of the text in the current classification subcluster as an abnormal classification subcluster.

[0093] On the basis of the above embodiments, it also includes an explanation label adding module, which can be specifically used to: after identifying abnormal classification subclusters according to the proportion of text classification annotation data in the classification subcluster in the corresponding cluster, add explanation labels to each text classification annotation data in the identified abnormal classification subcluster, and feed back to the user.

[0094] The above-mentioned device for detecting anomalies of text classification and annotation samples can execute the method for detecting anomalies of text classification and annotation samples provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] Embodiment 4

[0096] Figure 4 Schematic diagram of the structure of a computer device provided by Embodiment 4 of the present invention. Figure 4 As shown, the device includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of processors 410 in the device can be one or more. Figure 4 A processor 410 is taken as an example; the processor 410, memory 420, input device 430 and output device 440 in the device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0097] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method for detecting abnormalities of text classification and annotation samples in the embodiment of the present invention (for example, the text classification and annotation data set acquisition module 310, the cluster determination module 320, the classification sub-cluster determination module 330 and the abnormal classification sub-cluster identification module 340). The processor 410 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 420, that is, implements the above-mentioned method for detecting abnormalities of text classification and annotation samples, which includes:

[0098] A text classification and annotation data set to be denoised is obtained; the semantic similarity between two text classification and annotation data is calculated, and according to the semantic similarity calculation result, the text classification and annotation data set is clustered to obtain at least one cluster cluster; in each cluster cluster, the text classification and annotation data with the same classification label are clustered again to obtain classification subclusters corresponding to each cluster cluster; according to the proportion of the text classification and annotation data in the classification subcluster in the cluster cluster to which it belongs, abnormal classification subclusters are identified.

[0099] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0100] The input device 430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 440 may include a display device such as a display screen.

[0101] Embodiment 5

[0102] Embodiment 5 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a method for detecting anomalies of text classification and annotation samples when executed by a computer processor, the method comprising: obtaining a text classification and annotation data set to be denoised; calculating the semantic similarity between two text classification and annotation data sets, and clustering the text classification and annotation data sets according to the semantic similarity calculation results to obtain at least one cluster cluster; in each cluster cluster, performing secondary clustering on the text classification and annotation data sets with the same classification label to obtain classification subclusters corresponding to each cluster cluster; identifying abnormal classification subclusters according to the proportion of the text classification and annotation data in the classification subcluster in the cluster cluster to which it belongs.

[0103] Of course, the computer-readable storage medium provided in an embodiment of the present invention has computer-readable instructions that are not limited to the method operations described above, and can also execute related operations in the anomaly detection method for text classification and annotation samples provided in any embodiment of the present invention.

[0104] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0105] It is worth noting that in the embodiment of the above-mentioned anomaly detection device for text classification and annotation samples, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0106] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for detecting anomalies in text classification and annotation samples, characterized in that: include: Obtain a text classification annotation data set to be denoised, wherein each text classification annotation data includes a classification label; wherein the text classification annotation data set is a data set including a plurality of text classification annotation data; the text classification annotation data is an annotation data having different annotations according to different classification labels, and the text classification annotation data is divided into different categories based on the classification labels; Calculating the semantic similarity between two text classification and annotation data, and performing clustering processing on the text classification and annotation data sets according to the semantic similarity calculation results to obtain at least one cluster cluster; In each cluster, the text classification annotation data with the same classification label are clustered again to obtain classification sub-clusters corresponding to each cluster; Identify abnormal classification subclusters based on the proportion of text classification annotation data in the classification subclusters in the corresponding clustering clusters.

2. The method according to claim 1, characterized in that Calculate the semantic similarity between two text classification annotation data, including: The pairwise text classification and annotation data are respectively input into a pre-trained semantic similarity model to obtain the semantic similarity between the pairwise text classification and annotation data.

3. The method according to claim 2, characterized in that Before inputting the pairwise text classification annotation data into the pre-trained semantic similarity model, it also includes: The two sample classification labeling data obtained are respectively input into the parameter sharing layer to obtain multiple word vectors corresponding to the two sample classification labeling data; Input each of the word vectors corresponding to the first sample classification label data into the pooling layer to obtain a first sample classification label data vector, and input each of the word vectors corresponding to the second sample classification label data into the pooling layer to obtain a second sample classification label data vector; Calculate the absolute value of the difference between the first sample classification labeling data vector and the second sample classification labeling data vector to obtain the sample classification labeling data difference vector; Concatenate the first sample classification labeling data vector, the second sample classification labeling data vector, and the sample classification labeling data difference vector to obtain a sample classification labeling data concatenation vector; The sample classification annotation data concatenation vector is input into the semantic classifier for training, and after the training is completed, a semantic similarity model is obtained.

4. The method according to claim 1, characterized in that According to the semantic similarity calculation result, clustering processing is performed on the text classification and annotation data set to obtain at least one clustering cluster, including: According to the semantic similarity calculation result, a semantic similarity matrix is ​​constructed, wherein a matrix element in the semantic similarity matrix is ​​the semantic similarity between two text classification annotation data; In the text classification and annotation dataset, obtaining an unprocessed target data, and marking the target data as being in a processed state; Taking the target data as a starting point, querying the semantic similarity matrix, and traversing all density-connected data with the target data one by one; The target data and each of the density-connected data are combined into a cluster, and the processed status of each of the density-connected data is marked; Return to the operation of obtaining an unprocessed target data in the text classification and annotation data set until the processing of all the text classification and annotation data is completed.

5. The method according to claim 4, characterized in that According to the semantic similarity calculation result, clustering is performed on the text classification and annotation data set to obtain at least one cluster, and the method further includes: The isolated text classification annotation data that does not belong to any cluster is identified as abnormal annotation data.

6. The method according to any one of claims 1 to 5, characterized in that: According to the proportion of text classification annotation data in the classification sub-cluster in the cluster to which it belongs, identify abnormal classification sub-clusters, including: Count the number of labeled data of each category in the current classification sub-cluster, and calculate the weight value of the proportion of the number of labeled data of each category to the number of the corresponding cluster; It is determined whether each weight value is greater than a preset weight filtering threshold. If not, each classification annotation data of the text in the current classification sub-cluster is identified as an abnormal classification sub-cluster.

7. The method according to any one of claims 1 to 5, characterized in that: After identifying abnormal classification subclusters based on the proportion of text classification annotation data in the classification subclusters in the corresponding cluster clusters, the following is also included: An explanation label is added to each text classification annotation data in the identified abnormal classification sub-cluster and fed back to the user.

8. An anomaly detection device for text classification and annotation samples, characterized in that: include: A text classification annotation data set acquisition module is used to acquire a text classification annotation data set to be denoised, wherein each text classification annotation data includes a classification label; wherein the text classification annotation data set is a data set including a plurality of text classification annotation data; the text classification annotation data is an annotation data having different annotations according to different classification labels, and the text classification annotation data is divided into different categories based on the classification labels; A cluster determination module, used to calculate the semantic similarity between two text classification and annotation data, and perform clustering processing on the text classification and annotation data set according to the semantic similarity calculation result to obtain at least one cluster; A classification sub-cluster determination module is used to perform secondary clustering on each text classification annotation data with the same classification label in each cluster cluster to obtain classification sub-clusters corresponding to each cluster cluster; The abnormal classification sub-cluster identification module is used to identify abnormal classification sub-clusters according to the proportion of the text classification annotation data in the classification sub-cluster in the corresponding cluster.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting anomalies of text classification and annotation samples according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting anomalies of text classification and annotation samples as described in any one of claims 1 to 7 is implemented.

Images