A method and device for detecting outliers in text data

Abstract

The present application provides a method and device for detecting outliers in text data, relating to the field of text processing technology. The method can obtain a first benchmark sentence element and a benchmark threshold based on the sentence elements in the to-be-processed sentence set of target text data; obtain a preliminary sentence set based on the first benchmark sentence element, the benchmark threshold, and the to-be-processed sentence set; obtain an intersection sentence set corresponding to the preliminary sentence set based on the benchmark threshold and the preliminary sentence elements contained in the preliminary sentence set; obtain a fusion topic feature value based on the intersection sentence elements contained in the intersection sentence set; determine the outlier sentence elements of the target text data based on the first similarity value between the topic of the edge sentence elements contained in the edge sentence set and the fusion topic feature value; the edge sentence set is determined based on the elements in the preliminary sentence set excluding the intersection sentence set and the first benchmark sentence element. The method can detect outliers in text data and improve the overall quality of text data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of text processing technology, and in particular to a method and device for detecting outliers in text data. Background Art

[0002] In practical applications, in unstructured text datasets, there may be cases where documents do not belong to the text topic, such as scientific articles appearing in political news datasets, or fax covers mixed with medical records, which will affect the subsequent text processing results.

[0003] In order to avoid the situation where a document does not belong to the text topic, there is a practical need to detect and filter out abnormal information in the document that does not belong to the text topic.

[0004] Currently, existing outlier detection tools, although they can implement outlier detection, do not support text data as input, or can only be used for outlier detection of unstructured data such as images.

[0005] In the related art, there is a lack of outlier detection methods for text data, and the overall quality of text data is poor. Based on this, how to implement outlier detection in text data is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0006] To solve existing technical problems, the embodiments of the present application provide a method and device for detecting outliers in text data, which can detect outliers in text data and improve the overall quality of text data.

[0007] To achieve the above objectives, the technical solution of the embodiment of the present application is implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides a method for detecting outliers in text data, comprising:

[0009] A first benchmark sentence element and a benchmark threshold are obtained based on sentence elements in a to-be-processed sentence set of target text data; the benchmark threshold is the distance between a first topic of the first benchmark sentence element and a second topic of a second benchmark sentence element; the second benchmark sentence element and the first benchmark sentence element are adjacent to each other in the to-be-processed sentence set, and the first topic and the second topic meet a preset correlation condition;

[0010] Based on the first benchmark sentence element, the benchmark threshold and the set of sentences to be processed, a preliminary selected sentence set is obtained; the preliminary selected sentence set includes the preliminary selected sentence element in the set of sentences to be processed; the distance between the preliminary selected sentence element and the first topic is not greater than the benchmark threshold;

[0011] Based on the reference threshold and the preliminary selected sentence elements included in the preliminary selected sentence set, an intersection sentence set corresponding to the preliminary selected sentence set is obtained; the intersection sentence set includes the intersection sentence elements in the preliminary selected sentence set; the distance between the intersection sentence subject of the intersection sentence element and the preliminary selected sentence subject of any of the preliminary selected sentence elements is not greater than the reference threshold;

[0012] Obtaining a fusion topic feature value based on the intersection sentence elements contained in the intersection sentence set;

[0013] The abnormal sentence elements of the target text data are determined based on the first similarity value between the subject of the edge sentence elements contained in the edge sentence set and the fusion subject feature value; the edge sentence set is determined based on the elements in the preliminary sentence set except the intersection sentence set and the first benchmark sentence elements.

[0014] In an optional embodiment, obtaining the first benchmark sentence element and the benchmark threshold according to the sentence elements in the to-be-processed sentence set of the target text data includes:

[0015] Among the sentence elements included in the to-be-processed sentence set of the target text data, any adjacent preset number of the sentence elements are read one by one to obtain a sentence element group including a first sentence element and a second sentence element. For each sentence element group obtained, the following element group operation is performed until a first benchmark sentence element and a benchmark threshold are obtained:

[0016] For a currently obtained sentence element group, determining the subject of the sentence elements in the sentence element group; determining the subject of the sentence elements in the sentence element group includes obtaining the subject corresponding to the first sentence element according to the semantics of the first sentence element in the sentence element group, and obtaining the subject corresponding to the second sentence element according to the semantics of the second sentence element;

[0017] Performing a first judgment on whether the topic corresponding to the first sentence element and the topic corresponding to the second sentence element meet the preset correlation condition;

[0018] If the result of the first judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element is used as the benchmark threshold.

[0019] In an optional embodiment, the sentence element group further includes a third sentence element; and determining the subject of the sentence elements in the sentence element group further includes obtaining the subject corresponding to the third sentence element according to the semantics of the third sentence element;

[0020] The element group operation further includes:

[0021] If the result of the first judgment is no, performing a second judgment on whether the subject corresponding to the first sentence element and the subject corresponding to the third sentence element meet the preset correlation condition;

[0022] If the result of the second judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the third sentence element is used as the benchmark threshold.

[0023] In an optional embodiment, the performing of the first judgment on whether the topic corresponding to the first sentence element and the topic corresponding to the second sentence element satisfy the preset correlation condition includes:

[0024] Determining a second similarity value between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element according to the topic corresponding to the first sentence element and the topic corresponding to the second sentence element;

[0025] If the second similarity value is greater than or equal to a preset topic relevance threshold, the result of the first judgment is yes; otherwise, the result of the first judgment is no.

[0026] In an optional embodiment, obtaining the fusion topic feature value based on the intersection sentence elements included in the intersection sentence set includes:

[0027] The intersection sentence elements contained in the intersection sentence set are input into a pre-training model to obtain a fusion topic feature value; the pre-training model is obtained by pre-training the BYOL framework model for exploring its own potential through a self-supervised learning method.

[0028] In an optional embodiment, determining the abnormal sentence elements of the target text data according to the first similarity value between the subject of the edge sentence elements included in the edge sentence set and the fused subject feature value includes:

[0029] Vectorizing the subject of each edge sentence element contained in the edge sentence set to obtain an edge sentence subject vector for each edge sentence element;

[0030] Determine a similarity value between the edge sentence topic vector of each edge sentence element and the fusion topic feature value as a first similarity value of each edge sentence element;

[0031] The marginal sentence element whose first similarity value is smaller than a preset abnormal information identification threshold is determined as an abnormal sentence element of the target text data.

[0032] In an optional embodiment, before obtaining the first benchmark sentence element and the benchmark threshold based on the sentence elements in the to-be-processed sentence set of the target text data, the method further includes:

[0033] Acquire the target text data;

[0034] Sentence segmentation is performed on the target text data according to the set segmentation rules to obtain sentence information elements corresponding to the target text data; each sentence information element corresponds to a sentence;

[0035] Based on the sentence information element, the set of sentences to be processed is obtained.

[0036] The text data outlier detection method provided in the embodiment of the present application can obtain a first benchmark sentence element and a benchmark threshold value based on the sentence elements in the to-be-processed sentence set of the target text data; the benchmark threshold value is the distance between the first topic of the first benchmark sentence element and the second topic of the second benchmark sentence element; the second benchmark sentence element and the first benchmark sentence element are adjacent to each other in the to-be-processed sentence set, and the first topic and the second topic meet a preset correlation condition; based on the first benchmark sentence element, the benchmark threshold value and the to-be-processed sentence set, a preliminary sentence set is obtained; the preliminary sentence set contains the preliminary sentence elements in the to-be-processed sentence set; the distance between the preliminary sentence element and the first topic is not greater than the benchmark threshold value Based on the benchmark threshold and the preliminary selected sentence elements included in the preliminary selected sentence set, an intersection sentence set corresponding to the preliminary selected sentence set is obtained; the intersection sentence set includes the intersection sentence elements in the preliminary selected sentence set; the distance between the intersection sentence subject of the intersection sentence element and the preliminary selected sentence subject of any of the preliminary selected sentence elements is not greater than the benchmark threshold; based on the intersection sentence elements included in the intersection sentence set, a fusion topic feature value is obtained; based on the first similarity value between the subject of the edge sentence elements included in the edge sentence set and the fusion topic feature value, the abnormal sentence elements of the target text data are determined; the edge sentence set is determined based on the elements in the preliminary selected sentence set other than the intersection sentence set and the first benchmark sentence element. The method obtains a first benchmark sentence element and a benchmark threshold based on the sentence elements in the to-be-processed sentence set of the target text data, then obtains a preliminary selected sentence set and an intersection sentence set corresponding to the preliminary selected sentence set based on the first benchmark sentence element and the benchmark threshold, and obtains a fusion topic feature value, thereby determining the abnormal sentence elements of the target text data, thereby detecting abnormal values ​​of the text data and improving the comprehensive quality of the text data.

[0037] In a second aspect, an embodiment of the present application further provides a device for detecting outliers in text data, the device comprising:

[0038] A benchmark determination unit is configured to obtain a first benchmark sentence element and a benchmark threshold value based on sentence elements in a set of sentences to be processed of target text data; the benchmark threshold value is a distance between a first topic of the first benchmark sentence element and a second topic of a second benchmark sentence element; the second benchmark sentence element and the first benchmark sentence element are adjacent to each other in the set of sentences to be processed, and the first topic and the second topic meet a preset correlation condition;

[0039] A first selection unit is configured to obtain a preliminary selection sentence set based on the first benchmark sentence element, the benchmark threshold, and the set of sentences to be processed; the preliminary selection sentence set includes the preliminary selection sentence element in the set of sentences to be processed; and the distance between the preliminary selection sentence element and the first topic is not greater than the benchmark threshold;

[0040] A second selection unit is configured to obtain, based on the reference threshold and the preliminary selected sentence elements included in the preliminary selected sentence set, an intersection sentence set corresponding to the preliminary selected sentence set; the intersection sentence set includes the intersection sentence elements in the preliminary selected sentence set; and the distance between the intersection sentence subject of the intersection sentence element and the preliminary selected sentence subject of any of the preliminary selected sentence elements is not greater than the reference threshold;

[0041] a fusion calculation unit, configured to obtain a fusion topic feature value based on the intersection sentence elements contained in the intersection sentence set;

[0042] A result determination unit is used to determine the abnormal sentence elements of the target text data based on the first similarity value between the subject of the edge sentence elements contained in the edge sentence set and the fusion subject feature value; the edge sentence set is determined based on the elements in the preliminary sentence set except the intersection sentence set and the first benchmark sentence element.

[0043] In an optional embodiment, the benchmark determination unit is specifically configured to:

[0044] Among the sentence elements included in the to-be-processed sentence set of the target text data, any adjacent preset number of the sentence elements are read one by one to obtain a sentence element group including a first sentence element and a second sentence element. For each sentence element group obtained, the following element group operation is performed until a first benchmark sentence element and a benchmark threshold are obtained:

[0045] For a currently obtained sentence element group, determining the subject of the sentence elements in the sentence element group; determining the subject of the sentence elements in the sentence element group includes obtaining the subject corresponding to the first sentence element according to the semantics of the first sentence element in the sentence element group, and obtaining the subject corresponding to the second sentence element according to the semantics of the second sentence element;

[0046] Performing a first judgment on whether the topic corresponding to the first sentence element and the topic corresponding to the second sentence element meet the preset correlation condition;

[0047] If the result of the first judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element is used as the benchmark threshold.

[0048] In an optional embodiment, the sentence element group further includes a third sentence element; and the benchmark determination unit is further configured to:

[0049] For a currently obtained sentence element group, obtaining a topic corresponding to the third sentence element according to the semantics of the third sentence element;

[0050] The benchmark determination unit is further configured to:

[0051] If the result of the first judgment is no, performing a second judgment on whether the subject corresponding to the first sentence element and the subject corresponding to the third sentence element meet the preset correlation condition;

[0052] If the result of the second judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the third sentence element is used as the benchmark threshold.

[0053] In an optional embodiment, the benchmark determination unit is specifically configured to:

[0054] Determining a second similarity value between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element according to the topic corresponding to the first sentence element and the topic corresponding to the second sentence element;

[0055] If the second similarity value is greater than or equal to a preset topic relevance threshold, the result of the first judgment is yes; otherwise, the result of the first judgment is no.

[0056] In an optional embodiment, the fusion calculation unit is specifically configured to:

[0057] The intersection sentence elements contained in the intersection sentence set are input into a pre-training model to obtain a fusion topic feature value; the pre-training model is obtained by pre-training the BYOL framework model for exploring its own potential through a self-supervised learning method.

[0058] In an optional embodiment, the result determination unit is specifically configured to:

[0059] Vectorizing the subject of each edge sentence element contained in the edge sentence set to obtain an edge sentence subject vector for each edge sentence element;

[0060] Determine a similarity value between the edge sentence topic vector of each edge sentence element and the fusion topic feature value as a first similarity value of each edge sentence element;

[0061] The marginal sentence element whose first similarity value is smaller than a preset abnormal information identification threshold is determined as an abnormal sentence element of the target text data.

[0062] In an optional embodiment, the device further includes:

[0063] A text input unit, used to obtain the target text data;

[0064] A sentence segmentation unit is used to segment the target text data into sentences according to a set segmentation rule to obtain sentence information elements corresponding to the target text data; each sentence information element corresponds to a sentence;

[0065] The sentence set construction unit is used to obtain the sentence set to be processed based on the sentence information element.

[0066] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the text data outlier detection method of the first aspect is implemented.

[0067] In a fourth aspect, an embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the processor implements the text data outlier detection method of the first aspect.

[0068] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; when the processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device performs the steps of any one of the above-mentioned text data outlier detection methods.

[0069] The technical effects brought about by any one of the implementation methods in the second to fifth aspects can refer to the technical effects brought about by the implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0071] Figure 1 A flowchart of a method for detecting outliers in text data provided in an embodiment of the present application;

[0072] Figure 2 A schematic diagram of a flow chart for determining a first benchmark sentence element and a benchmark threshold value in a method for detecting outliers in text data provided in an embodiment of the present application;

[0073] Figure 3 A flowchart of determining abnormal sentence elements in a method for detecting outliers in text data provided in an embodiment of the present application;

[0074] Figure 4 A flowchart of another method for detecting outliers in text data provided in an embodiment of the present application;

[0075] Figure 5 A schematic diagram of the structure of a text data outlier detection device provided in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of the structure of another device for detecting outliers in text data provided in an embodiment of the present application;

[0077] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0079] The following explains some of the terms used in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0080] (1) BYOL (Bootstrap Your Own Latent) Framework Model: The BYOL framework model is based on the BYOL algorithm. The BYOL algorithm relies on two neural networks, the online network and the target network. The online network and the target network interact and learn from each other. During training, the online network continuously approaches the target network. The target network uses momentum to update its parameters, which is equivalent to slowly moving closer to the online network. This mutual approximation improves the overall performance of the network.

[0081] (2) Word2Vec model: Google's open source word vector tool, which can convert words into word vectors by using the semantic relationship between words in text data, and can also identify words by using the semantic distance relationship between word vectors.

[0082] In order to improve the overall quality of text data and facilitate post-processing of text data, the present invention provides a method and device for detecting outliers in text data. In order to better understand the technical solution provided by the present invention, the basic principle of the solution is briefly explained here.

[0083] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0084] The technical solutions provided by the embodiments of the present application are described below with reference to the accompanying drawings.

[0085] In practical applications, in unstructured text datasets, there may be cases where documents do not belong to the text topic, such as scientific articles appearing in political news datasets, or fax covers mixed with medical records, which will affect the subsequent text processing results.

[0086] In order to avoid the situation where a document does not belong to the text topic, there is a practical need to detect and filter out abnormal information in the document that does not belong to the text topic.

[0087] Currently, existing outlier detection tools, although they can implement outlier detection, do not support text data as input, or can only be used for outlier detection of unstructured data such as images.

[0088] In the related art, there is a lack of outlier detection methods for text data, and the overall quality of text data is poor. Based on this, how to implement outlier detection in text data is a technical problem that those skilled in the art urgently need to solve.

[0089] In view of this, an embodiment of the present application provides a method and device for detecting outliers in text data, wherein the method for detecting outliers in text data can obtain a first benchmark sentence element and a benchmark threshold based on the sentence elements in the to-be-processed sentence set of the target text data; obtain a preliminary sentence set based on the first benchmark sentence element, the benchmark threshold and the to-be-processed sentence set; obtain an intersection sentence set corresponding to the preliminary sentence set based on the benchmark threshold and the preliminary sentence elements contained in the preliminary sentence set; obtain a fusion topic feature value based on the intersection sentence elements contained in the intersection sentence set; determine the abnormal sentence elements of the target text data based on the first similarity value between the topic of the edge sentence elements contained in the edge sentence set and the fusion topic feature value; the edge sentence set is determined based on the elements in the preliminary sentence set except the intersection sentence set and the first benchmark sentence element. The method obtains a first benchmark sentence element and a benchmark threshold value based on the sentence elements in the to-be-processed sentence set of the target text data, and then obtains a preliminary sentence set and an intersection sentence set corresponding to the preliminary sentence set based on the first benchmark sentence element and the benchmark threshold value, and obtains a fusion theme feature value, thereby determining the abnormal sentence elements of the target text data, realizing the detection of abnormal values ​​of the text data, and being able to improve the comprehensive quality of the text data.

[0090] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0091] The following further explains the text data outlier detection method provided in the embodiment of the present application. The text data outlier detection method provided in the present application is used to upgrade the system to be upgraded to the target system, such as Figure 1 As shown, the following steps are included:

[0092] Step S101 : obtaining a first benchmark sentence element and a benchmark threshold value according to sentence elements in a to-be-processed sentence set of target text data.

[0093] Among them, the benchmark threshold is the distance between the first topic of the first benchmark sentence element and the second topic of the second benchmark sentence element; the second benchmark sentence element and the first benchmark sentence element are adjacent to each other in the set of sentences to be processed, and the first topic and the second topic meet the preset correlation conditions.

[0094] For example, according to the sentence set to be processed {Sentence i Sentence element in} i , get the first benchmark sentence element S_1 and the benchmark threshold Base_Value_1. Among them, the first benchmark sentence element S_1 is the distance between the first subject Subject_1 of the first benchmark sentence element S_1 and the second subject Subject_2 of the second benchmark sentence element S_2; the second benchmark sentence element S_2 and the first benchmark sentence element S_1 are in the sentence set to be processed {Sentence i}, and the first topic Subject_1 and the second topic Subject_2 meet the preset correlation conditions.

[0095] In an optional embodiment, a first benchmark sentence element and a benchmark threshold are obtained based on the sentence elements in the to-be-processed sentence set of the target text data, such as Figure 2 Shown, including:

[0096] Step S201 : Reading any preset number of adjacent sentence elements in sequence from the sentence elements included in the to-be-processed sentence set of the target text data to obtain a sentence element group including a first sentence element and a second sentence element.

[0097] Wherein, each time a sentence element group is obtained, the element group operation of the following steps is performed until the first benchmark sentence element and the benchmark threshold are obtained.

[0098] For example, in the sentence set to be processed of the target text data File_1 {Sentence i Sentence elements included i In the example, any adjacent preset number N of sentence elements are read one by one to obtain a sentence element group {Sub_sen j}. Sentence element group {Sub_sen j} includes the first sentence element Sub_sen1 and the second sentence element Sub_sen2. j}, perform the element group operation of the following steps S202 to S204 until the first benchmark sentence element S_1 and the benchmark threshold Base_Value_1 are obtained.

[0099] Step S202, for a currently obtained sentence element group, determine the subject of the sentence elements in a sentence element group; determining the subject of the sentence elements in a sentence element group includes obtaining the subject corresponding to the first sentence element according to the semantics of the first sentence element in a sentence element group, and obtaining the subject corresponding to the second sentence element according to the semantics of the second sentence element.

[0100] For example, for a sentence element group {Sub_sen j}, determine a sentence element group {Sub_sen j} in the subject of the sentence element; determine a sentence element group {Sub_sen j}, including the subject of a sentence element group {Sub_sen j}, the semantics of the first sentence element Sub_sen1 in the sentence is used to obtain the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1, and the semantics of the second sentence element Sub_sen2 is used to obtain the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2.

[0101] Step S203 , performing a first judgment on whether the topic corresponding to the first sentence element and the topic corresponding to the second sentence element meet a preset correlation condition.

[0102] Exemplarily, a first judgment is performed, specifically determining whether the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2 meet a preset correlation condition.

[0103] In a specific implementation, determining whether a topic satisfies a preset correlation condition with another topic can be performed by determining the similarity between the two topics. In the embodiments of the present application, the similarity between two topics refers to the similarity between the topic vectors corresponding to the two topics. The present application does not limit the specific form of the similarity between topic vectors. For example, it can be the cosine similarity, Euclidean distance, etc. between the topic vectors corresponding to the topics.

[0104] In some embodiments of the present application, the topic vector corresponding to the topic of a sentence element, for example, the topic vector corresponding to the topic of the first sentence element, may be obtained by processing the sentence element through a Word2Vec model to obtain the word vector corresponding to the sentence element, and then obtaining the word vector corresponding to the sentence element. In other embodiments of the present application, the topic vector corresponding to the topic of a sentence element may also be obtained by obtaining the topic information of the sentence element through a pre-trained neural network model and then performing a vectorization operation on the topic information.

[0105] In an optional embodiment, the process of performing the first judgment of whether the topic corresponding to the first sentence element and the topic corresponding to the second sentence element meet the preset relevance condition can be implemented by the following steps:

[0106] Step A1: Determine a second similarity value between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element based on the topic corresponding to the first sentence element and the topic corresponding to the second sentence element.

[0107] For example, vectorization processing is performed based on the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2, and a cosine similarity operation is performed to determine a second similarity value Similar_2 between the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2. The second similarity value Similar_2 may be a cosine similarity.

[0108] Step A2: If the second similarity value is greater than or equal to a preset topic relevance threshold, the result of the first judgment is yes; otherwise, the result of the first judgment is no.

[0109] Exemplarily, if the second similarity value Similar_2 is greater than or equal to a preset subject relevance threshold value subject_ralate_Value_2, the result of the first judgment is yes; otherwise, the result of the first judgment is no.

[0110] Step S204: If the result of the first judgment is yes, the first sentence element is used as a first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element is used as a benchmark threshold.

[0111] For example, if it is determined that the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2 meet the preset correlation condition, the first sentence element Sub_sen1 is used as the first benchmark sentence element S_1, and the distance between the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2 is used as the benchmark threshold Base_Value_1.

[0112] In an optional embodiment, the sentence element group further includes a third sentence element; in step S202, the process of determining the subject of a sentence element in a sentence element group further includes obtaining the subject corresponding to the third sentence element according to the semantics of the third sentence element;

[0113] When performing the element group operation of steps S202 to S204, the following steps are also included:

[0114] Step B1: If the result of the first judgment is no, a second judgment is performed to determine whether the topic corresponding to the first sentence element and the topic corresponding to the third sentence element meet a preset correlation condition.

[0115] For example, the sentence element group {Sub_sen j} also includes a third sentence element Sub_sen3; in step S202, the process of determining the subject of the sentence elements in a sentence element group also includes obtaining the subject Sub_sen3_Subject corresponding to the third sentence element Sub_sen3 according to the semantics of the third sentence element Sub_sen3. Taking the first judgment as an example of determining whether the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2 meet the preset correlation condition, in each sentence element group {Sub_sen j When performing the element group operation, the method further includes: if it is determined that the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen2_Subject corresponding to the second sentence element Sub_sen2 do not meet the preset correlation condition, performing a second judgment. The second judgment specifically determines whether the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen3_Subject corresponding to the third sentence element Sub_sen3 meet the preset correlation condition.

[0116] Step B2: If the result of the second judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the third sentence element is used as the benchmark threshold.

[0117] For example, if it is determined that the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen3_Subject corresponding to the third sentence element Sub_sen3 do not meet the preset correlation condition, the first sentence element Sub_sen1 is used as the first benchmark sentence element S_1, and the distance between the subject Sub_sen1_Subject corresponding to the first sentence element Sub_sen1 and the subject Sub_sen3_Subject corresponding to the third sentence element Sub_sen3 is used as the benchmark threshold Base_Value_1.

[0118] In some embodiments, during the process of executing the element group operation of steps S202 to S204 above, if the topic corresponding to the first sentence element and the topic corresponding to the second sentence element do not meet the preset correlation conditions, and the topic corresponding to the first sentence element and the topic corresponding to the third sentence element do not meet the preset correlation conditions, then the sentence element corresponding to the first sentence element in the set of sentences to be processed of the target text data is discarded.

[0119] In some embodiments, discarding the sentence element corresponding to the first sentence element in the set of sentences to be processed of the target text data may be deleting the sentence element corresponding to the first sentence element in the set of sentences to be processed of the target text data.

[0120] In other embodiments, the sentence elements corresponding to the first sentence element in the set of sentences to be processed of the target text data are discarded, and the sentence elements corresponding to the first sentence element in the set of sentences to be processed of the target text data can be specially marked; after the current sentence element group is processed, when a new sentence element group is obtained by executing step S201, specifically, any adjacent preset number of sentence elements without special marks are read one by one from the sentence elements included in the set of sentences to be processed of the target text data.

[0121] Step S102: obtaining a preliminary sentence set based on the first benchmark sentence element, the benchmark threshold, and the set of sentences to be processed.

[0122] The preliminary selected sentence set includes preliminary selected sentence elements in the sentence set to be processed; and the distance between the preliminary selected sentence elements and the first topic is not greater than a benchmark threshold.

[0123] In specific implementation, after calculating the first benchmark sentence element and the benchmark threshold, we can further select sentence elements from the set of sentences to be processed, with the first benchmark sentence element as the center, whose corresponding topics are less than or equal to the benchmark threshold in distance from the topic of the first benchmark sentence element, to form a preliminary sentence set.

[0124] For example, based on the first benchmark sentence element S_1, the benchmark threshold Base_Value_1 and the set of sentences to be processed {Sentence i}, get the preliminary sentence set {Sentence ′ i}. Among them, the primary sentence set {Sentence′ i}Contains the sentence set to be processed {Sentence i The primary sentence element Sentence′ in i ; Preliminary sentence element Sentence′ i Is the set of sentences to be processed {Sentence i}, the sentence elements whose distance from the subject of the first baseline sentence element S_1 is not greater than the baseline threshold Base_Value_1.

[0125] Step S103 : obtaining an intersection sentence set corresponding to the preliminary selected sentence set based on the benchmark threshold and the preliminary selected sentence elements contained in the preliminary selected sentence set.

[0126] The intersection sentence set includes intersection sentence elements in the preliminary sentence set; the distance between the intersection sentence topic of the intersection sentence element and the preliminary sentence topic of any preliminary sentence element is not greater than the benchmark threshold.

[0127] During specific implementation, each preliminary sentence element in the preliminary sentence set is selected one by one, and each time a preliminary sentence element is selected, the following operations are performed to obtain a target preliminary sentence set corresponding to the selected preliminary sentence element and including the target preliminary sentence element: the preliminary sentence topic corresponding to the selected preliminary sentence element is taken as the center, and the target preliminary sentence element is determined such that the distance between the corresponding topic and the topic corresponding to the selected preliminary sentence element is less than or equal to the benchmark threshold. The target preliminary sentence sets corresponding to each preliminary sentence element are respectively calculated for intersection, to obtain an intersection sentence set. The intersection sentence set contains the intersection sentence elements in the preliminary sentence set; the distance between the intersection sentence topic of the intersection sentence element and the preliminary sentence topic of any preliminary sentence element is not greater than the benchmark threshold.

[0128] For example, based on the baseline threshold Base_Value_1 and the primary sentence set {Sentence i} contains the primary sentence element Sentence′ i , and get the same sentence set as the primary sentence set {Sentence′i}corresponding intersection sentence set {Sentence" i}. Among them, the intersection sentence set {Sentence" i} contains the intersection sentence element Sentence" i ; Intersection sentence element Sentence" i , is the primary sentence set {Sentence′ i}, the preliminary sentence elements that meet the following conditions: the distance between the corresponding subject and the preliminary sentence subject Sentence'_Subject of any preliminary sentence element is not greater than the baseline threshold Base_Value_1. Intersection sentence element Sentence" i The subject can be recorded as the intersection sentence subject Sentence"_Subject, then the intersection sentence element Sentence" i The distance between the intersection sentence subject Sentence"_Subject and the preliminary sentence subject Sentence'_Subject of any preliminary sentence element is not greater than the benchmark threshold Base_Value_1.

[0129] Step S104: obtaining a fusion topic feature value based on the intersection sentence elements included in the intersection sentence set.

[0130] For example, based on the intersection sentence set {Sentence" i} contains the intersection sentence element Sentence" i , and obtain the fusion subject feature value Co_Subject_Info.

[0131] In the embodiment of the present application, the fused topic feature value may be a vector.

[0132] In an optional embodiment, a fusion topic feature value is obtained based on the intersection sentence elements contained in the intersection sentence set. Specifically, the fusion topic feature value can be obtained by inputting the intersection sentence elements contained in the intersection sentence set into a pre-trained model; the pre-trained model is obtained by pre-training the BYOL framework model through self-supervised learning.

[0133] For example, the intersection sentence set {Sentence" i} contains the intersection sentence element Sentence" i Input into the pre-trained model Co_Model_1 to obtain the fusion subject feature value Co_Subject_Info; the pre-trained model Co_Model_1 is obtained by pre-training the BYOL framework model through self-supervised learning.

[0134] The BYOL framework model is based on the BYOL algorithm. The BYOL algorithm relies on two neural networks, the online network and the target network. The online network and the target network interact and learn from each other. During training, the online network continuously approaches the target network, and the target network uses momentum to update its parameters, gradually aligning the target network with the online network. This mutual approximation effectively improves overall network performance. This pre-trained model can be used to obtain fused topic features corresponding to text data with unknown topics, improving the efficiency and accuracy of topic determination.

[0135] Step S105, determining abnormal sentence elements of the target text data based on the first similarity value between the subject of the edge sentence elements contained in the edge sentence set and the fusion subject feature value; wherein the edge sentence set is determined based on the elements in the preliminary sentence set except the intersection sentence set and the first benchmark sentence element.

[0136] During specific implementation, the edge sentence set is first determined, wherein the edge sentence set is composed of elements in the preliminary sentence set excluding the intersection sentence set and the first benchmark sentence element. Then, the similarity between the sentence topic vector of the subject of the edge sentence element contained in the edge sentence set and the fusion topic feature value is calculated respectively to obtain a first similarity value. If the first similarity value corresponding to an edge sentence element contained in the edge sentence set is less than a preset threshold value, the edge sentence element is regarded as an abnormal sentence element of the target text data, that is, the edge sentence element is an abnormal value. Furthermore, after the edge sentence element is regarded as an abnormal sentence element of the target text data, the edge sentence element can be output as a result of text data abnormal value detection.

[0137] For example, according to the edge sentence set {Off_Sentence" i} contains the edge sentence element Off_Sentence" i The first similarity value Similar_1 between the subject and the fusion subject feature value Co_Subject_Info is used to determine the abnormal sentence elements of the target text data File_1; wherein, the edge sentence set {Off_Sentence" i} is based on the primary sentence set {Sentence i}, except for the intersection sentence set {Sentence" i} and the first reference sentence element S_1.

[0138] In an optional embodiment, the process of determining abnormal sentence elements of the target text data according to the first similarity value between the subject of the edge sentence elements contained in the edge sentence set and the fusion subject feature value is as follows: Figure 3 As shown, this can be achieved by following the steps below:

[0139] Step S301 , vectorizing the topic of each edge sentence element included in the edge sentence set to obtain an edge sentence topic vector of each edge sentence element.

[0140] For example, the edge sentence set {Off_Sentence" i Each edge sentence element contained in}Off_Sentence" i The subject Off_Sentence"_Subject is vectorized to obtain each edge sentence element Off_Sentence" i The marginal sentence topic vector Off_Sentence"_Sub_vector.

[0141] In some embodiments, the topic of each edge sentence element included in the edge sentence set is vectorized to obtain a TF-IDF vector corresponding to the topic.

[0142] Step S302 : determining a similarity value between the edge sentence topic vector of each edge sentence element and the fusion topic feature value as a first similarity value of each edge sentence element.

[0143] For example, each edge sentence element Off_Sentence is determined i The similarity value between the edge sentence topic vector Off_Sentence"_Sub_vector and the fusion topic feature value Co_Subject_Info is used as the similarity value of each edge sentence element Off_Sentence" i The first similarity value Similar_1.

[0144] Step S303 : determining marginal sentence elements whose first similarity values ​​are less than a preset abnormal information identification threshold as abnormal sentence elements of the target text data.

[0145] For example, the marginal sentence element Off_Sentence whose first similarity value Similar_1 is less than the preset abnormal information identification threshold ordinary_pick_Value_3 is selected. i , determined to be an abnormal sentence element of the target text data File_1.

[0146] The text data outlier detection method provided by the above embodiment can obtain a first benchmark sentence element and a benchmark threshold based on the sentence elements in the to-be-processed sentence set of the target text data, and then obtain a preliminary sentence set and an intersection sentence set corresponding to the preliminary sentence set based on the first benchmark sentence element and the benchmark threshold, and obtain a fusion theme feature value, and then determine the abnormal sentence elements of the target text data, thereby realizing outlier detection of text data and improving the overall quality of text data.

[0147] Figure 4 FIG. 1 is a flow chart showing another method for detecting abnormal values ​​in text data provided by an embodiment of the present application. Figure 4 As shown, the text data outlier detection method includes the following steps:

[0148] Step S401: Acquire target text data.

[0149] Exemplarily, the target text data File_1 is acquired.

[0150] Step S402 , segmenting the target text data into sentences according to the set segmentation rules to obtain sentence information elements corresponding to the target text data; each sentence information element corresponds to a sentence.

[0151] In a specific implementation, after obtaining the target text data, the initial target text data is first split into sentences. In an embodiment of the present application, the target text data can be split into sentences using a sentence segmentation function that can implement a sentence segmentation function to obtain sentence information elements corresponding to the target text data; each sentence information element corresponds to a sentence.

[0152] For example, the target text data File_1 is segmented into sentences according to the set splitting rules to obtain the sentence information element Orignal_Sentence corresponding to the target text data File_1 i ; Each sentence information element Original_Sentence i A statement corresponding to the target text data File_1.

[0153] Step S403: obtaining a set of sentences to be processed based on the sentence information elements.

[0154] In a specific implementation, a set of sentences to be processed can be constructed based on the sentence information elements. The set of sentences to be processed can be constructed by performing a set construction based on the individual sentence information elements, or by performing a processing operation on the individual sentence information elements and then constructing a set based on the results of the processing operation to obtain the set of sentences to be processed.

[0155] In some embodiments, a set of sentences to be processed is obtained based on sentence information elements, which can be performed by performing sentence-level preprocessing on each sentence information element to obtain a preprocessing result for each sentence information element; and sentence information elements whose preprocessing results do not meet preset deletion control conditions are determined as sentence elements in the set of sentences to be processed.

[0156] In one embodiment, sentence-level preprocessing of each sentence information element may include language filtering based on a language control threshold, a target language, and preset rule conditions to filter sentences by the target language. The language control threshold is used to filter languages ​​that do not meet the preset rule conditions. The language control threshold can be determined based on actual needs or human experience. When language filtering is performed based on the language control threshold, the target language, and the preset rule conditions, all sentences whose feature values ​​obtained according to the preset rule conditions are lower than the language control threshold will be filtered out.

[0157] For example, assuming that only Chinese sentence information units need to be processed, the preset rule condition Rule_1 can represent "only filtering Chinese", assuming that the language control threshold is Lang_Value, when a sentence information unit Original_Sentence M When the eigenvalue result obtained according to Rule_1 is lower than Lang_Value, it represents the sentence information unit Orignal_Sentence M It is not Chinese, so the sentence information unit Orignal_Sentence whose feature value result is lower than Lang_Value can be deleted M .

[0158] In one embodiment, sentence-level preprocessing is performed on each sentence information element, and abnormal sentence information elements may be filtered out based on the similarity between sentence information elements. An abnormal sentence information element is a sentence information element whose abnormal similarity ratio exceeds a preset similarity ratio threshold; the abnormal similarity ratio is the ratio of a first value to a second value; the first value is the number of sentence information elements whose similarity to the current sentence information element is lower than a preset cross-control threshold; and the second value is the number of sentence information elements other than the current sentence information element.

[0159] Specifically, based on the similarity between sentence information elements, abnormal sentence information elements are filtered out. This can be done by first selecting an arbitrary sentence information element and determining the similarity between the arbitrary sentence information element and the remaining sentence information elements and calculating the similarity between the sentence vectors respectively; then, based on the similarity between the sentence vectors, determining the similarity abnormality rate corresponding to each sentence information element; then, based on the similarity abnormality rate corresponding to each sentence information element and a preset similarity ratio threshold, determining the sentence information element whose similarity abnormality rate exceeds the similarity ratio threshold as the abnormal sentence information element; finally, deleting the abnormal sentence information element from the sentence information elements.

[0160] In an embodiment of the present application, the similarity between sentence vectors may be cosine similarity between sentence vectors, Jaccard similarity between sentences, or the like.

[0161] Step S404 : obtaining a first benchmark sentence element and a benchmark threshold value according to the sentence elements in the to-be-processed sentence set of the target text data.

[0162] Among them, the benchmark threshold is the distance between the first topic of the first benchmark sentence element and the second topic of the second benchmark sentence element; the second benchmark sentence element and the first benchmark sentence element are adjacent to each other in the set of sentences to be processed, and the first topic and the second topic meet the preset correlation conditions.

[0163] Step S405 : obtaining a preliminary sentence set based on the first benchmark sentence element, the benchmark threshold, and the set of sentences to be processed.

[0164] The preliminary selected sentence set includes preliminary selected sentence elements in the sentence set to be processed; and the distance between the preliminary selected sentence elements and the first topic is not greater than a benchmark threshold.

[0165] Step S406 : obtaining an intersection sentence set corresponding to the preliminary selected sentence set based on the benchmark threshold and the preliminary selected sentence elements contained in the preliminary selected sentence set.

[0166] The intersection sentence set includes intersection sentence elements in the preliminary sentence set; the distance between the intersection sentence topic of the intersection sentence element and the preliminary sentence topic of any preliminary sentence element is not greater than the benchmark threshold.

[0167] Step S407: obtaining a fusion topic feature value based on the intersection sentence elements included in the intersection sentence set.

[0168] Step S408, determining abnormal sentence elements of the target text data based on the first similarity value between the subject of the edge sentence elements contained in the edge sentence set and the fusion subject feature value; wherein the edge sentence set is determined based on the elements in the preliminary sentence set except the intersection sentence set and the first benchmark sentence element.

[0169] The implementation process of steps S404 to S408 can be performed with reference to the implementation process of the above embodiment, and will not be described in detail here.

[0170] The method of the above embodiment performs sentence-level preprocessing on each sentence information element after sentence splitting to obtain the preprocessing result of each sentence information element; the sentence information element whose preprocessing result does not meet the preset deletion control condition is determined as a sentence element in the set of sentences to be processed, which can effectively reduce the scope of subsequent text data outlier detection, thereby significantly improving the rate and efficiency of text data outlier detection.

[0171] Based on the same inventive concept, the present invention also provides a device for detecting outliers in text data. Since this device corresponds to the method for detecting outliers in text data in the present invention, and the principles for solving the problem are similar to those of the method, the implementation of this device can be referenced to the implementation of the aforementioned method, and any repetitions will not be repeated.

[0172] Figure 5 A schematic diagram of the structure of a text data outlier detection device provided in an embodiment of the present application is shown. Figure 5 As shown, it includes: a reference determination unit 501, a first selection unit 502, a second selection unit 503, a fusion calculation unit 504 and a result determination unit 505.

[0173] The benchmark determination unit 501 is configured to obtain a first benchmark sentence element and a benchmark threshold value based on sentence elements in the to-be-processed sentence set of the target text data; the benchmark threshold value is the distance between the first topic of the first benchmark sentence element and the second topic of the second benchmark sentence element; the second benchmark sentence element is adjacent to the first benchmark sentence element in the to-be-processed sentence set, and the first topic and the second topic meet a preset correlation condition;

[0174] A first selection unit 502 is configured to obtain a preliminary selection sentence set based on the first benchmark sentence element, the benchmark threshold, and the set of sentences to be processed; the preliminary selection sentence set includes the preliminary selection sentence element in the set of sentences to be processed; and the distance between the preliminary selection sentence element and the first topic is not greater than the benchmark threshold;

[0175] The second selection unit 503 is configured to obtain an intersection sentence set corresponding to the preliminary selected sentence set based on the reference threshold and the preliminary selected sentence elements included in the preliminary selected sentence set; the intersection sentence set includes the intersection sentence elements in the preliminary selected sentence set; and the distance between the intersection sentence topic of the intersection sentence element and the preliminary selected sentence topic of any preliminary selected sentence element is not greater than the reference threshold;

[0176] A fusion calculation unit 504 is used to obtain a fusion topic feature value based on the intersection sentence elements included in the intersection sentence set;

[0177] The result determination unit 505 is used to determine the abnormal sentence elements of the target text data based on the first similarity value between the subject of the edge sentence elements contained in the edge sentence set and the fusion subject feature value; the edge sentence set is determined based on the elements in the preliminary sentence set except the intersection sentence set and the first benchmark sentence element.

[0178] In an optional embodiment, the benchmark determination unit 501 is specifically configured to:

[0179] Among the sentence elements included in the to-be-processed sentence set of the target text data, any adjacent preset number of sentence elements are read one by one to obtain a sentence element group including a first sentence element and a second sentence element. For each sentence element group obtained, the following element group operation is performed until a first benchmark sentence element and a benchmark threshold are obtained:

[0180] For a currently obtained sentence element group, determining the subject of the sentence elements in the sentence element group; determining the subject of the sentence elements in the sentence element group includes obtaining a subject corresponding to the first sentence element according to the semantics of the first sentence element in the sentence element group, and obtaining a subject corresponding to the second sentence element according to the semantics of the second sentence element;

[0181] Performing a first judgment on whether a topic corresponding to the first sentence element and a topic corresponding to the second sentence element meet a preset relevance condition;

[0182] If the result of the first judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element is used as the benchmark threshold.

[0183] In an optional embodiment, the sentence element group further includes a third sentence element; and the benchmark determining unit 501 is further configured to:

[0184] For a currently obtained sentence element group, obtaining a topic corresponding to the third sentence element according to the semantics of the third sentence element;

[0185] The benchmark determination unit 501 is further configured to:

[0186] If the result of the first judgment is no, then performing a second judgment on whether the subject corresponding to the first sentence element and the subject corresponding to the third sentence element meet a preset relevance condition;

[0187] If the result of the second judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the third sentence element is used as the benchmark threshold.

[0188] In an optional embodiment, the benchmark determination unit 501 is specifically configured to:

[0189] Determine a second similarity value between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element based on the topic corresponding to the first sentence element and the topic corresponding to the second sentence element;

[0190] If the second similarity value is greater than or equal to the preset topic relevance threshold, the result of the first judgment is yes; otherwise, the result of the first judgment is no.

[0191] In an optional embodiment, the fusion calculation unit 504 is specifically configured to:

[0192] The intersection sentence elements contained in the intersection sentence set are input into the pre-training model to obtain the fusion topic feature value; the pre-training model is obtained by pre-training the BYOL framework model for exploring its own potential through self-supervised learning.

[0193] In an optional embodiment, the result determination unit 505 is specifically configured to:

[0194] Vectorize the topic of each edge sentence element contained in the edge sentence set to obtain the edge sentence topic vector of each edge sentence element;

[0195] Determine a similarity value between the edge sentence topic vector of each edge sentence element and the fusion topic feature value as a first similarity value of each edge sentence element;

[0196] The marginal sentence elements whose first similarity value is less than a preset abnormal information identification threshold are determined as abnormal sentence elements of the target text data.

[0197] In an optional embodiment, as Figure 6 As shown, the device also includes:

[0198] A text input unit 601 is used to obtain target text data;

[0199] Sentence segmentation unit 602, configured to segment the target text data into sentences according to a set segmentation rule, and obtain sentence information elements corresponding to the target text data; each sentence information element corresponds to a sentence;

[0200] The sentence set construction unit 603 is used to obtain a sentence set to be processed based on the sentence information element.

[0201] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. The electronic device can be used for detecting abnormal values ​​in text data. In one embodiment, the electronic device can be a server. In this embodiment, the structure of the electronic device can be as follows: Figure 7 As shown, it includes a memory 701 , a communication module 703 and one or more processors 702 .

[0202] Memory 701 is used to store computer programs executed by processor 702. Memory 701 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0203] Memory 701 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 701 may be a combination of the above memories.

[0204] The processor 702 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 702 is configured to implement the above-mentioned text data outlier detection method when calling the computer program stored in the memory 701 .

[0205] The communication module 703 is used to communicate with electronic devices and other terminals or servers.

[0206] The specific connection medium between the memory 701, the communication module 703 and the processor 702 is not limited in the embodiment of the present application. Figure 7 In the embodiment, the memory 701 and the processor 702 are connected via a bus 704. Figure 7 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus 704 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0207] An embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored. The computer executable instructions are used to implement the text data outlier detection method described in any embodiment of the present application.

[0208] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to perform the text data outlier detection method of any of the above embodiments.

[0209] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0210] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for detecting outliers in text data, characterized in that: include: Obtaining a first benchmark sentence element and a benchmark threshold according to sentence elements in a to-be-processed sentence set of target text data; The reference threshold is the distance between the first topic of the first reference sentence element and the second topic of the second reference sentence element; The second benchmark sentence element and the first benchmark sentence element are adjacent to each other in the set of sentences to be processed, and the first topic and the second topic meet a preset correlation condition; Based on the first benchmark sentence element, the benchmark threshold and the set of sentences to be processed, a preliminary selected sentence set is obtained; the preliminary selected sentence set includes the preliminary selected sentence elements in the set of sentences to be processed; The distance between the preliminary selected sentence element and the first topic is not greater than the benchmark threshold; Based on the reference threshold and the preliminary selected sentence elements included in the preliminary selected sentence set, an intersection sentence set corresponding to the preliminary selected sentence set is obtained; the intersection sentence set includes the intersection sentence elements in the preliminary selected sentence set; The distance between the intersection sentence topic of the intersection sentence element and the preliminary sentence topic of any of the preliminary sentence elements is not greater than the reference threshold; Obtaining a fusion topic feature value based on the intersection sentence elements contained in the intersection sentence set; Determining abnormal sentence elements of the target text data according to a first similarity value between a topic of an edge sentence element included in the edge sentence set and the fused topic feature value; The edge sentence set is determined based on the elements in the preliminary selected sentence set excluding the intersection sentence set and the first reference sentence elements.

2. The method according to claim 1, characterized in that The step of obtaining a first benchmark sentence element and a benchmark threshold based on sentence elements in the to-be-processed sentence set of the target text data includes: Among the sentence elements included in the to-be-processed sentence set of the target text data, any adjacent preset number of the sentence elements are read one by one to obtain a sentence element group including a first sentence element and a second sentence element. For each sentence element group obtained, the following element group operation is performed until a first benchmark sentence element and a benchmark threshold are obtained: For a currently obtained sentence element group, determining the subject of the sentence elements in the sentence element group; determining the subject of the sentence elements in the sentence element group includes obtaining the subject corresponding to the first sentence element according to the semantics of the first sentence element in the sentence element group, and obtaining the subject corresponding to the second sentence element according to the semantics of the second sentence element; Performing a first judgment on whether the topic corresponding to the first sentence element and the topic corresponding to the second sentence element meet the preset correlation condition; If the result of the first judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element is used as the benchmark threshold.

3. The method according to claim 2, characterized in that The sentence element group further includes a third sentence element; determining the subject of the sentence elements in the sentence element group further includes obtaining the subject corresponding to the third sentence element according to the semantics of the third sentence element; The element group operation further includes: If the result of the first judgment is no, performing a second judgment on whether the subject corresponding to the first sentence element and the subject corresponding to the third sentence element meet the preset correlation condition; If the result of the second judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the third sentence element is used as the benchmark threshold.

4. The method according to claim 2, characterized in that The performing of the first judgment of whether the subject corresponding to the first sentence element and the subject corresponding to the second sentence element meet the preset correlation condition includes: Determining a second similarity value between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element according to the topic corresponding to the first sentence element and the topic corresponding to the second sentence element; If the second similarity value is greater than or equal to a preset topic relevance threshold, the result of the first judgment is yes; otherwise, the result of the first judgment is no.

5. The method according to claim 1, wherein The step of obtaining a fusion topic feature value based on the intersection sentence elements contained in the intersection sentence set includes: The intersection sentence elements contained in the intersection sentence set are input into a pre-training model to obtain a fusion topic feature value; the pre-training model is obtained by pre-training the BYOL framework model for exploring its own potential through a self-supervised learning method.

6. The method according to claim 1, wherein The determining of abnormal sentence elements of the target text data according to the first similarity value between the subject of the edge sentence elements included in the edge sentence set and the fused subject feature value includes: Vectorizing the subject of each edge sentence element contained in the edge sentence set to obtain an edge sentence subject vector for each edge sentence element; Determine a similarity value between the edge sentence topic vector of each edge sentence element and the fusion topic feature value as a first similarity value of each edge sentence element; The marginal sentence element whose first similarity value is smaller than a preset abnormal information identification threshold is determined as an abnormal sentence element of the target text data.

7. The method according to claim 1, characterized in that Before obtaining the first benchmark sentence element and the benchmark threshold according to the sentence elements in the to-be-processed sentence set of the target text data, the method further includes: Acquire the target text data; Sentence segmentation is performed on the target text data according to the set segmentation rules to obtain sentence information elements corresponding to the target text data; each sentence information element corresponds to a sentence; Based on the sentence information element, the set of sentences to be processed is obtained.

8. A device for detecting abnormal values ​​in text data, characterized in that: The device comprises: A benchmark determination unit is configured to obtain a first benchmark sentence element and a benchmark threshold value based on sentence elements in a set of sentences to be processed of target text data; the benchmark threshold value is a distance between a first topic of the first benchmark sentence element and a second topic of a second benchmark sentence element; the second benchmark sentence element and the first benchmark sentence element are adjacent to each other in the set of sentences to be processed, and the first topic and the second topic meet a preset correlation condition; A first selection unit is configured to obtain a preliminary selection sentence set based on the first benchmark sentence element, the benchmark threshold, and the set of sentences to be processed; the preliminary selection sentence set includes the preliminary selection sentence element in the set of sentences to be processed; and the distance between the preliminary selection sentence element and the first topic is not greater than the benchmark threshold; A second selection unit is configured to obtain, based on the reference threshold and the preliminary selected sentence elements included in the preliminary selected sentence set, an intersection sentence set corresponding to the preliminary selected sentence set; the intersection sentence set includes the intersection sentence elements in the preliminary selected sentence set; and the distance between the intersection sentence subject of the intersection sentence element and the preliminary selected sentence subject of any of the preliminary selected sentence elements is not greater than the reference threshold; a fusion calculation unit, configured to obtain a fusion topic feature value based on the intersection sentence elements contained in the intersection sentence set; A result determination unit is used to determine the abnormal sentence elements of the target text data based on the first similarity value between the subject of the edge sentence elements contained in the edge sentence set and the fusion subject feature value; the edge sentence set is determined based on the elements in the preliminary sentence set except the intersection sentence set and the first benchmark sentence element.

9. The device according to claim 8, characterized in that The benchmark determination unit is specifically configured to: Among the sentence elements included in the to-be-processed sentence set of the target text data, any adjacent preset number of the sentence elements are read one by one to obtain a sentence element group including a first sentence element and a second sentence element. For each sentence element group obtained, the following element group operation is performed until a first benchmark sentence element and a benchmark threshold are obtained: For a currently obtained sentence element group, determining the subject of the sentence elements in the sentence element group; determining the subject of the sentence elements in the sentence element group includes obtaining the subject corresponding to the first sentence element according to the semantics of the first sentence element in the sentence element group, and obtaining the subject corresponding to the second sentence element according to the semantics of the second sentence element; Performing a first judgment on whether the topic corresponding to the first sentence element and the topic corresponding to the second sentence element meet the preset correlation condition; If the result of the first judgment is yes, the first sentence element is used as the first benchmark sentence element, and the distance between the topic corresponding to the first sentence element and the topic corresponding to the second sentence element is used as the benchmark threshold.

10. The device according to claim 8, characterized in that The fusion computing unit is specifically used for: The intersection sentence elements contained in the intersection sentence set are input into a pre-training model to obtain a fusion topic feature value; the pre-training model is obtained by pre-training the BYOL framework model in a self-supervised learning manner.

11. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the method for detecting outliers in text data according to any one of claims 1 to 7 is implemented.

12. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the method for detecting outliers in text data according to any one of claims 1 to 7 is implemented.

13. A computer program product, characterized in that It includes computer instructions, which are stored in a computer-readable storage medium; when a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device executes the text data outlier detection method according to any one of claims 1 to 7.

Images