Method, System and Medium for Automatically Generating Regular Expressions for Similar Texts

Abstract

This invention proposes a method, system, and medium for automatically generating regular expressions for similar texts, which relate to the fields of natural language processing and information security. For input similar texts, a text embedding representation method is used to obtain embedded representations of the similar texts. Similar texts of the same type are then clustered into the same category. For each category of similar texts, common and different parts are extracted. The extracted common and different parts are combined to generate a regular expression corresponding to each category of similar texts. This invention can efficiently and quickly process large amounts of similar texts in batches, without requiring human intervention, and can generate regular expressions for each category of similar texts.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and information security, and in particular to a method, system and medium for automatically generating regular expressions for similar texts. Background Art

[0002] Regular expressions offer strong descriptive power and flexible syntax. With the rapid evolution of online content, regular expressions need to be updated quickly to accurately identify and filter new services and content. However, regular expression syntax is complex, making construction challenging and requiring strong professional knowledge. Therefore, automatically generating regular expressions from text data is an urgent problem to be solved. Currently, there are two main areas of research in automatic regular expression generation: generating regular expressions from natural language descriptions, and constructing regular expressions for unstructured entities.

[0003] 1. Generate regular expressions from natural language descriptions. This type of research takes as input a textual description of a regular expression, and outputs the corresponding regular expression. For example, if the input natural language description is "contains 3 or more uppercase letters," then the goal is to generate "[A-Z]" for "uppercase letters," "{3,}" for "3 or more," and ".*" for "contains." The resulting regular expression is "(.*[A-Z].*){3,}."

[0004] Existing methods generate regular expressions by learning to pair words and regular expressions from natural language descriptions. One approach involves semantic parsing, while the other involves using neural network models. Semantic parsing constructs a dictionary to pair words or phrases with regular expressions. This dictionary is initialized by pairing each sentence in the training data with its associated complete regular expression. These vocabulary entries are then iteratively refined, considering all possible ways to split regular sentences and all possible ways to split sentences. Neural network models can better learn the correspondence between words and regular expressions. The encoder inputs a natural language word sequence, and the decoder outputs regular expression characters.

[0005] 2. Generate regular expressions for entities. Most real-world entities follow underlying grammatical rules, such as email addresses, phone numbers, domain names, and postal codes. Therefore, to automatically generate regular expressions from given entity examples, some research begins with classification. Based on positive and negative sample strings—strings that are described by the regular expression language and strings that are not—then infer whether other strings belong to the entity class.

[0006] In order to construct regular expressions for non-specific entities, some methods initialize a regular expression with high precision and low recall, and continuously improve the precision and recall of the initial regular expression by identifying and matching positive and negative samples. Other methods hope to directly induce regular expressions from positive and negative samples without inputting the initial regular expression. A commonly used method is to use heuristic and random search based on genetic algorithms (GP) to find the optimal solution. The initial regular expression population is generated through samples, and genetic operations (such as mutation and crossover) are used to iteratively modify the regular expression until the set iteration number is reached, or the fitness function is used to determine whether the optimal regular expression is generated.

[0007] Existing technical solutions mainly generate regular expressions based on natural language descriptions and regular expressions for entities, and their shortcomings are as follows: 1. Generating regular expressions based on natural language descriptions realizes the correspondence between natural language and regular expressions. However, due to the grammatical complexity of regular expressions, it is currently only applicable to short, formulaic natural language descriptions to generate simple regular expressions. 2. Generating regular expressions from examples of specific entities is oriented towards specific categories of text and requires certain manual input, such as initial regular expressions, or manually labeled positive and negative samples. These methods do not fully realize the process of automatically generating regular expressions from text. In addition, when extracting the common parts of each type of text, the problem can be formulated as solving the longest common subsequence of multiple strings, but the existing methods have high time complexity and are not applicable to arbitrary text datasets with large alphabets, long texts, and multiple data. Summary of the Invention

[0008] The purpose of the present invention is to provide a method, system and medium for automatically generating regular expressions for similar texts, which can efficiently and quickly process a large number of similar texts in batches without human intervention, and generate regular expressions for each type of similar text, thereby improving the accuracy of content recognition using regular expressions.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] The present invention provides a method for automatically generating regular expressions for similar texts, which comprises the following steps:

[0011] For similar texts input, a text embedding representation method is used to obtain an embedded representation of the similar texts; and a clustering method is used to cluster similar texts of the same type into the same category based on the embedded representation.

[0012] For each type of similar text, the common part and the difference part are extracted. The method for extracting the common part is to solve the longest common subsequence of all similar text strings in each type to obtain the longest common part of the similar text in that type. The method for extracting the difference part is to calculate the difference of the difference part through information entropy. If the information entropy is less than a set threshold, the content and length of the part of text are counted and used as the extracted difference part.

[0013] The extracted common parts and difference parts are combined to generate regular expressions corresponding to each type of similar text.

[0014] Furthermore, the text embedding representation method is a TF-IDF method, a Word2vec method or a BERT method.

[0015] Furthermore, the text embedding representation method is preferably a Word2vec method.

[0016] Furthermore, in the Word2vec method, a word is predicted based on its context by training a Skip-gram model, and a word is predicted based on its context by training a CBOW model.

[0017] Furthermore, the clustering method is a density-based method, a partition-based method or a hierarchical clustering method, wherein the density-based method selects the DBSCAN algorithm, the partition-based method selects the K-Means algorithm, and the hierarchical clustering method selects the divisive clustering method or the agglomerative clustering method.

[0018] Furthermore, the clustering method is preferably an agglomerative clustering method in a hierarchical clustering method.

[0019] Furthermore, the method for solving the longest common subsequence is a randomized MLCS method, which includes the following steps:

[0020] First, all strings of similar text in each category are randomly arranged;

[0021] Secondly, divide the strings into groups of two in order;

[0022] Again use the dynamic programming algorithm to calculate the longest common subsequence of the two strings in each group;

[0023] Finally, the results obtained in the previous step are divided into groups of two in order, and the dynamic programming algorithm is used to calculate the longest common subsequence of the two strings in each group. This step is repeated until the longest common subsequence of all similar text strings in this class is obtained.

[0024] Furthermore, in the method of extracting the difference part, the difference part is replaced by the symbol .*, which can match any content. Information entropy is used to describe the uncertainty of the length of the .* matching part. If the information entropy is less than a set threshold, the text content and length of the matching object of the .* matching part are counted.

[0025] The present invention also provides a regular expression automatic generation system for similar texts, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above method when executing the program.

[0026] The present invention also provides a computer-readable storage medium storing a computer program, which implements the steps of the above method when executed by a processor.

[0027] The beneficial effects achieved by the present invention are as follows:

[0028] (1) The present invention can classify texts with similar structures and contents into the same category through text clustering, which facilitates the subsequent extraction of common subsequences from similar texts of the same type.

[0029] (2) When generating regular expressions, by extracting common parts and different parts, common and individual content can be extracted from similar texts, which greatly improves the accuracy and applicability of regular expressions.

[0030] (3) By converting the extraction of the common part into the problem of solving the longest common sequence, and further converting it into the problem of solving the longest common subsequence of each string, the difficulty of extracting the common part can be greatly simplified. Compared with the traditional method of solving the longest common subsequence, which requires a small number of strings, short sequences, a small amount of text, and a small alphabet, and has high complexity, the present invention can solve the longest common subsequence of strings through a specially designed randomized MLCS method. It does not require the number of strings, sequence length, amount of text, and alphabet size, and has low complexity, completely overcoming the related defects of the traditional method of solving the longest common subsequence.

[0031] (4) When extracting the difference part, the present invention measures the certainty of the difference part by introducing information entropy, which can more accurately determine the text information of the difference part. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an example flow chart of the method for automatically generating regular expressions according to an embodiment of the present invention.

[0033] Figure 2 This is a flowchart of text clustering according to an embodiment of the present invention.

[0034] Figure 34 is a flow chart of the randomized MLCS method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0036] This embodiment provides a method for automatically generating regular expressions for similar texts. The method generates regular expressions through two stages: text clustering and generating regular expressions. The processing flow for a specific example is as follows: Figure 1 As shown in Figure 2. Similar texts share commonalities in both content and format. Therefore, we first cluster similar texts using text representation and clustering methods. Then, for each set of similar texts, we extract their common and different parts, and finally combine these two to form the final regular expression. The following describes the technical solution of this method in detail.

[0037] 1: Text Clustering

[0038] The goal of text clustering is to cluster similar texts without class labels, so that texts with similar structures and contents are classified into one category. The commonly used method is to first generate text embedding representations through feature extraction, and then use clustering methods to classify the vectorized texts. There are many ways to obtain text representations, so three classic methods are selected for comparison, namely the TF-IDF method based on feature engineering, the Word2vec model based on deep learning, and the pre-trained language model BERT. Common clustering methods include density-based methods, partition-based methods, and hierarchical clustering. The classic models among them: DBSCAN, K-Means, and agglomerative clustering methods are selected for comparison. The overall process of text clustering is as follows: Figure 2 shown.

[0039] Among text representation methods, the TF-IDF method evaluates the importance of each word by counting the frequency of each word. Only when a word appears very frequently in one article and very rarely in other articles, does the word have a good ability to distinguish different categories of text. The TF-IDF method uses this strategy to avoid the interference of common words and improve the relevance between keywords and articles. However, in texts with similar content, it is necessary to strengthen words that appear frequently in multiple texts, which is contrary to the idea of TF-IDF. Word2vec is a word embedding representation method that trains the representation of a word through a set of words around the word. There are two training methods, namely the Skip-gram model and the CBOW model. The former predicts a word based on its context, and the latter predicts its context through the word. Word2vec believes that even if different words have the same context, they will have similar embedding representations, which is in line with the effect that the present invention hopes to achieve. Compared to Word2vec, the BERT method considers both word context and word order. Therefore, the resulting word vectors are not static but incorporate contextual information. This training ensures that even the same word will have different vector representations in contexts with opposing meanings. While BERT performs better than Word2vec in most cases, it may introduce too much interactive information between words, affecting the embedding representation of differences in a collection of texts with similar content.

[0040] Among clustering methods, hierarchical clustering methods are divided into divisive clustering and agglomerative clustering, depending on the order of hierarchical decomposition. The former first classifies all samples into a single class, then splits it into the two most distant classes, repeating this process until the desired number of classes is reached. The latter first treats each sample in the data as a class, then agglomerates the two classes with the smallest distance, repeating this process until the desired number of classes is reached. Partition-based clustering methods aim to ensure that "points within a class are sufficiently close, and points between classes are sufficiently far." A representative algorithm, K-Means, first randomly selects k samples to initialize as the centers of the k classes. It then calculates the distance between each remaining sample and each class center, assigning the samples to the class closest to them based on their distance. It then recalculates the center of each class, and repeats this process until the class centers no longer change significantly. Density-based clustering methods can handle irregular clusters that K-Means cannot handle. A representative algorithm is the DBSCAN algorithm. Its idea is to first discover points with high density and then gradually connect closely spaced high-density points to generate clusters.

[0041] The following experiments detail the effectiveness of combining different embedding representations with different clustering methods for text clustering. The experiments also demonstrate that the Word2vec method and hierarchical clustering methods achieve the highest classification accuracy and offer greater stability.

[0042] Two: Generate a regular expression

[0043] Generate a regular expression from a set of similar statements d1, d2, ..., d m Extract the regular expression rules from the group, which can describe all the statements in the group. To simplify the problem, assume that the generated regular expression is in the form of S1.*S2.*…*S k , where S i (1≤i≤k) is the common part of this set of statements, so the goal is to extract the longest common part and the smallest difference part. This is transformed into solving the following problem:

[0044]

[0045] It is equivalent to solving the longest common subsequence problem of m strings:

[0046]

[0047] There have been many studies on solving the Longest Common Subsequence Problem (LCS). The classic method for solving the LCS problem between two strings is the dynamic programming algorithm, which has a complexity of O(n 2 ); However, solving the longest common subsequence (MLCS) of multiple strings is an NP-hard problem, and the time complexity of solving it using the dynamic programming algorithm is O(n m )(m strings), so it is no longer applicable to solving this problem. Currently, there are two methods for solving MLCS problems in scenarios with short sequences, small amounts of text, and small alphabets: precise methods based on control points and heuristic approximation algorithms. Obviously, the strings to be processed in this embodiment do not belong to this scenario. In view of the extremely high time complexity of the existing MLCS algorithm in processing long strings, an approximate solution algorithm for the longest common subsequence of multiple strings is proposed: the randomized MLCS method, such as Figure 3 shown.

[0048] Specifically, for the input m strings The process of solving their longest common subsequence using the randomized MLCS method is as follows:

[0049] 1. Random arrangement: Randomly arrange the m strings;

[0050] 2. Grouping: divide the strings into groups of two in order;

[0051] 3. Calculate lcs: Dynamic programming algorithm calculates the longest common subsequence of two strings in each group;

[0052] 4. Repeat: Repeat the process of grouping and calculating lcs for the results obtained in the previous step until lcs is calculated 1,m (Longest common subsequence of m strings).

[0053] However, because the accuracy of the result depends on the initial order of the strings, this process is repeated q times, and the longest sequence in q times is taken as the longest common subsequence of the final m strings. The time complexity of the randomized MLCS method proposed in this invention is O(q×m×n 2 ).

[0054] After solving the longest common subsequence of a type of text as the common part, the difference part is replaced by .* to get S1.*S2.*…*S k The regular expression is in the form of , but .* can match anything. Therefore, for the extracted .* sequence, information entropy is introduced to describe the uncertainty of the length of the .* matching part. The larger the information entropy, the greater the uncertainty, that is, the less certain the length. The calculation formula of information entropy is:

[0055]

[0056] Among them, X represents the object of the * matching part, and the length of all texts contained in the object is counted. t is the number of all different lengths, and p is the length of all texts in the object. i (1≤i≤t) is the probability of each length occurring.

[0057] For example, the object X in the matching part of .* can be the meal pickup code information. If X = {098, 152, 557, 012, 109, 185, 390}, the length is 3, so t = 1, p1 = 1, H(X) = 0, which means that the length of the meal pickup code is a fixed value, so .* can be replaced by \d{3}. For example, the object X of the .* matching part can be a person's name. If X = {Li Li, Guo Hongwu, Wu Ruihai, Wang Junyi, Fang Yanhong}, the corresponding text length is X = {2, 3, 3, 3}. There are two lengths, 2 and 3, with probabilities of occurrence p1 = 1 / 5 and p2 = 4 / 5, respectively. Then H(X) = -(1 / 5 × log2(1 / 5) + 4 / 5 × log2(4 / 5) = 0.72, indicating that the length of the person's name has a certain degree of uncertainty. If the threshold is set to 1 and 0.72 is less than the threshold, it is considered that .* can be concretized, and the concretization form is .{2, 3}. Therefore, a threshold is set. If the information entropy is greater than the threshold, the length is determined to be uncertain and .* does not need to be concretized. If the information entropy is less than the threshold, .* is further concretized. By counting the character types and lengths of the matching objects of the .* part, the form of its regular expression is further restricted.

[0058] Combining the text clustering method and the regular expression generation method, the automatic generation method of regular expressions for multiple categories of similar texts can be expressed as the following Algorithm 1. The input of this algorithm is a collection of multiple categories of similar texts, a total of n texts, and the output is the regular expression corresponding to each category of similar text.

[0059]

[0060]

[0061] The description of Algorithm 1 above is as follows:

[0062] Text clustering: Use text clustering to separate multiple text categories. Each category contains only text with similar structure and content. For example, in lines 1-2 of Algorithm 1, line 1 embeds the text, and line 2 uses clustering to divide the vectorized text into num categories.

[0063] Generate regular expressions: For each type of text, use the randomized MLCS method to extract the longest common subsequence and the difference part, as shown in line 4 of Algorithm 1. The difference part is replaced by .* to obtain S1.*S2.*…*S k The regular expression of the form is then calculated using information entropy to determine the length uncertainty of the .* matching part and decide whether to further finalize the .*, as shown in lines 5-12 of Algorithm 1. Finally, the final regular expression is obtained by combining the longest common subsequence and whether the .* part is finalized.

[0064] This embodiment also provides a regular expression automatic generation system for similar texts, including a memory and a processor. A computer program is stored in the memory, and the processor implements the steps of the above method when executing the program.

[0065] This embodiment also provides a computer-readable storage medium storing a computer program, which implements the steps of the above method when executed by a processor.

[0066] Here is a specific example:

[0067] The original input of this example is a collection of similar texts, such as the domain name text data in Table 1. The order may be chaotic. The process of automatically generating regular expressions is divided into the following stages:

[0068] Table 1 Similar text data

[0069]

[0070]

[0071] 1. Text clustering: First, use the Word2vec method and hierarchical clustering method to cluster these texts into different classes. For example, the first column in Table 1 is the first class and the second column is the second class.

[0072] 2. Extract the longest common subsequence: For each type of text, use the randomized MLCS method to extract their longest common subsequence, and replace the non-common part with .*. The generated initial regular expression results are as follows in Table 2:

[0073] Table 2 Initial regular expression

[0074] appx.koreasys.*.com .*.ddns.net

[0075] 3. Concrete.*: Calculate the information entropy of the.* part of the regular expression generated for each type of text. If it is greater than the set threshold, no further concretization is performed. If it is less than the threshold, statistics are performed. * The length and type of some text are specified. For example, in the first regular expression in the table below, .* is specified as \d{1,2}, where \d represents any digit from 0 to 9 and {1,2} represents repetition once or twice. The results are shown in Table 3 below:

[0076] Table 3. Specific results

[0077]

[0078]

[0079] The following is an experimental proof:

[0080] Regular expressions were extracted from 8670 pieces of data containing 12 types of DGA (Domain Generation Algorithms) families in malicious domains.

[0081] First, in the text clustering experiment, nine text clustering methods consisting of three text representation methods (TF-IDF, Word2vec, BERT) and three clustering methods (K-Means, DBSCAN, Agglomerative Clustering), as well as a traditional data mining-based LDA (Latent Dirichlet Allocation) method, were tested for ARI performance (Adjusted Rand Index) and time performance. The closer the ARI value is to 1, the closer the clustering result is to the actual situation. The experimental results are shown in Tables 4-5 below.

[0082] Table 4 ARI performance results

[0083]

[0084] Table 5 Time performance results

[0085]

[0086] The experimental results show that the Word2vec + hierarchical clustering method performs best on this dataset, achieving 100% accuracy and the lowest time performance. Therefore, this combination was selected as the text clustering method of the present invention. It also verifies that the Word2vec model can better characterize the characteristic information of texts with similar structure and content, and that the hierarchical clustering and K-Means models can better discover similar text clusters, indicating that similar text sets are clusters with regular shapes, not density-based clusters.

[0087] In the regular expression generation experiment, we compared methods for finding the longest common subsequence of multiple strings, including the randomized MLCS method and two precise control point-based methods (Fast_LCS and Top_MLCS). The experimental data consisted of 100 texts of length randomly generated from an alphabet. The runtime and accuracy of each method were tested with alphabet sizes |Σ| of 4 and 20, and the number of texts m of 6, 30, and 60, respectively. The accuracy is expressed as the calculated length of the longest common subsequence, which is the value in [] in the table. The runtime is expressed in seconds. The experimental results are shown in Table 6.

[0088] Table 6 Comparison of methods for solving the longest common subsequence of multiple strings

[0089]

[0090] The experimental results show that the Fast_LCS method can only handle datasets with small alphabets and small amounts of text. While the Top_MLCS method can handle datasets with large alphabets and large amounts of text, it has extremely high time complexity when large amounts of text contain repeated letters and there is excessive interaction between different texts. Although the randomized MLCS method is less accurate than the previous two precise methods, it can handle text of arbitrary length and size. Compared to the previous two precise methods, the randomized MLCS method achieves the fastest time reduction from hundreds of seconds to seconds (when |Σ| = 4, m = 6), with only a 0.48% decrease in accuracy (see Table 6). Furthermore, a certain amount of error is allowed in this problem; unrecognized longest common subsequences can be replaced with .*.

[0091] The combination of the Word2vec text representation method and hierarchical clustering effectively completes the text clustering process. The combination of the randomized MLCS method and information entropy technology can extract regular expressions for each text category. Through these steps, a complete regular expression generation method is implemented. To verify the effectiveness of the generated regular expressions, each text category is randomly divided into a training set and a test set. The training set accounts for 80% and is used to generate regular expressions, while the test set accounts for 20% and is used to test whether the generated regular expressions can match the text in the test set. Ten generation experiments are conducted on the dataset. For each generated regular expression, the proportion of regular expressions that can be matched in the test set is calculated. When the generated regular expressions are the same, the average of all proportions is calculated, and the accuracy of the generated regular expressions is shown in Table 7. In summary, if the training set is large enough and contains all possible sample cases, the generation accuracy will be high.

[0092] Table 7 Examples of automatic regular expression generation results

[0093]

[0094] Although the present invention has been disclosed as above by way of embodiments, they are not intended to limit the present invention. Any appropriate modification or equivalent substitution of the technical solution of the present invention by a person skilled in the art should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on that defined in the claims.

Claims

1. A method for automatically generating regular expressions for similar texts, characterized in that: The following steps are involved: For similar texts input, a text embedding representation method is used to obtain an embedded representation of the similar texts; and a clustering method is used to cluster similar texts of the same type into the same category based on the embedded representation. For each type of similar text, extract the common part and the difference part. The method of extracting the common part is to solve the longest common subsequence of all similar text strings in each type to obtain the longest common part of the similar text in this type. The method of extracting the difference part is to calculate the difference of the difference part through information entropy. If the information entropy is less than a certain set threshold, the content and length of the part of the text are counted and used as the extracted difference part. The method for solving the longest common subsequence is the randomized MLCS method, which includes the following steps: First, all similar text strings in each category are randomly arranged, and the number of strings is m; Secondly, divide the strings into groups of two in order; Again use the dynamic programming algorithm to calculate the longest common subsequence of the two strings in each group; Finally, the results obtained in the previous step are divided into groups of two again in order, and the dynamic programming algorithm is used to calculate the longest common subsequence of the two strings in each group. This step is repeated q times, and the longest sequence among q times is taken as the longest common subsequence of the final m strings; In the method of extracting the difference part, the difference part is replaced by the symbol .*, which can match any content. The information entropy is used to describe the uncertainty of the length of the .* matching part. If the information entropy is less than a set threshold, the text content and length of the matching object of the .* matching part are counted; the calculation formula of the information entropy is Among them, X represents the object of the * matching part, and the length of all texts contained in the object is counted. t is the number of all different lengths, and p is the length of all texts in the object. i (1≤i≤t) is the probability of occurrence of each length; The extracted common parts and difference parts are combined to generate regular expressions corresponding to each type of similar text.

2. The method according to claim 1, wherein The text embedding representation method is the TF-IDF method, the Word2vec method or the BERT method.

3. The method according to claim 2, wherein The text embedding representation method uses the Word2vec method.

4. The method according to claim 3, wherein In the Word2vec method, a Skip-gram model is trained to predict a word based on its context, and a CBOW model is trained to predict the context of a word based on its context.

5. The method according to claim 1, wherein The clustering method is a density-based method, a partition-based method or a hierarchical clustering method, wherein the density-based method selects the DBSCAN algorithm, the partition-based method selects the K-Means algorithm, and the hierarchical clustering method selects the divisive clustering method or the agglomerative clustering method.

6. The method according to claim 5, wherein The clustering method selected is the agglomerative clustering method in the hierarchical clustering method.

7. A regular expression automatic generation system for similar texts, characterized by: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium storing a computer program, wherein when the program is executed by a processor, the program implements the steps of the method according to any one of claims 1 to 6.

Images