A data processing method and related apparatus

Abstract

The application discloses a data processing method, comprising: performing feature extraction on a first data set and a second data set respectively to obtain first data features and second data features; wherein the data features are obtained by performing statistical feature extraction on the data set according to a format matching rule, identical characters and / or identical character types; judging whether the similarity between the first data features and the second data features is greater than a threshold value; if yes, the data is similar data. By performing feature extraction on the first data set and the second data set to obtain corresponding data features, the data features are obtained by performing statistical feature extraction on the data set according to the format matching rule, the identical characters and / or the identical character types, instead of only using a fixed regular expression for similarity judgment, thereby improving the accuracy and effect of the similarity judgment. The application also discloses a data processing device, a server and a computer readable storage medium, which have the above beneficial effects.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, a data processing device, a server, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of information technology, the amount of data generated is increasing. When managing data assets, it is necessary to associate similar data in different locations in order to automate the organization and management of the data.

[0003] In related technologies, data fingerprinting is generally used to determine whether data is similar. For example, the length, data type, maximum value, minimum value, and null value of a column are extracted as data fingerprints for that column, and then the data fingerprints of each column are used to determine whether corresponding columns are similar. However, when the data is complex, the similarity judgment is less effective and has lower accuracy.

[0004] Therefore, improving the accuracy of data similarity judgment in structured data is a key issue of concern to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method, a data processing device, a server, and a computer-readable storage medium to improve the accuracy of data feature extraction and maintain the precision of similarity judgment.

[0006] To address the aforementioned technical problems, this application provides a data processing method, comprising:

[0007] Feature extraction is performed on the first dataset and the second dataset respectively to obtain the first data feature and the second data feature; wherein, the data feature is obtained by statistical feature extraction of the dataset based on format matching rules, identical characters and / or identical character types;

[0008] Determine whether the similarity between the first data feature and the second data feature is greater than a threshold;

[0009] If so, then the first dataset and the second dataset are determined to be similar data;

[0010] If not, then the first dataset and the second dataset are determined to be dissimilar.

[0011] Optionally, feature extraction is performed on the first dataset and the second dataset respectively to obtain first data features and second data features, including:

[0012] The first dataset and the second dataset are deduplicated to obtain a first deduplicated dataset and a second deduplicated dataset.

[0013] Use the first deduplicated dataset as the first data feature;

[0014] The second deduplicated dataset is used as the second data feature.

[0015] Optionally, determining whether the similarity between the first data feature and the second data feature is greater than a threshold includes:

[0016] The distance between the same characters in the first data feature and the second data feature is calculated to obtain the distance between the same characters;

[0017] Determine whether the distance between the identical characters is less than a preset distance;

[0018] If so, then the similarity is determined to be greater than the threshold;

[0019] If not, then the similarity is determined to be less than the threshold.

[0020] Optionally, feature extraction is performed on the first dataset and the second dataset respectively to obtain first data features and second data features, including:

[0021] Extract the same character type from the first dataset to obtain a first set of the same character types, and calculate the proportion of each character type in the first set of the same character types to obtain the first data feature;

[0022] Extract the same character type from the second dataset to obtain a second set of the same character type, and calculate the proportion of each character type in the second set of the same character type to obtain the second data feature.

[0023] Optionally, determining whether the similarity between the first data feature and the second data feature is greater than a threshold includes:

[0024] Calculate the character type distance between the first data feature and the second data feature;

[0025] Determine whether the distance between the character types is less than a preset distance;

[0026] If so, then the similarity is determined to be greater than the threshold;

[0027] If not, then the similarity is determined to be less than the threshold.

[0028] Optionally, feature extraction is performed on the first dataset and the second dataset respectively to obtain first data features and second data features, including:

[0029] Perform regular expression matching on the first dataset to obtain a first set of regular expressions, which are then used as the first data features;

[0030] The second dataset is subjected to regular expression matching to obtain a second set of regular expressions, which are used as the second data features.

[0031] Optionally, determining whether the similarity between the first data feature and the second data feature is greater than a threshold includes:

[0032] Determine whether the first data feature is the same as the second data feature;

[0033] If so, then the similarity is determined to be greater than the threshold;

[0034] If not, then the similarity is determined to be less than the threshold.

[0035] This application also provides a data processing apparatus, including:

[0036] The feature extraction module is used to extract features from the first dataset and the second dataset respectively to obtain the first data features and the second data features; wherein, the data features are obtained by statistical feature extraction of the dataset based on format matching rules, identical characters and / or identical character types;

[0037] The feature determination module is used to determine whether the similarity between the first data feature and the second data feature is greater than a threshold.

[0038] The first determination module is used to determine that the first dataset and the second dataset are similar data when the similarity is greater than a threshold.

[0039] The second determination module is used to determine that the first dataset and the second dataset are not similar data when the similarity is less than or equal to the threshold.

[0040] This application also provides a server, including:

[0041] Memory, used to store computer programs;

[0042] A processor for executing the computer program to implement the data processing method described above.

[0043] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described above.

[0044] The data processing method provided in this application includes: extracting features from a first dataset and a second dataset respectively to obtain a first data feature and a second data feature; wherein the data features are obtained by statistical feature extraction of the datasets according to format matching rules, identical characters and / or identical character types; determining whether the similarity between the first data feature and the second data feature is greater than a threshold; if yes, determining that the first dataset and the second dataset are similar data; if no, determining that the first dataset and the second dataset are not similar data.

[0045] By extracting features from the first and second datasets, corresponding data features are obtained. These data features are then statistically extracted from the datasets based on format matching rules, identical characters, and / or identical character types. The corresponding data features are then used to determine whether the similarity is greater than a threshold, rather than simply using fixed regular expressions for similarity judgment, thereby improving the accuracy and effectiveness of similarity judgment.

[0046] This application also provides a data processing apparatus, a server, and a computer-readable storage medium, which have the above-mentioned beneficial effects, and will not be elaborated here. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart of the first data processing method provided in the embodiments of this application;

[0049] Figure 2 A flowchart illustrating the second data processing method provided in the embodiments of this application;

[0050] Figure 3 A flowchart illustrating the third data processing method provided in this application embodiment;

[0051] Figure 4 A flowchart illustrating the fourth data processing method provided in this application embodiment;

[0052] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0053] The core of this application is to provide a data processing method, a data processing device, a server, and a computer-readable storage medium to improve the accuracy of data feature extraction and maintain the precision of similarity judgment.

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] In related technologies, data fingerprinting is generally used to determine whether data is similar. For example, the length, data type, maximum value, minimum value, and null value of a column are extracted as data fingerprints for that column, and then the data fingerprints of each column are used to determine whether corresponding columns are similar. However, when the data is complex, the similarity judgment is less effective and has lower accuracy.

[0056] Therefore, this application provides a data processing method that extracts features from a first dataset and a second dataset to obtain corresponding data features. These data features are obtained by statistically extracting features from the datasets based on format matching rules, identical characters, and / or identical character types. The method then uses the corresponding data features to determine whether the similarity is greater than a threshold, rather than simply using a fixed regular expression for similarity judgment, thereby improving the accuracy and effectiveness of similarity judgment.

[0057] The following example illustrates a data processing method provided in this application.

[0058] Please refer to Figure 1 , Figure 1 This is a flowchart of a first data processing method provided in an embodiment of this application.

[0059] In this embodiment, the method may include:

[0060] S101, Perform feature extraction on the first dataset and the second dataset respectively to obtain the first data feature and the second data feature; wherein, the data feature is obtained by statistical feature extraction of the dataset based on format matching rules, identical characters and / or identical character types;

[0061] As can be seen, this step mainly involves extracting data features from the first and second datasets respectively. Further, this step performs feature extraction on the first and second datasets respectively, obtaining corresponding first and second data features. These data features are obtained by statistically extracting features from the datasets based on format matching rules, identical characters, and / or identical character types. In other words, both the extracted first and second data features are obtained by statistically extracting features from the datasets based on format matching rules, identical characters, and / or identical character types.

[0062] Here, the format matching rule refers to performing format matching on the data in the dataset to obtain all formats that match the data in the dataset. Furthermore, the format matching rule can be a regular expression, a format judgment condition, or a character judgment rule. Therefore, the specific form of the format matching rule in the technical solution of this application is not unique, and no specific limitation is made here.

[0063] The "identical characters" feature involves statistically analyzing the identical characters found in the dataset to obtain data characteristics. For example, if a dataset contains the names of various provinces, removing duplicates from the dataset will reveal the data characteristics based on identical characters.

[0064] The "same character type" refers to statistically analyzing the identical character types within a dataset to obtain corresponding data features. It can also involve extracting the same character types from a dataset to obtain a corresponding set of identical character types.

[0065] S102, determine whether the similarity between the first data feature and the second data feature is greater than the threshold; if yes, proceed to S103; if no, proceed to S104.

[0066] Based on S101, this step aims to determine whether the similarity between the first data feature and the second data feature is greater than a threshold.

[0067] The similarity metric is primarily used to determine the degree of similarity between the first data feature and the second data feature. Furthermore, when this similarity metric is greater than a threshold, meaning the degree of similarity between the two features exceeds the threshold, it can be determined that the first data feature and the second data feature are similar data.

[0068] The specific method for calculating similarity can be determined based on the specific type of data features. When the data feature is a matched format, the similarity can be determined by whether the formats of the two data features are the same, or by the number of identical or dissimilar formats between the two data features.

[0069] When data features include identical characters, the distance between the two can be calculated as the similarity score. A smaller distance indicates greater similarity, and a higher similarity score. Conversely, a larger distance indicates greater difference, and a lower similarity score.

[0070] When the data features are of the same character type, the distance between the two can be calculated as the similarity. A smaller distance indicates greater similarity, and a higher similarity score. Conversely, a larger distance indicates greater difference, and a lower similarity score.

[0071] As can be seen, the method for determining the similarity between the first and second data features in this step can be selected based on the specific content of the data features. However, generally speaking, it involves determining whether the similarity between the two data features exceeds a threshold to determine whether they are similar data.

[0072] S103, determine that the first dataset and the second dataset are similar data;

[0073] Based on S102, when the similarity is greater than the threshold, the first dataset and the second dataset are determined to be similar data.

[0074] In other words, the first dataset and the second dataset are highly similar, and can be considered as similar data.

[0075] S104, determine that the first dataset and the second dataset are not similar data.

[0076] Based on S102, when the similarity is less than or equal to the threshold, the first dataset and the second dataset are determined to be similar data.

[0077] In other words, the first dataset and the second dataset have a low degree of similarity, and are therefore determined to be dissimilar data.

[0078] In summary, this embodiment extracts features from the first and second datasets to obtain corresponding data features. These data features are obtained by statistically extracting features from the datasets based on format matching rules, identical characters, and / or identical character types. Then, the corresponding data features are used to determine whether the similarity is greater than a threshold, rather than simply using fixed regular expressions for similarity judgment, thereby improving the accuracy and effectiveness of similarity judgment.

[0079] The following embodiment further illustrates a data processing method provided in this application.

[0080] Please refer to Figure 2 , Figure 2 This is a flowchart of a second data processing method provided in an embodiment of this application.

[0081] In this embodiment, the method may include:

[0082] S201, Perform deduplication on the first dataset and the second dataset respectively to obtain the first deduplicated dataset and the second deduplicated dataset;

[0083] S202, calculate the distance between the same characters in the first deduplicated dataset and the second deduplicated dataset to obtain the distance between the same characters;

[0084] S203, determine whether the distance between identical characters is less than a preset distance;

[0085] S204, if so, then the first dataset and the second dataset are considered similar data;

[0086] S205. If not, then the first dataset and the second dataset are determined to be dissimilar.

[0087] As can be seen, in this embodiment, the first deduplicated dataset and the second deduplicated dataset are actually used as the first data feature and the second data feature, respectively, in order to determine the similarity between the first dataset and the second dataset.

[0088] Clearly, in this embodiment, the common characters in each dataset are used as the data features of that dataset. Then, the distance between the two datasets is calculated, and finally, the similarity between the two is determined by the distance.

[0089] The distance between the first deduplicated dataset and the second deduplicated dataset can be calculated using Jaro distance, cosine distance, Hamming distance, or any distance calculation method provided by existing technology; no specific limitation is made here.

[0090] Jaro distance is a measure of string similarity, also known as edit distance. A higher Jaro distance indicates higher similarity between the two texts. Therefore, if Jaro distance is used for calculation, we can determine whether the obtained distance is greater than a preset distance. If it is, the data is considered similar; otherwise, it is not.

[0091] As can be seen, this embodiment improves the accuracy and effectiveness of similarity judgment by extracting features from the first and second datasets, obtaining corresponding data features, and then using whether the distance between the corresponding data features is less than a preset distance, rather than simply using a fixed regular expression for similarity judgment.

[0092] The following embodiment further illustrates a data processing method provided in this application.

[0093] Please refer to Figure 3 , Figure 3This is a flowchart of a third data processing method provided in an embodiment of this application.

[0094] In this embodiment, the method may include:

[0095] S301, extract the same character type from the first dataset to obtain a first set of the same character types, and calculate the proportion of each character type in the first set of the same character types to obtain a first data feature; extract the same character type from the second dataset to obtain a second set of the same character types, and calculate the proportion of each character type in the second set of the same character types to obtain a second data feature;

[0096] S302, Calculate the character type distance between the first data feature and the second data feature;

[0097] S303, determine whether the character type distance is less than the preset distance;

[0098] S304, if so, then the first dataset and the second dataset are considered similar data;

[0099] S305. If not, then the first dataset and the second dataset are determined to be dissimilar.

[0100] As can be seen, in this embodiment, the proportion of the same character type in the dataset is used as the corresponding data feature for judgment.

[0101] The character type distance between the first data feature and the second data feature can be calculated using Euclidean distance, cosine distance, Hamming distance, or any distance calculation method provided by existing technology. No specific limitation is made here.

[0102] Euclidean distance is a commonly used definition of distance, referring to the true distance between two points in m-dimensional space, or the natural length of a vector (i.e., the distance from that point to the origin).

[0103] As can be seen, this embodiment improves the accuracy and effectiveness of similarity judgment by extracting features from the first and second datasets, obtaining corresponding data features, and then using whether the distance between the corresponding data features is less than a preset distance, rather than simply using a fixed regular expression for similarity judgment.

[0104] The following embodiment further illustrates a data processing method provided in this application.

[0105] Please refer to Figure 4 , Figure 4 This is a flowchart of the fourth data processing method provided in the embodiments of this application.

[0106] In this embodiment, the method may include:

[0107] S401, Perform regular expression matching on the first dataset to obtain the first set of regular expressions; perform regular expression matching on the second dataset to obtain the second set of regular expressions;

[0108] S402, Determine whether the first set of regular expressions is the same as the second set of regular expressions;

[0109] S403, if so, then the first dataset and the second dataset are considered similar data;

[0110] S404. If not, then the first dataset and the second dataset are determined to be dissimilar.

[0111] As can be seen, in this embodiment, the first set of regular expressions and the second set of regular expressions are actually used as the first data feature and the second data feature, respectively, in order to determine the similarity between the first dataset and the second dataset.

[0112] Furthermore, in this embodiment, the overlap rate between the first set of regular expressions and the second set of regular expressions can be calculated, and then it can be determined whether the overlap rate is greater than a preset overlap rate. If so, the two are determined to be similar data; if not, the two are determined to be different data.

[0113] As can be seen, the embodiments of this application improve the accuracy of similarity judgment by performing regular expression matching on the dataset and further judging the similarity.

[0114] The following specific embodiment will further illustrate a data processing method provided in this application.

[0115] In this embodiment, the method may include:

[0116] Step 1: Perform regular expression matching on the first dataset to obtain the first set of regular expressions; perform regular expression matching on the second dataset to obtain the second set of regular expressions.

[0117] This step uses regular expressions to determine the data type of cells in a structured dataset. Examples include built-in fields like name, gender, province, ID number, postal code, phone number, and landline number. Each cell is evaluated for a matching regular expression. If a particular regular expression type is found to match more than a certain percentage of all cells, the column is considered to conform to that regular expression, denoted as 'r'. 'r' is a string representing a regular expression; if the column does not match any built-in regular expressions, 'r' is empty.

[0118] Step 2: Perform deduplication on the first dataset and the second dataset respectively to obtain the first deduplicated dataset and the second deduplicated dataset;

[0119] This step retrieves the number of duplicate cells in the dataset. If the number of duplicate cells does not exceed a certain limit, the dataset is considered an enumerated dataset, where the enumeration type represents the content of each duplicate cell, denoted as e. e is a dataset containing the values ​​of each duplicate cell [e1, e2, ..., e]. n If the column is not an enumerated column, then e is empty.

[0120] Step 3: Extract the same character type from the first dataset to obtain the first set of the same character type, and calculate the proportion of each character type in the first set of the same character type to obtain the proportion of the first character type; extract the same character type from the second dataset to obtain the second set of the same character type, and calculate the proportion of each character type in the second set of the same character type to obtain the proportion of the second character type;

[0121] In this step, character-level feature statistics are performed on the dataset, including but not limited to the following: percentage of numeric characters; percentage of uppercase letters; percentage of lowercase letters; percentage of Chinese characters; percentage of special characters (e.g., .-_, : / \{}); percentage of other characters; percentage of characters with lengths of 0, 1, 2, 3-4, 5-9, 10-24, and 24 or longer; percentage of cells containing special characters; percentage of cells containing only numeric characters with lengths of 1, 2, ..., 20 and longer; and percentage of cells containing fixed Chinese characters (e.g., 'province', 'city', ..., 'university', 'engineering', etc.). A total of 81 features are denoted as f = (x1, x2, ... x...). 81 ).

[0122] As can be seen, after the above feature extraction, the first dataset and the second dataset yielded the first data feature and the second data feature, respectively. The first data feature includes the first set of regular expressions, the first deduplicated dataset, and the proportion of the first character type, i.e., (r1, e1, f1). The second data feature includes the second set of regular expressions, the second deduplicated dataset, and the proportion of the second character type, i.e., (r2, e2, f2).

[0123] Step 4: Determine if the first set of regular expressions is the same as the second set of regular expressions; if yes, proceed to step 5; if no, proceed to step 6.

[0124] In other words, we determine whether r1 and r2 are the same. If r1 = r2 and is not empty, then the two columns have the same regular expression.

[0125] Step 5: Determine that the first dataset and the second dataset are similar data;

[0126] Step 6: Calculate the distance between the same characters in the first deduplicated dataset and the second deduplicated dataset to obtain the distance between the same characters; determine whether the distance between the same characters is less than the preset distance; if yes, proceed to step 5; if no, proceed to step 7.

[0127] This step can also involve calculating the similarity between e1 and e2. Sort the values ​​in e1 and e2 and concatenate them to form two strings s1 and s2, then calculate the Jaro distance between s1 and s2. Where m is the number of characters matched by s1 and s2 respectively, and t is the number of transpositions. Two characters from s1 and s2 are matched if they are no more than [a certain distance]. Two strings can be matched; the number of transpositions t is determined by the number of matching characters in different orders. Simply put, t is half the number of transpositions. Therefore, d represents the similarity between e1 and e2. If d is greater than a set threshold, the two columns are considered to have the same enumeration type, and step 5 is executed.

[0128] Step 7: Calculate the character type distance between the proportion of the first character type and the proportion of the second character type; determine whether the character type distance is less than the preset distance; if yes, proceed to step 5; if no, proceed to step 8.

[0129] This step calculates the similarity between f1 and f2. The Euclidean distance between vectors x and y is defined as... The similarity between f1 and f2 is defined as follows: If sim exceeds the set threshold, the data format characteristics of the two columns are considered to be the same, and step 5 is executed.

[0130] Step 8: Determine that the first dataset and the second dataset are not similar data.

[0131] As can be seen, this embodiment obtains corresponding data features by extracting features from the first dataset and the second dataset. These data features are obtained by statistically extracting features from the datasets based on format matching rules, identical characters, and / or identical character types. Then, the corresponding data features are used to determine whether the similarity is greater than a threshold, rather than simply using a fixed regular expression for similarity judgment, thereby improving the accuracy and effectiveness of similarity judgment.

[0132] The data processing apparatus provided in the embodiments of this application is described below. The data processing apparatus described below and the data processing method described above can be referred to in correspondence.

[0133] Please refer to Figure 5 , Figure 5This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application.

[0134] In this embodiment, the device may include:

[0135] The feature extraction module 100 is used to extract features from the first dataset and the second dataset respectively to obtain first data features and second data features; wherein, the data features are obtained by statistical feature extraction of the dataset based on format matching rules, identical characters and / or identical character types;

[0136] The feature judgment module 200 is used to determine whether the similarity between the first data feature and the second data feature is greater than a threshold.

[0137] The first determination module 300 is used to determine that the first dataset and the second dataset are similar data when the similarity is greater than a threshold.

[0138] The second determination module 400 is used to determine that the first dataset and the second dataset are not similar data when the similarity is less than or equal to the threshold.

[0139] Optionally, the feature extraction module 100 can be used to perform deduplication processing on the first dataset and the second dataset respectively to obtain a first deduplicated dataset and a second deduplicated dataset; use the first deduplicated dataset as the first data feature; and use the second deduplicated dataset as the second data feature.

[0140] Optionally, the feature judgment module 200 can be used to calculate the distance between the same characters of the first data feature and the second data feature to obtain the distance between the same characters; determine whether the distance between the same characters is less than a preset distance; if so, determine that the similarity is greater than a threshold; if not, determine that the similarity is less than a threshold.

[0141] Optionally, the feature extraction module 100 can be used to extract the same character type from the first dataset to obtain a first set of the same character types, and calculate the proportion of each character type in the first set of the same character types to obtain a first data feature; and to extract the same character type from the second dataset to obtain a second set of the same character types, and calculate the proportion of each character type in the second set of the same character types to obtain a second data feature.

[0142] Optionally, the feature judgment module 200 can be used to calculate the character type distance between the first data feature and the second data feature; determine whether the character type distance is less than a preset distance; if so, determine that the similarity is greater than a threshold; if not, determine that the similarity is less than a threshold.

[0143] Optionally, the feature extraction module 100 can be used to perform regular expression matching on the first dataset to obtain a first set of regular expressions, which are used as the first data feature; and to perform regular expression matching on the second dataset to obtain a second set of regular expressions, which are used as the second data feature.

[0144] Optionally, the feature judgment module 200 can be used to determine whether the first data feature and the second data feature are the same; if so, the similarity is determined to be greater than the threshold; if not, the similarity is determined to be less than the threshold.

[0145] This application embodiment also provides a server, including:

[0146] Memory, used to store computer programs;

[0147] A processor is configured to implement the steps of the data processing method as described in the above embodiments when executing the computer program.

[0148] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described in the above embodiments.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0152] The data processing method, data processing apparatus, server, and computer-readable storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: performing feature extraction on the first data set and the second data set respectively to obtain first data features and second data features; wherein the data features are obtained by performing statistical feature extraction on the data set according to a format matching rule, identical characters and / or identical character types; the format matching rule is regular expression matching; determining whether the similarity between the first data features and the second data features is greater than a threshold value; if yes, determining that the first data set and the second data set are similar data; if no, determining that the first data set and the second data set are not similar data; wherein determining whether the similarity between the first data features and the second data features is greater than a threshold value comprises: determining whether the first regular expression set and the second regular expression set obtained based on the regular expression matching are identical; if identical, the similarity is greater than the threshold value, if not identical, determining whether the identical character distance between the first deduplicated data set and the second deduplicated data set obtained based on deduplication processing is less than a preset distance; if yes, the similarity is greater than the threshold value, if not, determining whether the character type distance between the first character type proportion and the second character type proportion is less than a preset distance; if yes, the similarity is greater than the threshold value, if not, the similarity is not greater than the threshold value; wherein the first data features comprise the first regular expression set, the first deduplicated data set and the first character type proportion; the second data features comprise the second regular expression set, the second deduplicated data set and the second character type proportion.

2. The data processing method according to claim 1, characterized in that, performing feature extraction on the first data set and the second data set respectively to obtain first data features and second data features, comprising: performing deduplication processing on the first data set and the second data set respectively to obtain first deduplicated data set and second deduplicated data set; taking the first deduplicated data set as the first data features; taking the second deduplicated data set as the second data features.

3. The data processing method according to claim 2, characterized in that, determining whether the similarity between the first data features and the second data features is greater than a threshold value, comprising: performing distance calculation according to the identical characters of the first data features and the second data features to obtain an identical character distance; determining whether the identical character distance is less than a preset distance; if yes, determining that the similarity is greater than the threshold value; if no, determining that the similarity is less than the threshold value.

4. The data processing method of claim 1, wherein, performing feature extraction on the first data set and the second data set respectively to obtain first data features and second data features, comprising: performing identical character type extraction on the first data set to obtain a first identical character type set, and performing proportion calculation of each character type on the first identical character type set to obtain the first data features; performing identical character type extraction on the second data set to obtain a second identical character type set, and performing proportion calculation of each character type on the second identical character type set to obtain the second data features.

5. The data processing method according to claim 4, characterized in that, determining whether the similarity between the first data features and the second data features is greater than a threshold value, comprising: calculating the character type distance of the first data features and the second data features; determining whether the character type distance is less than a preset distance; if yes, determining that the similarity is greater than a threshold value; if no, determining that the similarity is less than the threshold value.

6. The data processing method of claim 1, wherein, performing feature extraction on the first data set and the second data set respectively to obtain first data features and second data features, including: performing regular expression matching on the first data set to obtain a first regular expression set, and taking the first regular expression set as the first data features; performing regular expression matching on the second data set to obtain a second regular expression set, and taking the second regular expression set as the second data features.

7. The data processing method according to claim 6, characterized in that, determining whether the similarity between the first data features and the second data features is greater than a threshold value, including: determining whether the first data features and the second data features are the same; if yes, determining that the similarity is greater than a threshold value; if no, determining that the similarity is less than the threshold value.

8. A data processing apparatus, characterized by, including: a feature extraction module, configured to perform feature extraction on a first data set and a second data set respectively to obtain first data features and second data features; wherein the data features are obtained by statistical feature extraction on the data set according to a format matching rule, identical characters and / or identical character types; the format matching rule is regular expression matching; a feature determination module, configured to determine whether the similarity between the first data features and the second data features is greater than a threshold value; a first determination module, configured to determine that the first data set and the second data set are similar data when the similarity is greater than a threshold value; a second determination module, configured to determine that the first data set and the second data set are not similar data when the similarity is less than or equal to a threshold value; wherein the feature determination module is specifically configured to determine whether the first regular expression set and the second regular expression set obtained based on the regular expression matching are the same; if yes, the similarity is greater than a threshold value, if not, determine whether the identical character distance between the first deduplicated data set and the second deduplicated data set obtained based on deduplication processing is less than a preset distance; if yes, the similarity is greater than a threshold value, if not, determine whether the character type distance between the first character type proportion and the second character type proportion is less than a preset distance; if yes, the similarity is greater than a threshold value, if not, the similarity is not greater than a threshold value; wherein the first data features include the first regular expression set, the first deduplicated data set and the first character type proportion; the second data features include the second regular expression set, the second deduplicated data set and the second character type proportion.

9. A server, characterized by including: a memory, configured to store a computer program; a processor, configured to implement the steps of the data processing method of any one of claims 1 to 7 when the computer program is executed.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the data processing method of any one of claims 1 to 7.

Images