A data processing method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a data processing method, apparatus, device and storage medium, wherein the method comprises: clustering the word vectors in the semantic space according to at least one word vector corresponding to the sample data to obtain at least one cluster; performing step-by-step hierarchical word vector expansion on each cluster to construct a vocabulary closeness rule corresponding to the cluster; wherein each layer of word vector expansion has an attenuation factor, and the attenuation factor corresponding to the word vector expansion in the outer layer is stronger; and using the vocabulary closeness rule to detect sensitive information in the data to be detected.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of information security technology, and relate to but are not limited to a data processing method, apparatus, device and storage medium. Background Art

[0002] While promoting the open sharing of data resources, how to effectively ensure the security of users' sensitive data is a key issue that needs to be addressed.

[0003] At present, the main methods for identifying sensitive data include: regular expression feature library, calculating vocabulary weights to obtain sensitive words, using sensitive word library, etc. Among them, the regular expression feature library can quickly and effectively identify information, but due to the limited regular library, a lot of sensitive data is missed, and the actual situation is relatively complex and difficult to deal with; calculating vocabulary weights to obtain sensitive words is to process documents, calculate the vector weight value of each word, manually identify and classify the basic data corpus, and form a sensitive word corpus; using the sensitive word vector library, classify it, set the benchmark words for each category, and convert the test data into a classification process.

[0004] However, the current method of identifying sensitive data needs to rely on rule bases and sensitive words for annotation and statistics. The method is limited by the quantity, quality and human factors of the rule base itself, and therefore has a large false alarm rate and missed alarm rate. Summary of the invention

[0005] In view of this, embodiments of the present application provide a data processing method, apparatus, device, and storage medium.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0008] According to at least one word vector corresponding to the sample data, clustering is performed on the word vectors in the semantic space to obtain at least one clustering cluster;

[0009] The word vector of each cluster is expanded step by step and in layers to construct the vocabulary closeness rule of the corresponding cluster. Each layer of word vector expansion has a decay factor, and the decay factor corresponding to the word vector expansion in the outer layer is stronger.

[0010] The vocabulary closeness rule is used to detect sensitive information in the data to be detected.

[0011] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0012] A clustering processing unit, configured to perform clustering processing on the word vectors in the semantic space according to at least one word vector corresponding to the sample data, to obtain at least one cluster cluster;

[0013] The word vector expansion unit is used to expand the word vector of each cluster in a step-by-step hierarchical manner and construct the vocabulary closeness rules of the corresponding cluster. Each layer of word vector expansion has an attenuation factor, and the attenuation factor corresponding to the word vector expansion in the outer layer is stronger.

[0014] The identification processing unit is used to detect sensitive information in the data to be detected by using the vocabulary closeness rule.

[0015] In a third aspect, an embodiment of the present application provides a data processing device, the device comprising at least: a processor and a storage medium configured to store executable instructions, wherein: the processor is configured to execute the stored executable instructions;

[0016] The executable instructions are configured to execute the above-mentioned data processing method.

[0017] In a fourth aspect, an embodiment of the present application provides a storage medium, in which computer executable instructions are stored, and the computer executable instructions are configured to execute the above-mentioned data processing method.

[0018] The data processing method, device, equipment and storage medium provided in the embodiments of the present application, by gradually and hierarchically expanding the word vectors of each cluster obtained by clustering processing, constructs the vocabulary closeness rules of the corresponding cluster, wherein each layer of word vector expansion has an attenuation factor, and the attenuation factor corresponding to the word vector expansion in the outer layer is stronger. In this way, more comprehensive vocabulary closeness rules can be automatically constructed and mined adaptively, thereby being able to better break through the resource limitations of the current rule library, regular library, etc. while reducing human participation, greatly improving the accuracy of using vocabulary closeness rules to detect sensitive information in the data to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the accompanying drawings (which are not necessarily drawn to scale), like reference numerals may describe similar components in different views. Like reference numerals with different letter suffixes may represent different examples of similar components. The accompanying drawings generally illustrate various embodiments discussed herein by way of example and not limitation.

[0020] Figure 1 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0021] Figure 2 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0022] Figure 3 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0023] Figure 4 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0024] Figure 5 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0025] Figure 6 A conceptual diagram of vocabulary closeness provided for an embodiment of the present application;

[0026] Figure 7 A flow chart of judgment rules provided for an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a rule tree construction process provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0031] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0032] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0033] 1) Sensitive rules: Sensitive rules are composed of words and relationships. They are a set of words that contain the three relationships of "and", "or", and "not" (indicated by the "&", "|", and "!" symbols). For example, "login & account! password" means that the login and account are included but the password is not included.

[0034] 2) Regular rules: Regular rules are also called regular expressions. They are a pattern matching method and a logical formula for string and special character operations. They use some pre-defined specific characters and the combination of these specific characters to form a "regular string". Regular expressions are usually used to retrieve and replace text that conforms to a certain pattern (rule).

[0035] 3) Semantic space: Using the context, each word is mapped to a word vector in a fixed M-dimensional semantic space.

[0036] Since the text is calculated by mapping words to points in space, the "bag of words" form is used to map words to an N-dimensional vector space (where N is determined by the number of words in the vector space, and N can reach tens of thousands of dimensions). However, the disadvantages of the "bag of words" vector space method are: (1) it is easily troubled by the curse of dimensionality, and a vector space may reach tens of thousands of dimensions; (2) the isolation between words is serious, and it is not possible to well characterize the similarity between words.

[0037] For example, the two words "account number" and "account" are very similar in meaning in banking-related business, but the "distance" between them is very far. The so-called "distance" between two words represents the similarity between the syntax and semantics of the two words. For example, the "bag of words" form of "account number" in vector space is: [0000100000000000...]1*N; the "bag of words" form of "account" in vector space is: [0000000000100000...]1*N.

[0038] In order to solve this problem, the semantic space is introduced. The semantic space uses the context (i.e., considering the co-occurring words in the T sliding windows to the left and right of the word) to map each word to a word vector in a fixed M-dimensional semantic space (such as setting M = 300). For example, the "word vector" form of "account" in the semantic space is: [0.01 0.07 0.96 0.32 0.24...] 1*300; the "word vector" form of "account" in the vector space is: [0.02 0.24 0.84 0.45 0.01...] 1*300.

[0039] Advantages of semantic space: (1) Reduce the curse of vector dimensionality and improve computing power; (2) Eliminate isolation. Words with similar meanings are very close in semantic space.

[0040] In order to facilitate understanding of the solution of the embodiment of the present application, the method of sensitive data in the related art is first analyzed and explained:

[0041] Implementing the national big data strategy and promoting the open sharing of data resources is a good thing that benefits the country and the people. However, once a large amount of user privacy information is leaked or illegally used, it will seriously endanger the privacy and personal information security of users, and even seriously affect the security of national critical network infrastructure. Therefore, while promoting the open sharing of data resources, how to effectively protect the security of user sensitive data has become a key issue.

[0042] In the related technology, the main methods for identifying sensitive data include: regular expression feature library, calculating vocabulary weights to obtain sensitive words, using sensitive word library, etc. Among them, the regular expression feature library can quickly and effectively identify information, but due to the limited regular library, a lot of sensitive data is missed, and the actual situation is relatively complex and difficult to deal with; calculating vocabulary weights to obtain sensitive words: processing documents, calculating the vector weight value of each word, manually identifying and classifying the basic data corpus, and forming a sensitive word corpus; using the sensitive word vector library, classifying it, setting the benchmark words for each category, and converting the data to be detected into a classification process.

[0043] After analyzing the relevant technologies, we found the following shortcomings:

[0044] Regular expression feature library: Regular expressions can be used to quickly identify sensitive information such as ID number, password account, email address, phone number, address, etc. However, the number of regular rule libraries is limited, and the identification methods are limited. They will be helpless when encountering complex data. For example, "disease" corresponds to physical health status, and "income" corresponds to income and position information. For this information, a single attribute may be normal data, but if multiple attributes are combined, user privacy may be leaked. For example, a combination of attributes such as "name, ID card, disease" will leak user privacy.

[0045] Obtain sensitive words by vocabulary weight: The vector weight value of the vocabulary set in the corpus can be calculated by using vocabulary weight calculation methods such as term frequency-inverse document frequency index (TF-IDF). Words with high frequency in all training data sets are considered to be more important and have high corresponding weight values. Representative words are further manually selected and marked as sensitive data words to form a sensitive vocabulary corpus. However, this method has a significantly slower processing speed when facing big data, and it requires high personnel capabilities. Moreover, different experts may have different understandings, which may cause large errors.

[0046] Use sensitive word libraries to filter sensitive data: Building a sensitive word library is a relatively difficult task and will rely too much on human factors. If a comprehensive and accurate sensitive word library can be automatically built, it will be of great help in identifying sensitive data.

[0047] In summary, the methods in the relevant technologies are all based on statistical methods of marking rule bases and sensitive words, which are limited by the quantity, quality and human factors of the rule base itself. Even when using machine learning algorithms, there will be large false alarm rates and missed alarm rates because semantics are not analyzed.

[0048] Based on at least one problem existing in the related art, the embodiments of the present application provide a data processing method, apparatus, device and storage medium, which fully considers the text context based on the text semantic space, proposes the idea of ​​"lexical closeness", and adaptively and automatically constructs and mines a more comprehensive lexical closeness rule library, thereby being able to better break through the resource limitations of the current rule library, regular library, etc. while reducing human participation, and greatly improving the accuracy of identifying sensitive information using lexical closeness rules.

[0049] Figure 1 An optional flow chart of the data processing method provided in the embodiment of the present application is as follows: Figure 1 As shown, the method comprises the following steps:

[0050] Step S101 , clustering the word vectors in the semantic space according to at least one word vector corresponding to the sample data to obtain at least one cluster.

[0051] Here, the sample data may be any type of sample data, and the sample data may include both sensitive sample data and normal sample data, wherein the sensitive sample data contains sensitive information, for example, the sensitive information may be any sensitive information such as the user's ID number, health status, salary income, etc. The information contained in the normal sample data is normal information, and the normal information is any other information except the sensitive information, for example, weather conditions, hobbies, etc.

[0052] In an embodiment of the present application, the text in the sample data includes at least one word, each word corresponds to a word vector, and therefore, the sample data corresponds to at least one word vector. The word vectors in the semantic space are clustered according to the at least one word vector, and each word vector in the at least one word vector is used as a cluster center point to cluster the word vectors in the semantic space, and cluster the word vectors in the semantic space that belong to the same class as any word vector in the at least one word vector to form a cluster cluster.

[0053] When clustering the word vectors in the semantic space, clustering can also be performed based on the attribute information of the word vectors, and the word vectors with the same attribute information can be clustered into a cluster.

[0054] Step S102, performing a step-by-step hierarchical word vector expansion on each cluster, and constructing a vocabulary closeness rule corresponding to the cluster.

[0055] Here, the word vector expansion refers to constructing a binary tree for the word vectors in each cluster, thereby achieving a step-by-step hierarchical word vector expansion for the clusters.

[0056] In an embodiment of the present application, when expanding the word vector for each cluster, the word vector expansion can be performed based on the closeness between the word vectors, wherein the closeness between the word vectors refers to the distance between two word vectors.

[0057] For example, the word vector of "account" in the semantic space is expressed as: [0.01 0.07 0.96 0.320.24…]1*300; the word vector of "account" in the vector space is expressed as: [0.02 0.24 0.84 0.450.01…]1*300. Then, by calculating the distance between the word vectors of these two words in the semantic space, we can get the closeness between the word vectors corresponding to the two words "account" and "account".

[0058] In an embodiment of the present application, the word vector expansion of each cluster is performed step by step in a hierarchical manner, and the vocabulary closeness rule of the corresponding cluster constructed can be a vocabulary closeness rule tree in the form of a binary tree, and each layer of the vocabulary closeness rule tree includes at least one node formed by a word vector. At the bottom layer of the vocabulary closeness rule tree, there is a node formed by the word vector of the center point of the cluster. The closer to the top of the vocabulary closeness rule tree, the more nodes there are.

[0059] In the embodiment of the present application, each layer of word vector expansion has an attenuation factor, and the attenuation factor corresponding to the word vector expansion in the outer layer is stronger. That is, in the process of word vector expansion, the attenuation factor corresponding to the word vector expansion gradually increases. Among them, the attenuation factor refers to the parameter that reduces the distance between word vectors when word vector expansion is performed.

[0060] For example, when the first layer of word vectors is expanded, the distance between the word vector of the expansion center point and the expanded word vector is L1, when the second layer of word vectors is expanded, the distance between the word vector of the expansion center point and the expanded word vector is L2, and when the third layer of word vectors is expanded, the distance between the word vector of the expansion center point and the expanded word vector is L3, where the reduction parameter between L1 and L2 is K1, and the reduction parameter between L2 and L3 is K2, and K2 is greater than K1.

[0061] Step S103: using the vocabulary closeness rule to detect sensitive information in the data to be detected.

[0062] Here, after the vocabulary closeness rule is determined, the vocabulary closeness rule is used to detect sensitive information in the data to be detected to determine whether sensitive information exists in the data to be detected.

[0063] The data processing method provided in the embodiment of the present application first clusters the word vectors in the semantic space according to at least one word vector corresponding to the sample data to obtain at least one cluster; then, the word vectors of each cluster are expanded step by step and in layers to construct the vocabulary closeness rules of the corresponding cluster; finally, the vocabulary closeness rules are used to detect the sensitive information in the data to be detected. In this way, more comprehensive vocabulary closeness rules can be automatically constructed and mined adaptively, so that the resource limitations of the current rule base, regular library, etc. can be better broken through with less human participation, greatly improving the accuracy of using vocabulary closeness rules to detect sensitive information in the data to be detected.

[0064] Figure 2 An optional flow chart of the data processing method provided in the embodiment of the present application is as follows: Figure 2 As shown, the method comprises the following steps:

[0065] Step S201: filter the sample data using a regular rule library to obtain filtered sample data.

[0066] Here, the regular rule library includes at least one regular rule, which can be customized according to the classification and grading principle. The regular rule analyzes the characteristics of the sample data, classifies the user identity and other related data according to the data sensitivity, and calculates the risk based on the preset leakage loss characteristics to obtain data classification, which is divided into extremely sensitive level, sensitive level, relatively sensitive level, and low sensitivity level. Among them, the extremely sensitive level is the primary concern and the data that needs to be protected the most. The extremely sensitive level mainly includes ID number, password, passport, religious belief, address, personal health and other situations.

[0067] In the embodiment of the present application, the sample data can be automatically filtered using the regular rule library of the extremely sensitive level to obtain the filtered sample data, and the filtered sample data is the sample data of the extremely sensitive level. Of course, the sample data can also be filtered using the regular rule library of the extremely sensitive level and the sensitive level at the same time to obtain the sample data of the sensitive level.

[0068] Step S202: using a word segmentation algorithm to convert the filtered sample data into at least one word.

[0069] Here, at least one word is obtained by performing word segmentation processing on the text of the filtered sample data.

[0070] Step S203: Map the vocabulary that meets the preset part-of-speech condition in the at least one vocabulary to the semantic space to obtain at least one word vector corresponding to the sample data.

[0071] Here, the preset part-of-speech condition can be nouns and verbs, etc., or noun phrases or verb phrases. Since in the text, nouns and actions, or noun phrases and verb phrases can best reflect the meaning of the text, subsequent processing can be performed according to the part of speech that best reflects the content to be expressed in the text. At the same time, useless words such as stop words in the text can be removed. That is, only words of the part of speech such as nouns, noun phrases, verbs and verb phrases that can truly express the meaning of the sentence are retained.

[0072] In the embodiment of the present application, after useless words are removed, the remaining words are mapped to a semantic space to obtain at least one word vector corresponding to the sample data.

[0073] Step S204: clustering the word vectors in the semantic space according to at least one word vector corresponding to the sample data to obtain at least one cluster.

[0074] Step S205 , performing a step-by-step hierarchical word vector expansion on each cluster, and constructing a vocabulary closeness rule corresponding to the cluster.

[0075] Step S206: using the vocabulary closeness rule to detect sensitive information in the data to be detected.

[0076] It should be noted that steps S204 to S206 are the same as the above-mentioned steps S101 to S103, and will not be repeated in the embodiment of the present application.

[0077] The data processing method provided in the embodiment of the present application uses a regular rule library to filter the sample data before clustering the word vectors in the semantic space to obtain the filtered sample data; uses a word segmentation algorithm to convert the filtered sample data into at least one vocabulary; maps the vocabulary that meets the preset part-of-speech condition in the at least one vocabulary to the semantic space to obtain at least one word vector corresponding to the sample data. In this way, only the vocabulary that can truly express the meaning of the sentence is retained, so that when these vocabulary are used to form word vectors, effective word vectors can be formed, and sensitive information is formed based on these effective word vectors to detect sensitive information in the data to be detected, which can greatly improve the accuracy of sensitive information detection in the data to be detected.

[0078] Figure 3 An optional flow chart of the data processing method provided in the embodiment of the present application is as follows: Figure 3As shown, the method comprises the following steps:

[0079] Step S301 , taking at least one word vector corresponding to the sample data as a cluster center point, clustering the word vectors in the semantic space to obtain at least one first cluster.

[0080] Here, when performing clustering processing, the cluster center point is first determined. During the initial clustering processing, at least one word vector corresponding to the sample data is determined as the cluster center point, and then the word vectors in the semantic space are clustered according to the cluster center point.

[0081] In an embodiment of the present application, during clustering processing, the word vectors in the semantic space that belong to the same category as the word vector of the cluster center point and the word vector corresponding to the cluster center point are determined as a cluster cluster, or the word vectors with semantically similar word vectors corresponding to the cluster center point are determined as a cluster cluster, thereby obtaining the at least one first cluster cluster.

[0082] Step S302: when the first number of the first clusters is equal to a preset number threshold, determine the first cluster as the at least one cluster.

[0083] The preset number threshold is the upper limit of the number of the cluster center points, that is, the preset number threshold is the upper limit of the number of the cluster clusters. The preset number threshold can be preset according to actual needs.

[0084] In an embodiment of the present application, after completing the clustering process of the word vectors in the semantic space according to the word vectors corresponding to the sample data, the first number of the first clusters formed by the clustering process is determined. When the first number of the first clusters is equal to the preset number threshold, it indicates that the number of cluster centers has reached an upper limit, or the number of the clusters has reached an upper limit, and therefore, the first cluster formed is determined as the at least one cluster.

[0085] Step S303: when the first number is less than a preset number threshold, determine the isolated word vectors in the clustering process.

[0086] Here, when the first number is less than the preset number threshold, it indicates that the number of cluster center points has not reached the upper limit, or the number of cluster clusters has not reached the upper limit. Therefore, it is necessary to continue clustering to obtain cluster clusters so that the number of cluster clusters reaches the preset number threshold.

[0087] In an embodiment of the present application, when the first number is less than the preset number threshold, clustering processing continues. At this time, we can focus on the isolated points that appear during the clustering processing in step S301, that is, the isolated word vectors in the clustering processing process, and discover new categories with the help of isolated word vectors.

[0088] Step S304: clustering the word vectors in the semantic space with the isolated word vector as the cluster center point to obtain at least one second cluster.

[0089] Here, the isolated word vectors that are far from the cluster center point corresponding to the sample data are set as the cluster center point of the new class, and the word vectors in the semantic space are clustered again to obtain at least one second cluster.

[0090] If after clustering again, the sum of the number of second clusters and the number of first clusters is still less than the preset threshold, continue to look for isolated word vectors, and continue clustering with isolated word vectors to obtain second clusters until the sum of the second number of the second clusters and the first number is equal to the preset threshold.

[0091] Step S305: determine the first cluster and the second cluster as the at least one cluster.

[0092] Here, when the sum of the second number of the second clusters and the first number of the first clusters is equal to the preset number threshold, it indicates that the number of cluster center points has reached an upper limit, or the number of clusters has reached an upper limit, and therefore, the first cluster and the second cluster formed are determined as the at least one cluster.

[0093] Step S306, each time the word vector of the cluster is expanded, determining the expansion center point of the current word vector expansion.

[0094] Here, when expanding the word vector of the cluster cluster, the method of expanding from a single word vector to a group of word vectors can be adopted. Therefore, Xiaoyangtai first selects the word vector corresponding to the expansion center point, so as to realize the word vector expansion based on the expansion center point.

[0095] Step S307 , performing a preset number of word vector expansions on the corresponding cluster according to the distance between the word vector of the expansion center point and other word vectors, and constructing a vocabulary closeness rule for the corresponding cluster.

[0096] Here, when performing word vector expansion on any cluster, the distance between the word vector of the expansion center point in the cluster and other word vectors in the cluster can be calculated, and then the word vector expansion is performed on the cluster a preset number of times according to the distance.

[0097] It should be noted that the preset number of times can be pre-set according to actual needs. When the preset number of word vector expansions is performed on the cluster cluster, the number of layers of the word vector closeness rule tree corresponding to the word vector closeness rule finally formed is equal to the value corresponding to the preset number of times plus 1. For example, when the preset number of times is 3 times, the number of layers of the word vector closeness rule tree finally formed is 4 layers.

[0098] Step S308: using the vocabulary closeness rule to detect sensitive information in the data to be detected.

[0099] The data processing method provided in the embodiment of the present application uses the isolated word vector as the cluster center point to cluster the word vectors in the semantic space to obtain at least one second cluster cluster. This makes it easy to discover new categories and avoid missing sensitive information of new categories, thereby improving the detection accuracy of sensitive information in the data to be detected.

[0100] Figure 4 An optional flow chart of the data processing method provided in the embodiment of the present application is as follows: Figure 4 As shown, the method comprises the following steps:

[0101] Step S401: cluster the word vectors in the semantic space according to at least one word vector corresponding to the sample data to obtain at least one cluster.

[0102] It should be noted that step S401 is the same as the above-mentioned step S101 and will not be described in detail in the embodiment of the present application.

[0103] Step S402: when the cluster is initially expanded, the center point word vector of the cluster is determined as the expansion center point.

[0104] Here, since the word vector expansion is performed on the clusters for a preset number of times, each time the word vector is expanded, it is necessary to first determine the expansion center point and expand with the expansion center point as the center.

[0105] In the embodiment of the present application, when the cluster is initially expanded, the center point word vector of the cluster is determined as the expansion center point, and the cluster is initially expanded using the center point word vector of the cluster.

[0106] Step S403: when the current word vector expansion is the i-th expansion of the cluster, the expanded word vector obtained by the i-1-th expansion is determined as the expansion center point of the current word vector expansion.

[0107] Here, i is greater than 1. When the current word vector expansion is the i-th expansion of the cluster, it indicates that the initial expansion has been completed, so the expanded word vector obtained in the last expansion is determined as the expansion center point, so as to continue to expand the cluster.

[0108] Step S404, when performing each word vector expansion, determine the distance between the word vector of the expansion center point and other word vectors.

[0109] Step S405 , determining a first preset number of other word vectors as the extended word vectors of the extended center point in order of the distance from the smallest to the smallest, thereby constructing a vocabulary closeness rule of the corresponding cluster.

[0110] Here, the distance between the word vector of the extended center point and other word vectors is determined, and a first preset number of other word vectors are selected from multiple other word vectors as the extended word vectors of the extended center point. When selecting, the selection is made based on the distance, and the first preset number of other word vectors with the closest distance are selected.

[0111] In the embodiment of the present application, after selecting a first preset number of other word vectors, a binary tree is constructed based on the selected other word vectors to form a vocabulary closeness rule corresponding to the clustering cluster.

[0112] In some embodiments, when any word vector is determined to be the extended word vector, the corresponding word vector is deleted from the cluster to avoid repeated expansion of the word vector.

[0113] Step S406: Use the vocabulary closeness rule to detect sensitive information in the data to be detected.

[0114] The data processing method provided by the embodiment of the present application determines the distance between the word vector of the expansion center point and other word vectors when performing each word vector expansion; and determines the first preset number of other word vectors as the expansion word vector of the expansion center point in order from the smallest to the smallest, thereby constructing the vocabulary closeness rule of the corresponding cluster. In this way, the expansion can be carried out in sequence according to the vocabulary closeness, and the target is more accurate when selecting the leaf node vocabulary of the next layer, rather than a divergent random combination of vocabulary, so that a more accurate vocabulary closeness rule can be obtained, and a binary tree structure is used to facilitate the deletion and final merging of rules.

[0115] In some embodiments, there is a first distance between the center point word vector and the extended word vector at the i-1th expansion, and there is a second distance between the center point word vector and the extended word vector at the i-th expansion;

[0116] The attenuation factor δ between the first distance and the second distance can be calculated using the following formula:

[0117]

[0118] Among them, L i is the i-th expansion layer.

[0119] Based on the above embodiments, the embodiments of the present application provide a data processing method, which constructs and mines a vocabulary closeness rule tree from massive user privacy data based on the "vocabulary closeness" concept in semantic space, so as to automatically discover, mine, and expand sensitive rules.

[0120] The method of the embodiment of the present application is divided into four steps: first, construct a regular rule library to filter sensitive data samples; second, construct a lexical semantic space based on sensitive samples; third, cluster the semantic space with the regular library rules as the center point; fourth, construct and mine the lexical closeness rule tree (i.e. the above-mentioned lexical closeness rule) in parallel based on the clustering clusters.

[0121] Figure 5 An optional flow chart of the data processing method provided in the embodiment of the present application is as follows: Figure 5 As shown, the method comprises the following steps:

[0122] Step S501: construct a regular rule library to filter sensitive data samples.

[0123] The biggest challenge when building a model for training samples is how to quickly collect and label samples (samples are divided into sensitive and normal data samples), which requires a lot of manual work, and often results in different sample labels due to subjective factors of human judgment. Although the regular rule library itself has defects such as "limited quantity" and "limited capabilities", which leads to the omission of some truly sensitive data and the misjudgment of some normal data as sensitive data, it can be used as a tool for preliminary automatic identification of some sensitive data.

[0124] According to the classification and grading principles, a regular rule library is customized, data characteristics are analyzed, and based on the characteristics of internal management and external open scenarios, user identity-related data is classified according to data sensitivity and risk calculation is performed based on preset leakage loss characteristics to obtain data classification, which is divided into extremely sensitive, sensitive, relatively sensitive, and low sensitivity levels. Among them, the extremely sensitive level is the primary concern and the data that needs the most protection. The extremely sensitive level mainly includes ID number, password, passport, religious belief, address, personal health, etc.

[0125] The data automatically filtered by the regular rule library of the extremely sensitive level is used as sensitive data, wherein the part hit by the regular rule is replaced by the corresponding "regular expression" in the regular rule library of Table 1 below.

[0126] Table 1 Regular rule base

[0127]

[0128]

[0129] Step S502: constructing a lexical semantic space based on sensitive samples.

[0130] When calculating text, the text needs to be represented by words. The calculation of text is essentially the calculation between words. First, the sensitive data samples filtered by the above steps are preprocessed, and the text is converted into a word vector using a word segmentation algorithm. Then, useless words such as stop words are removed, and words such as nouns, noun phrases, verbs, and verb phrases that can truly express the meaning of the sentence are retained. The words are mapped to word vectors in the semantic space.

[0131] Step S503: clustering is performed in the semantic space with the regular library rule as the center point.

[0132] In an embodiment of the present application, word vectors in a semantic space are dynamically clustered, wherein the clustering principles include: setting an upper limit of K on the number of new center points during the clustering process; performing a preliminary division on the entire data set, using the content identified by the regular rules in the regular rule library as the center point, and clustering semantically similar word vectors together; focusing on isolated points during the clustering process to discover new categories with their help, and if the upper limit K of the clustering center has not been reached, setting isolated points that are farther away from all center points as new class center points; and continuously performing iterations to ultimately obtain K stable class clusters of categories.

[0133] Step S504, constructing and mining vocabulary closeness rule trees in parallel based on the clusters.

[0134] The data processing method of the embodiment of the present application is a method based on "lexical closeness" of semantic space, which is a method of expanding from a single word vector to a set of word vector rules, that is, the principle of water wave diffusion of ideas. First, the center point of the word vector is selected, and the word vector is expanded step by step in a hierarchical manner, wherein each layer of expansion has an attenuation factor. The stronger the attenuation factor is, the dimensionality of the expansion will be controlled.

[0135] like Figure 6 As shown, the vocabulary density concept diagram provided by the embodiment of the present application has a center point 61 at the most central position, that is, the expansion center point. Outside the expansion center point, there are gradual and hierarchical expansions to form 1 round expansion 62, 2 round expansion 63...U round expansion 64, etc.

[0136] Corresponding deletion rules are made according to the situations of hitting normal data samples and sensitive data samples. In actual operation, it is more convenient to delete and adjust nodes using tree structure, so the method of constructing "lexical closeness rule tree" for each cluster after clustering is adopted to mine sensitive rules.

[0137] In an embodiment of the present application, the construction principles of the "lexical closeness rule tree" include: in each round of expansion, m word vectors are expanded, and the m word vectors closest to the selected word vector are calculated based on the semantic space. After each round of expansion, the selected word vector is deleted from the original cluster to prevent repeated calculation; at the same time, a binary tree is constructed, in which the left node is a child node and the right node is a brother node; the attenuation factor of each round of expanded words is calculated according to the number of expansion rounds, and is calculated in an exponential form with e as the base, in which the initial attenuation is slow and the subsequent attenuation is accelerated.

[0138] In some embodiments, in the calculation method of constructing a "lexical closeness rule tree", a tree node is a four-tuple <word vector, normal hit frequency, sensitive hit frequency, endpoint identification bit>,

[0139] The statistics and parameters are as follows: the hit frequency of normal corpus is TF_nom; the hit frequency of sensitive data corpus is TF_ses; the probability of misjudging normal corpus is P, where the probability of misjudging normal corpus P is the proportion of normal hit frequency to all corpus, and 1-P is the accuracy probability; the probability of misjudging normal corpus P can be calculated using the following formula:

[0140] P=(TF_nom) / (TF_nom+TF_ses);

[0141] Among them, the misjudgment probability threshold is α, the accuracy probability threshold is β, and the attenuation factor is

[0142] Among them, L i It is the i-th expansion layer. After the fourth layer, the attenuation is serious, so it is expanded to the fourth layer at most.

[0143] The statistical judgment process is as follows Figure 7 As shown, from the root node (Root) to the node path, the TF_nom and TF_ses of each path are counted respectively, and the probability P of misjudging normal corpus is calculated. For the three states of the node (deletion, terminal node identification bit = 1, non-terminal node identification bit = 0), if the node is retained, the probability that the node is judged to belong to a sensitive word is P_sen = δ(1-P);

[0144] Please refer to Figure 7 , is a flow chart of judgment rules provided in an embodiment of the present application, and the judgment rule flow includes the following steps:

[0145] Step S701, obtain all sample data, and count the normal corpus hit frequency TF_nom and the sensitive data corpus hit frequency TF_ses of each path.

[0146] Step S702, determining whether the probability P of misjudging normal corpus is greater than a misjudgment probability threshold α.

[0147] If the judgment result is yes, execute step S703, if the judgment result is no, execute step S704.

[0148] Step S703, delete the node.

[0149] Step S704, determining whether the accuracy probability 1-P is less than the accuracy probability threshold β.

[0150] If the judgment result is yes, execute step S705, if the judgment result is no, execute step S706.

[0151] Step S705: Set the termination node identification bit to 1.

[0152] Step S706, setting the end node identification bit to 0.

[0153] In some embodiments, the steps of constructing the vocabulary closeness rule tree are as follows, wherein the word vector node situation of each round is obtained by Figure 7 The judgment rule flow chart is calculated, and the rule tree is constructed as follows Figure 8 As shown, the construction process includes: first, constructing a central point Hash table, and establishing a Hash table 801 with the central points of each cluster, so as to facilitate and quickly find the corresponding tree structure. The regular rule class uses "regular expression" as the central point (see the regular rule library provided in Table 1), and the Hash table 801 includes the keyword Hash, ID card number regular, password regular, passport regular, religion regular, address regular, health regular and new category regular, wherein the new category regular uses the cluster center as the central point by Ci (see Figure 8 C8, C9...Ck, etc.) in it;

[0154] Then, the first round of expansion is carried out: the distances from all points to the center point in each cluster are calculated, the m word vectors with the closest semantics are selected, and inserted into the rule tree, named L1i, which is the i-th node of the first layer. The calculation method is used to calculate the probability P of misjudging normal corpus and judge the three states of the node.

[0155] Then, the second round of expansion: calculate the distance from all points in each cluster to the "center point & first round expansion node", select the m word vectors with the closest semantics, insert them into the tree, name them L2i, which is the i-th node of the second layer, and use the calculation method to calculate the probability P of misjudging normal corpus and judge the three states of the node.

[0156] Then, in the third round of expansion, repeat the above expansion steps.

[0157] Finally, traverse the binary tree to form the final rule. Figure 8 The final sensitive rules formed by the lexical closeness rule tree are as follows:

[0158] ID card number regular expression & (L11|L12|L13) Ck& (L13|L19)

[0159] ID card number regular expression &L21&L31 Ck&L11&(L21|L22|L25)

[0160] ID card number regular expression&L22

[0161] Among them, each word has the probability P_sen of being a sensitive word.

[0162] The data processing method provided in the embodiment of the present application expands new classes based on dynamic clustering in semantic space: dynamic clustering is performed based on semantic space, and on the basis of original classes, attention is paid to "isolated points" in order to discover new categories; and it is proposed to construct a "lexical closeness" rule tree based on semantic space and mine sensitive rules: using the "lexical closeness" idea, lexical closeness rule trees are constructed for multiple clustering clusters in parallel, and the attenuation factor is used to control the expansion of each layer of nodes in the constructed tree.

[0163] The data processing method provided in the embodiment of the present application first uses samples filtered by a regular rule library formulated according to the classification and grading principles as sensitive samples, deeply explores the implicit rules therein, and reduces the large amount of work of manually marking samples; and based on text semantic space calculation, fully considers the context, eliminates the situation where the vocabulary is isolated in the calculation space, and calculates the similarity of vocabulary at the grammatical and semantic levels; at the same time, constructs a vocabulary closeness rule tree to mine sensitive rules, and uses the idea of ​​"vocabulary closeness" to select the leaf node vocabulary of the next layer with a more accurate target, rather than a divergent and random combination of vocabulary; uses a binary tree structure to facilitate the deletion and final merging of rules.

[0164] Based on the foregoing embodiments, an embodiment of the present application provides a data processing device, which includes the modules included and the components included in each module, and can be implemented by a processor in a data processing device; of course, it can also be implemented by a logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0165] Fig. 9 A schematic diagram of the structure of the data processing device provided in the embodiment of the present application is shown in FIG. Fig. 9 As shown, the data processing device 900 includes:

[0166] A clustering processing unit 901 is used to perform clustering processing on the word vectors in the semantic space according to at least one word vector corresponding to the sample data to obtain at least one cluster cluster;

[0167] The word vector expansion unit 902 is used to expand the word vector of each cluster in a step-by-step hierarchical manner to construct a vocabulary closeness rule for the corresponding cluster; wherein each layer of word vector expansion has an attenuation factor, and the attenuation factor corresponding to the word vector expansion in the outer layer is stronger;

[0168] The identification processing unit 903 is used to detect sensitive information in the data to be detected by using the vocabulary closeness rule.

[0169] In some embodiments, the word vector expansion unit includes:

[0170] A first determination module is used to determine the expansion center point of the current word vector expansion each time the word vector expansion is performed on the cluster;

[0171] The word vector expansion module is used to perform a preset number of word vector expansions on the corresponding cluster cluster according to the distance between the word vector of the expansion center point and other word vectors, and construct a vocabulary closeness rule for the corresponding cluster cluster.

[0172] In some embodiments, the first determination module is also used for: when the cluster is initially expanded, determining the center point word vector of the cluster as the expansion center point; when the current word vector expansion is the i-th expansion of the cluster, determining the expanded word vector obtained by the i-1-th expansion as the expansion center point of the current word vector expansion, where i is greater than 1.

[0173] In some embodiments, the word vector expansion module is also used to: determine the distance between the word vector of the expansion center point and other word vectors when performing each word vector expansion; and determine a second preset number of other word vectors as the expanded word vectors of the expansion center point in order from the smallest to the smallest, thereby constructing the vocabulary closeness rules of the corresponding clustering clusters.

[0174] In some embodiments, the apparatus further comprises:

[0175] A deleting unit is used to delete the corresponding word vector from the cluster when any word vector is determined to be the extended word vector.

[0176] In some embodiments, the apparatus further comprises:

[0177] A filtering unit, used to filter the sample data using a regular rule library to obtain the filtered sample data;

[0178] A conversion unit, configured to convert the filtered sample data into at least one vocabulary by using a word segmentation algorithm;

[0179] A mapping unit is used to map the vocabulary that meets the preset part-of-speech condition in the at least one vocabulary to the semantic space to obtain at least one word vector corresponding to the sample data.

[0180] In some embodiments, the cluster processing unit includes:

[0181] A clustering processing module, configured to perform clustering processing on the word vectors in the semantic space by taking at least one word vector corresponding to the sample data as a cluster center point, so as to obtain at least one first cluster;

[0182] The second determining module is configured to determine the first cluster as the at least one cluster when the first number of the first clusters is equal to a first preset number threshold.

[0183] In some embodiments, the apparatus further comprises:

[0184] A first determining unit, configured to determine isolated word vectors in the clustering process when the first number is less than the first preset number threshold;

[0185] A second clustering processing unit, configured to perform clustering processing on the word vectors in the semantic space with the isolated word vector as a cluster center point to obtain at least one second cluster; the sum of a second number of the second clusters and the first number is equal to the first preset number threshold;

[0186] The second determining unit is configured to determine the first cluster and the second cluster as the at least one cluster.

[0187] It should be noted that in the embodiments of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product, which is stored in a storage medium and includes several instructions for a terminal to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0188] Correspondingly, an embodiment of the present application provides a data processing device, Fig.10 A schematic diagram of the structure of the data processing device provided in the embodiment of the present application is shown in FIG. Fig.10 As shown, the data processing device 1000 includes at least: a processor 1001 , a communication interface 1002 , and a storage medium 1003 configured to store executable instructions, wherein: the processor 1001 generally controls the overall operation of the data processing device 1000 .

[0189] The communication interface 1002 enables the data processing device to communicate with other terminals or servers through a network.

[0190] The storage medium 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or processed by the processor 1001 and each module in the data processing device 1000, which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0191] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0192] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, an element defined by the statement "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed.

[0193] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art may understand that all or part of the steps of implementing the above method embodiment may be completed by hardware related to program instructions, and the aforementioned program may be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories, magnetic disks or optical disks. Alternatively, if the above integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a terminal to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage media include: mobile storage devices, ROM, magnetic disks, optical disks, and other media that can store program codes.

[0194] The above is only an embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: According to at least one word vector corresponding to the sample data, clustering is performed on the word vectors in the semantic space to obtain at least one clustering cluster; When expanding the word vector of each cluster in a step-by-step hierarchical manner, determine the expansion center point of the current word vector expansion; According to the distance between the word vector of the expansion center point and other word vectors, the word vector expansion of the corresponding cluster cluster is performed for a preset number of times, and the vocabulary closeness rule of the corresponding cluster cluster is constructed; wherein, when performing each word vector expansion, the distance between the word vector of the expansion center point and other word vectors is determined; according to the order of the distance from small to large, the first preset number of other word vectors are sequentially determined as the expanded word vectors of the expansion center point; the vocabulary closeness rule includes a vocabulary closeness rule tree in the form of a binary tree; each layer of word vector expansion has an attenuation factor, and the attenuation factor corresponding to the word vector expansion of the outer layer is stronger; The vocabulary closeness rule is used to detect sensitive information in the data to be detected.

2. The method according to claim 1, characterized in that Each time the word vector is expanded for the cluster, determining the expansion center point of the current word vector expansion includes: When the cluster is initially expanded, the center point word vector of the cluster is determined as the expansion center point; When the current word vector expansion is the i-th expansion of the cluster, the expanded word vector obtained by the i-1-th expansion is determined as the expansion center point of the current word vector expansion, where i is greater than 1.

3. The method according to claim 2, characterized in that The method further comprises: When any word vector is determined to be the extended word vector, the corresponding word vector is deleted from the cluster.

4. The method according to claim 1, characterized in that: Before clustering the word vectors in the semantic space according to at least one word vector corresponding to the sample data, the method further includes: Using a regular rule library to filter the sample data to obtain the filtered sample data; Using a word segmentation algorithm, converting the filtered sample data into at least one word; The vocabulary that meets the preset part-of-speech condition in the at least one vocabulary is mapped to the semantic space to obtain at least one word vector corresponding to the sample data.

5. The method according to claim 1, characterized in that The step of clustering the word vectors in the semantic space according to at least one word vector corresponding to the sample data to obtain at least one clustering cluster includes: Taking at least one word vector corresponding to the sample data as a cluster center, clustering the word vectors in the semantic space to obtain at least one first cluster; When the first number of the first clusters is equal to a preset number threshold, the first cluster is determined as the at least one cluster.

6. The method according to claim 5, characterized in that The method further comprises: When the first number is less than the preset number threshold, determining isolated word vectors in the clustering process; Taking the isolated word vector as the cluster center, clustering the word vectors in the semantic space to obtain at least one second cluster; the sum of a second number of the second clusters and the first number is equal to the preset number threshold; The first cluster and the second cluster are determined as the at least one cluster.

7. A data processing device, characterized in that: The device comprises: A clustering processing unit, configured to perform clustering processing on the word vectors in the semantic space according to at least one word vector corresponding to the sample data, to obtain at least one cluster cluster; A word vector expansion unit is used to determine the expansion center point of the current word vector expansion when the word vector of each cluster is expanded step by step and hierarchically; and to perform a preset number of word vector expansions on the corresponding cluster according to the distance between the word vector of the expansion center point and other word vectors, and construct a vocabulary closeness rule for the corresponding cluster; wherein, when performing each word vector expansion, the distance between the word vector of the expansion center point and other word vectors is determined; and a first preset number of other word vectors are sequentially determined as the expanded word vectors of the expansion center point in the order of the distance from small to large; the vocabulary closeness rule includes a vocabulary closeness rule tree in the form of a binary tree; each layer of word vector expansion has an attenuation factor, and the attenuation factor corresponding to the word vector expansion in the outer layer is stronger; The identification processing unit is used to detect sensitive information in the data to be detected by using the vocabulary closeness rule.

8. A data processing device, characterized in that: The device comprises at least: a processor and a storage medium configured to store executable instructions, wherein: the processor is configured to execute the stored executable instructions; The executable instructions are configured to execute the data processing method provided in any one of claims 1 to 6 above.

9. A storage medium, characterized in that: The storage medium stores computer executable instructions, and the computer executable instructions are configured to execute the data processing method provided in any one of claims 1 to 6 above.

Images